Reinforcement Learning for Multi-agent Teamwork

Abstract

Recently, attention was focused on multi-agent systems consisting of teams of autonomous agents acting in real-time, noisy, collaborative, and adversarial environments. Robotic soccer , electronic commerce and military combat simulation are just few examples of these domains. A central issue in these systems is multi­ agent teamwork or how to coordinate the behavior of the agents to achieve a team goal. Traditionally, this had been accomplished by hand-coding the coordination strategy. However, this task is complex due to the dynamism of the environment and the existence of adversaries. Recent work in this area has focused on how multi-agent teamwork can be learnt. Reinforcement learning has gained much attention in this task because it doesn't need a model of its environment and can be done online.

This thesis introduces a new multi-agent reinforcement learning algorithm, namely Generalized team-partitioned, opaque-transition reinforcement learning (GTPOT-RL) which builds upon and extends its parent (TPOT-RL). The algorithm is designed for domains in which agents cannot necessarily observe the state changes when other team members act. It exploits local, action-dependent features to aggressively generalize its input representation for learning. The algorithm requires a partitioning of the state space into disjoint regions and an assignment of a role to each agent to achieve a local goal. The agents then, simultaneously, learn collaborative policies by observing the long-term effects of their actions. By partitioning the state space and distributing the responsibilities among the agents and finally linking their goals, an effective way of cooperation emerges. The effectiveness of the new algorithm is demonstrated and contrasted to TPOT-RL using simulated robotic soccer which happens to be the standard test bed for multi-agent research in the recent years.