Communication-Efficient Actor-Critic Methods for Homogeneous Markov Games
Document Type
Article
Abstract
Recent success in cooperative multi-agent reinforcement learning (MARL) relies on centralized training and policy sharing. Centralized training eliminates the issue of non-stationarity MARL yet induces large communication costs, and policy sharing is empirically crucial to efficient learning in certain tasks yet lacks theoretical justification. In this paper, we formally characterize a subclass of cooperative Markov games where agents exhibit a certain form of homogeneity such that policy sharing provably incurs no suboptimality. This enables us to develop the first consensus-based decentralized actor-critic method where the consensus update is applied to both the actors and the critics while ensuring convergence. We also develop practical algorithms based on our decentralized actor-critic method to reduce the communication cost during training, while still yielding policies comparable with centralized training.
Publication Info
Preprint version The Tenth International Conference on Learning Representations (ICLR 2022), 2022.
© The Authors, 2022
APA Citation
Chen, D., Li, Y., & Zhang, Q. (2022). Communication-efficient actor-critic methods for homogeneous Markov games. International Conference on Learning Representations. https://openreview.net/forum?id=xy_2w3J3kH