Document Type
Article
Abstract
Problems involving controlling the collective behavior of a population of structurally similar dynamical systems, the so-called ensemble control, arise in diverse emerging applications and pose a grand challenge in systems science and control engineering. Owing to the severely under-actuated nature and the difficulty of placing large-scale sensor networks, ensemble systems are limited to being actuated and monitored at the population level. Moreover, mathematical models describing the dynamics of ensemble systems are often elusive. Therefore, it is essential to design broadcast controls that excite the entire population in such a way that the heterogeneity in system dynamics are robustly compensated. In this paper, we propose a reinforcement learning-based data-driven control framework incorporating population-level aggregated measurement data to learn a global control signal for steering a dynamic population in the desired manner. In particular, we introduce the notion of ensemble moments induced by aggregated measurements and derive the associated moment system to the original ensemble system. Then, using the moment system, we learn an approximation of optimal value functions and the associated policies in terms of ensemble moments through reinforcement learning. We illustrate the feasibility and scalability of the proposed moment-based approach via numerical experiments using a population of linear, bilinear, and nonlinear dynamic ensemble systems. We report that the proposed method achieves the desired control objectives of various ensemble control tasks and obtains significantly better averaged-reward when compared with existing methods.
Digital Object Identifier (DOI)
Publication Info
Postprint version. Published in IEEE Transactions on Neural Networks and Learning Systems, 2023.
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
APA Citation
Yu, Y., Narayanan, V., & Li, J. (2023). Moment-based reinforcement learning for ensemble control. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/TNNLS.2023.3264151
Included in
Artificial Intelligence and Robotics Commons, Controls and Control Theory Commons, Dynamic Systems Commons