Core Skill Decomposition of Complex Wargames with Reinforcement Learning


Kömürcü K. K., İnce B., Ok T., Kılıçkaya E., Üre N. K.

AIAA Science and Technology Forum and Exposition, AIAA SciTech Forum 2022, California, United States Of America, 3 - 07 January 2022 identifier

  • Publication Type: Conference Paper / Full Text
  • Doi Number: 10.2514/6.2022-2084
  • City: California
  • Country: United States Of America
  • Istanbul Technical University Affiliated: Yes

Abstract

© 2022, American Institute of Aeronautics and Astronautics Inc.. All rights reserved.In recent years, Reinforcement Learning (RL) agents were able to solve the most challenging games to the extent that they competed and even surpassed the most successful human players. This suggests that RL methods are well suited for wargames where the complexity arises from very long decision horizons, sparse rewards, and large action spaces. Due to the complex nature of wargames, even with RL, convergence to a near-optimum solution requires an immense amount of experience and makes the solution sample inefficient. In order to address the inefficiency, we propose to divide the game into simpler sub-games, where each sub-game covers a core skill of the game. These sub-games have shorter decision horizons and smaller action spaces compared to the main game. We employ a curriculum learning setting with a hierarchical control structure, where the curriculum consists of simpler sub-games. We choose StarCraft II as our test bench as it possesses the common features of wargames and it has been extensively used in wargame scenarios. We empirically show that our proposed hierarchical architecture is able to solve a complex wargame environment based on StarCraft II game whereas the non-hierarchical agent fails to solve. We further observed that a set of core skills is sufficient to achieve near-optimal scores, and a larger set of skills beyond the core skills only marginally improves the performance.