Multi-Agent Reinforcement Learning (MARL) made significant progress in the last decade, mainly thanks to the major developments in the field of Deep Neural Networks (DNNs). However, DNNs suffer from a fundamental issue: their lack of interpretability. While this is true for most applications of DNNs, this is exacerbated in their applications in MARL. In fact, the mutual interactions between agents and environment, as well as across agents, make it particularly difficult to understand learned strategies in these settings. One possible way to achieve explainability in MARL is through the use of interpretable models, such as decision trees, that allow for a direct inspection and understanding of their inner workings. In this work, we make a step forward in this direction, proposing a population-based algorithm that combines evolutionary principles with RL for training interpretable models in multi-agent systems. We evaluate the proposed approach in a highly dynamic task where two teams of agents compete with each other. We test different variants of the proposed method in different settings, namely with/without coevolution and with/without initialization from a handcrafted policy. We find that, in most settings, our method is able to find fairly effective policies. Moreover, we show that the learned policies are easy to inspect and, possibly, interpreted based on domain knowledge.

A Population-Based Approach for Multi-Agent Interpretable Reinforcement Learning / Crespi, Marco; Ferigo, Andrea; Custode, Leonardo Lucio; Iacca, Giovanni. - In: APPLIED SOFT COMPUTING. - ISSN 1872-9681. - 2023, 147:(2023), pp. 11075801-11075812. [10.1016/j.asoc.2023.110758]

A Population-Based Approach for Multi-Agent Interpretable Reinforcement Learning

Ferigo, Andrea;Custode, Leonardo Lucio;Iacca, Giovanni
2023-01-01

Abstract

Multi-Agent Reinforcement Learning (MARL) made significant progress in the last decade, mainly thanks to the major developments in the field of Deep Neural Networks (DNNs). However, DNNs suffer from a fundamental issue: their lack of interpretability. While this is true for most applications of DNNs, this is exacerbated in their applications in MARL. In fact, the mutual interactions between agents and environment, as well as across agents, make it particularly difficult to understand learned strategies in these settings. One possible way to achieve explainability in MARL is through the use of interpretable models, such as decision trees, that allow for a direct inspection and understanding of their inner workings. In this work, we make a step forward in this direction, proposing a population-based algorithm that combines evolutionary principles with RL for training interpretable models in multi-agent systems. We evaluate the proposed approach in a highly dynamic task where two teams of agents compete with each other. We test different variants of the proposed method in different settings, namely with/without coevolution and with/without initialization from a handcrafted policy. We find that, in most settings, our method is able to find fairly effective policies. Moreover, we show that the learned policies are easy to inspect and, possibly, interpreted based on domain knowledge.
2023
Crespi, Marco; Ferigo, Andrea; Custode, Leonardo Lucio; Iacca, Giovanni
A Population-Based Approach for Multi-Agent Interpretable Reinforcement Learning / Crespi, Marco; Ferigo, Andrea; Custode, Leonardo Lucio; Iacca, Giovanni. - In: APPLIED SOFT COMPUTING. - ISSN 1872-9681. - 2023, 147:(2023), pp. 11075801-11075812. [10.1016/j.asoc.2023.110758]
File in questo prodotto:
File Dimensione Formato  
SSRN-id4467882.pdf

accesso aperto

Tipologia: Pre-print non referato (Non-refereed preprint)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 549.5 kB
Formato Adobe PDF
549.5 kB Adobe PDF Visualizza/Apri
1-s2.0-S1568494623007767-main.pdf

accesso aperto

Tipologia: Post-print referato (Refereed author’s manuscript)
Licenza: Creative commons
Dimensione 851.4 kB
Formato Adobe PDF
851.4 kB Adobe PDF Visualizza/Apri
1-s2.0-S1568494623007767-main.pdf

Solo gestori archivio

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 774.15 kB
Formato Adobe PDF
774.15 kB Adobe PDF   Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/385293
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 3
  • ???jsp.display-item.citation.isi??? 2
social impact