Home » Publication » 28136

Dettaglio pubblicazione

2024, National Conference of the American Association for Artificial Intelligence, Pages 21646-21655 (volume: 38)

Pure-Past Action Masking (04b Atto di convegno in volume)

Varricchione Giovanni, Alechina Natasha, Dastani Mehdi, De Giacomo Giuseppe, Logan Brian, Perelli Giuseppe

We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separation of concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.
© Università degli Studi di Roma "La Sapienza" - Piazzale Aldo Moro 5, 00185 Roma