Reinforcement learning (RL) is a popular technique that allows an agent to learn by trial and error while interacting with a dynamic environment. The traditional Reinforcement Learning (RL) approach has been successful in learning and predicting Euclidean robotic manipulation skills such as positions, velocities, and forces. However, in robotics, it is common to encounter non-Euclidean data such as orientation or stiffness, and failing to account for their geometric nature can negatively impact learning accuracy and performance. In this paper, to address this challenge, we propose a novel framework for RL that leverages Riemannian geometry, which we call Geometric Reinforcement Learning (G-RL), to enable agents to learn robotic manipulation skills with non-Euclidean data. Specifically, G-RL utilizes the tangent space in two ways: a tangent space for parameterization and a local tangent space for mapping to a non-Euclidean manifold. The policy is learned in the parameterization tangent space, which remains constant throughout the training. The policy is then transferred to the local tangent space via parallel transport and projected onto the non-Euclidean manifold. The local tangent space changes over time to remain within the neighborhood of the current manifold point, reducing the approximation error. Therefore, by introducing a geometrically grounded pre- and post-processing step into the traditional RL pipeline, our G-RL framework enables several model-free algorithms designed for Euclidean space to learn from non-Euclidean data without modifications. Experimental results, obtained both in simulation and on a real robot, support our hypothesis that G-RL is more accurate and converges to a better solution than approximating non-Euclidean data.

Geometric Reinforcement Learning for Robotic Manipulation / Alhousani, Naseem; Saveriano, Matteo; Sevinc, Ibrahim; Abdulkuddus, Talha; Kose, Hatice; Abu-Dakka, Fares J.. - In: IEEE ACCESS. - ISSN 2169-3536. - 11:(2023), pp. 111492-111505. [10.1109/access.2023.3322654]

Geometric Reinforcement Learning for Robotic Manipulation

Saveriano, Matteo
Secondo
;
2023-01-01

Abstract

Reinforcement learning (RL) is a popular technique that allows an agent to learn by trial and error while interacting with a dynamic environment. The traditional Reinforcement Learning (RL) approach has been successful in learning and predicting Euclidean robotic manipulation skills such as positions, velocities, and forces. However, in robotics, it is common to encounter non-Euclidean data such as orientation or stiffness, and failing to account for their geometric nature can negatively impact learning accuracy and performance. In this paper, to address this challenge, we propose a novel framework for RL that leverages Riemannian geometry, which we call Geometric Reinforcement Learning (G-RL), to enable agents to learn robotic manipulation skills with non-Euclidean data. Specifically, G-RL utilizes the tangent space in two ways: a tangent space for parameterization and a local tangent space for mapping to a non-Euclidean manifold. The policy is learned in the parameterization tangent space, which remains constant throughout the training. The policy is then transferred to the local tangent space via parallel transport and projected onto the non-Euclidean manifold. The local tangent space changes over time to remain within the neighborhood of the current manifold point, reducing the approximation error. Therefore, by introducing a geometrically grounded pre- and post-processing step into the traditional RL pipeline, our G-RL framework enables several model-free algorithms designed for Euclidean space to learn from non-Euclidean data without modifications. Experimental results, obtained both in simulation and on a real robot, support our hypothesis that G-RL is more accurate and converges to a better solution than approximating non-Euclidean data.
2023
Alhousani, Naseem; Saveriano, Matteo; Sevinc, Ibrahim; Abdulkuddus, Talha; Kose, Hatice; Abu-Dakka, Fares J.
Geometric Reinforcement Learning for Robotic Manipulation / Alhousani, Naseem; Saveriano, Matteo; Sevinc, Ibrahim; Abdulkuddus, Talha; Kose, Hatice; Abu-Dakka, Fares J.. - In: IEEE ACCESS. - ISSN 2169-3536. - 11:(2023), pp. 111492-111505. [10.1109/access.2023.3322654]
File in questo prodotto:
File Dimensione Formato  
Geometric_Reinforcement_Learning_for_Robotic_Manipulation.pdf

accesso aperto

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Creative commons
Dimensione 5.35 MB
Formato Adobe PDF
5.35 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11572/403989
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 1
  • ???jsp.display-item.citation.isi??? 0
social impact