Multi-Objective Evolutionary Hindsight Experience Replay for Robot Manipulation Tasks

Reinforcement learning (RL) algorithms often face challenges in efficiently learning effective policies for sparse-reward multi-goal robot manipulation tasks, thus requiring a vast amount of experiences. The state-of-the-art algorithm in the field, Hindsight Experience Replay (HER), addresses this issue by using failed trajectories and replacing the desired goal with hindsight goals. However, HER performs poorly when the desired goal is distant from the initial state. To address this limitation, Hindsight Goal Generation (HGG) has been proposed, which generates a curriculum of goals from already visited states. This curriculum generation is based on a single objective, and does not take obstacles into account. Here, we make a step forward by proposing Multi-Objective Evolutionary Hindsight Experience Replay (MOEHER), a novel curriculum RL algorithm that reformulates curriculum generation considering multiple objectives and obstacles. MOEHER utilizes NSGA-II to generate a curriculum that is optimized w.r.t. four objectives, namely the Q-function, the goal-proximity function, and two distance metrics, while simultaneously satisfying constraints on the obstacles. We evaluate MOEHER on four different sparse-reward robot manipulation tasks, with and without obstacles, and compare it with HER and HGG. The results demonstrate that MOEHER surpasses or performs on par with these methods on the tested tasks.