EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL - Institut des Systèmes Intelligents et de Robotique Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL

Résumé

Reinforcement learning (RL) in long horizon and sparse reward tasks is notoriously difficult and requires a lot of training steps. A standard solution to speed up the process is to leverage additional reward signals, shaping it to better guide the learning process. In the context of language-conditioned RL, the abstraction and generalisation properties of the language input provide opportunities for more efficient ways of shaping the reward. In this paper, we leverage this idea and propose an automated reward shaping method where the agent extracts auxiliary objectives from the general language goal. These auxiliary objectives use a question generation (QG) and question answering (QA) system: they consist of questions leading the agent to try to reconstruct partial information about the global goal using its own trajectory. When it succeeds, it receives an intrinsic reward proportional to its confidence in its answer. This incentivizes the agent to generate trajectories which unambiguously explain various aspects of the general language goal. Our experimental study shows that this approach, which does not require engineer intervention to design the auxiliary objectives, improves sample efficiency by effectively directing exploration.
Fichier principal
Vignette du fichier
EAGER__Asking_and_Answering_Questions_for_Automatic_Reward_Shaping_in_Language_guided_RL-6.pdf (2.68 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03902423 , version 1 (15-12-2022)

Identifiants

  • HAL Id : hal-03902423 , version 1

Citer

Thomas Carta, Sylvain Lamprier, Pierre-Yves Oudeyer, Olivier Sigaud. EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL. NeurIPS 2022 - Thirty-sixth Conference on Neural Information Processing Systems, Nov 2022, Nouvelle-Orléans, United States. ⟨hal-03902423⟩
52 Consultations
95 Téléchargements

Partager

Gmail Facebook X LinkedIn More