Commit cd5b7dc8 authored by Guilhem Saurel's avatar Guilhem Saurel
Browse files

Merge branch 'main' into 'main'

amit_icra_22

See merge request !5
parents 4b449c05 e82feb75
Pipeline #17635 passed with stage
in 14 seconds
---
title: "Value learning from trajectory optimization and Sobolev descent: A step toward reinforcement learning with superlinear convergence properties"
subtitle: IEEE ICRA - International Conference on Robotics and Automation, 2022
author:
- Amit Parag ^1, 2^
- Sébastien Kleff ^1,3^
- Léo Saci ^1^
- <a href="https://gepettoweb.laas.fr/index.php/Members/NicolasMansard">Nicolas Mansard</a> ^1,2^
- <a href="https://homepages.laas.fr/ostasse/drupal/">Olivier Stasse</a> ^1,2^
org:
- ^1^ Gepetto Team, LAAS-CNRS, France.
- ^2^ Artificial and Natural Intelligence Toulouse Institute, Toulouse.
- ^3^ New York University, USA
hal: https://hal.archives-ouvertes.fr/hal-03356261
peertube: https://peertube.laas.fr/videos/embed/9a3c5258-e5b7-49a5-a153-02e804a06f65
sourcecode: https://github.com/amitparag/Kuka-arm-DvP
...
## Abstract
The recent successes in deep reinforcement learning largely rely on the capabilities of generating masses of data, which in turn implies the use of a simulator.
In particular, current progress in multi body dynamic simulators are underpinning the implementation of reinforcement learning for end-to-end control of robotic systems.
Yet simulators are mostly considered as black boxes while we have the knowledge to make them produce a richer information.
In this paper, we are proposing to use the derivatives of the simulator to help with the convergence of the learning.
For that, we combine model-based trajectory optimization to produce informative trials using 1st- and 2nd-order simulation derivatives.
These locally-optimal runs give fair estimates of the value function and its derivatives, that we use to accelerate the convergence of the critics using Sobolev learning.
We empirically demonstrate that the algorithm leads to a faster and more accurate estimation of the value function.
The resulting value estimate is used in model-predictive controller as a proxy for shortening the preview horizon.
We believe that it is also a first step toward superlinear reinforcement learning algorithm using simulation derivatives, that we need for end-to-end legged locomotion.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment