Commit e82feb75 authored by Amit Parag's avatar Amit Parag
Browse files

amit icra 22

parent 4b449c05
title: "Value learning from trajectory optimization and Sobolev descent: A step toward reinforcement learning with superlinear convergence properties"
subtitle: IEEE ICRA - International Conference on Robotics and Automation, 2022
- Amit Parag ^1, 2^
- Sébastien Kleff ^1,3^
- Léo Saci ^1^
- <a href="">Nicolas Mansard</a> ^1,2^
- <a href="">Olivier Stasse</a> ^1,2^
- ^1^ Gepetto Team, LAAS-CNRS, France.
- ^2^ Artificial and Natural Intelligence Toulouse Institute, Toulouse.
- ^3^ New York University, USA
## Abstract
The recent successes in deep reinforcement learning largely rely on the capabilities of generating masses of data, which in turn implies the use of a simulator.
In particular, current progress in multi body dynamic simulators are underpinning the implementation of reinforcement learning for end-to-end control of robotic systems.
Yet simulators are mostly considered as black boxes while we have the knowledge to make them produce a richer information.
In this paper, we are proposing to use the derivatives of the simulator to help with the convergence of the learning.
For that, we combine model-based trajectory optimization to produce informative trials using 1st- and 2nd-order simulation derivatives.
These locally-optimal runs give fair estimates of the value function and its derivatives, that we use to accelerate the convergence of the critics using Sobolev learning.
We empirically demonstrate that the algorithm leads to a faster and more accurate estimation of the value function.
The resulting value estimate is used in model-predictive controller as a proxy for shortening the preview horizon.
We believe that it is also a first step toward superlinear reinforcement learning algorithm using simulation derivatives, that we need for end-to-end legged locomotion.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment