Added arm benchmark + improved running process of benchmarks
This PR mainly tackles #259 (closed). Additionally it included the following improvements in the benchmark
- Added
ShootingProblem.calc
andShootingProblem.calcDiff
benchmarks for all the cases. - Added pure Python benchmark for the talos arm problem
- Improved the running process, now we can do it like this:
make benchmark-unicycle INPUT=100 # 100 is the number of trials
make benchmark-lqr INPUT=100 # 100 is the number of trials
make benchmark-arm-manipulator INPUT=100 # 100 is the number of trials
make benchmark-quadrupedal-gaits INPUT='100 walk' # walk is the gait, we have other options (trot, pace, bound and jump)
make benchmark-bipedal-walk INPUT=100 # 100 is the number of trials
Additionally, I don't see any breaking of the determinism (neither c++ and Python bindings code) using multithreading as reported #256 (closed). Indeed, I notice an improvement in the computation time of calcDiff. These are the results
C++:
DDP.solve [ms]: 1.59551 (1.55648-1.94576)
ShootingProblem.calc [ms]: 0.46114 (0.43442-0.676178)
ShootingProblem.calcDiff [ms]: 0.560603 (0.497396-0.984347)
Python bindings:
DDP.solve [ms]: 1.69171571732 (1.63292884827, 2.27212905884)
ShootingProblem.calc [ms]: 0.49816608429 (0.463008880615, 0.734090805054)
ShootingProblem.calcDiff [ms]: 0.578708648682 (0.531911849976, 1.02019309998)
and without multithreading we compute calcDiff
around 2.5ms.
@wxmerkt could you run the arm-manipulation benchmark in your PC?
@proyan it's working on the c++ benchmark for quadrupedal-gaits
Merge request reports
Activity
changed milestone to %Crocoddyl c++ implementation
added efficiency label
enabled an automatic merge when the pipeline for f57fad10 succeeds
mentioned in issue #256 (closed)
- Resolved by Carlos Mastalli
Benchmark arm manipulator:
Single core, vectorisation OFF:
C++: DDP.solve [ms]: 2.41596 (2.35941-3.28919) ShootingProblem.calc [ms]: 0.383199 (0.366688-0.688599) ShootingProblem.calcDiff [ms]: 1.60558 (1.51536-2.78337) Python bindings: DDP.solve [ms]: 2.49390125275 (2.40015983582, 3.60989570618) ShootingProblem.calc [ms]: 0.420245409012 (0.399827957153, 0.806093215942) ShootingProblem.calcDiff [ms]: 1.60745024681 (1.53994560242, 2.4881362915)
Multi-threading (8 cores), vectorisation ON:
C++: DDP.solve [ms]: 1.19437 (1.16402-2.00468) ShootingProblem.calc [ms]: 0.371434 (0.363886-0.558771) ShootingProblem.calcDiff [ms]: 0.263641 (0.238182-0.519628) Python bindings: DDP.solve [ms]: 1.23634848595 (1.20091438293, 2.17580795288) ShootingProblem.calc [ms]: 0.408774662018 (0.392913818359, 0.766038894653) ShootingProblem.calcDiff [ms]: 0.29428730011 (0.269889831543, 1.18613243103)
Via the Python bindings, I still see the non-determinism as reported in #256 (closed):
[DDP] Mean iter 29.1 +/- 8.757282683572571 [FDDP] Mean iter 31.2 +/- 9.88736567544662
mentioned in commit e4b352fc