SimBenchmark

SimBenchmark provides benchmark results of contact simulation on the state-of-the-art physics engines for various robotic tasks.

Table of Contents

Introduction

As the robots interact with environments by the contact force, accurate and fast simulation of contact dynamics is crucial for model-based control of robots. Notably, the motion of legged robots is highly dependent on the contact force generated by a feet-terrain interaction that drives the entire body. It is natural to pursue the realistic and efficient contact simulation for legged robotics research, which enables the successful control of robot locomotion strategy or end-to-end training.

Many rigid body simulation tools that simulate contact dynamics has been introduced to the robotics society, yet the field is fragmented without strong dominance. For robotics researchers, it is challenging to find and select the best simulator suits to their robotics tasks among numerous options.

In this project, we seek to provide an comprehensive evaluation of the accuracy and the speed of contact simulation on the most widely-used physics engines for various situation scenes from single-bodies with a limited number of contacts to complex articulated robotic systems with PD control input. Our primary goal is to aid robotics researchers to select the best physics engine for their applications.

Contact and Multibody Dynamics of Physics Engines

Solving contact dynamics is NP-hard problem[1] due to its non-convexity and discontinuity. To make the problem tractable, physics engines approximate the contact problem as a relaxed problem that enables using efficient solving methods. Each method has different characteristics, but some of them significantly sacrifice the accuracy of the contact solution that leads to the poor reality of the simulation.

Moreover, physics engine requires efficient multibody dynamics implementation to be used for complex robot systems. For robotic articulated systems that have multiple joints and links, solving the multibody dynamics is considerably expensive as $n$ links have $O(n^3)$ complexity. Most state-of-the-art physics engines, therefore, adopt linear complex algorithms that solve the problem in an efficient manner.

In the benchmark tests, we investigated different characteristics of the contact model and multibody dynamics algorithms that each physics engine applies.

Please read this page for more backgrounds.

List of Physics Engines

We evaluated the most widely-used physics engines for robotics and machine learning applications.[2] The list of the engines is as follows:

• RaiSim (unreleased)
  • What is RaiSim?
• Bullet Physics (Bullet)
• Open Dynamics Engine (ODE)
• Multi-Joint dynamics with Contact (MuJoCo)
• Dynamic Animation and Robotics Toolkit (DART)

You can find more details about each engine and how they were used in our benchmark software here.

Evaluation Metrics

The error of the contact simulation is caused by

1. inaccurate impulse/force solution from a contact solver
2. inherited discretized error from numerical integration.

To evaluate the performance in a fair manner, we have to take into account the whole speed-accuracy curve as proposed in [3]. In the following example of speed-accuracy curve plot, the ideal engine is on the top-right corner, while most of the engines are represented as the curves sloping downwards from left to right. The better engine's curve locates upper-right than others; thus the engine 1 represented by curve 1 is better than the engine 2 represented by curve 2.

Although our lack of knowledge on real physics, we use analytical solution of object trajectory as a reference obtained from Newton's rigid body dynamics and Coulomb's friction cone model when it is applicable: for systems with single rigid body with small number of contacts, we find analytical solution of the object trajectory and investigate numerical errors from simulation.

Besides, it may not be possible to find analytical trajectory for complex systems. In these cases, we investigate generic physical quantities such as total kinetic energy or linear momentum.

Finally, we also observe position level drift by measuring penetration error that is caused by discretization or inaccurate contact model. Apparently it should be zero for rigid body simulation with hard contacts.

More details of benchmark methodology are here.

Test and Results

Tests

• Rolling test: friction model test
• Bouncing test: single-body elastic collision test
• 666 balls test: single-body hard contact test
• Elastic 666 balls test: single-body energy test
• ANYmal PD control test: articulated-robot-system speed test for quadrupedal robot
• ANYmal momentum test: articulated-robot-system momentum test
• ANYmal energy test: articulated-robot-system energy test

Summary of Results

The following table describes the benchmark results assessed by the speed-accuracy curves.

• more + is better
• +: best results
• -: cannot simulate due to inaccurate model or excepted

ODE generally performs well with single-body primitive shaped objects while MuJoCo and DART have benefits on multibody simulations. The multibody system simulation on ODE is very expensive but notably, it outperformed in ANYmal momentum test: although it is significantly inefficient, the maximal coordinate representation has good momentum preservability. Bullet can simulates slip of the object with frictional contacts very accurately and also efficiently.

But, these engines also reveled their limitations:

• the LCP contact model of ODE and DART failed to simulate slip of the object in certain conditions.
• DART's simulation pipeline is not suitable for simulation scenes with many objects.
• Bullet has severe position level drift without the post-solver error correcting method.
• MuJoCo's soft contact model cannot control elasticity of contact.
• MuJoCo has consistent slip which requires additional post-process for legged robot simulation.

RaiSim generally has good performance for both single-body and multibody tasks by the virtue of its distinctive contact solving method and efficient multibody dynamics algorithm implementation. We expect RaiSim to be an good alternative of the current state-of-the-art engines for contact simulation of robotics applications.

Commentary

• Note that this benchmark is done by Dongho Kang and Jemin Hwangho who are the developers of RaiSim.
• If you have any concern, please contact Dongho Kang.

Acknowledgment

Special thanks to Erwin Coumans who is the inventor of Bullet Physics.

References

1. D. Kaufman et al., “Staggered projections for frictional contact in multibody systems,” 2008.
2. S. Ivaldi et al., "Tools for dynamics simulation of robots: a survey based on user feedback," 2014
3. Tom Erez et al., "Simulation Tools for Model-Based Robotics: Comparison of Bullet, Havok, MuJoCo, ODE and PhysX," 2015