Radiance Fields for Robotic Teleoperation

Maximum Wilder-Smith1, Vaishakh Patil1, Marco Hutter1
1ETH Zürich
Accepted to IROS 2024 [Oral]
Paper arXiv Video ROS Node VR Teleoperation

Leverage Radiance Fields for robotic teleoperation by feeding live ROS data into Radiance Field training, and visualization. Compatible with standard RViz systems or with a novel VR visualization.

Abstract

Radiance field methods such as Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting (3DGS), have revolutionized graphics and novel view synthesis. Their ability to synthesize new viewpoints with photo-realistic quality, as well as capture complex volumetric and specular scenes, makes them an ideal visualization for robotic teleoperation setups. Direct camera teleoperation provides high-fidelity operation at the cost of maneuverability, while reconstruction-based approaches offer controllable scenes with lower fidelity. With this in mind, we propose replacing the traditional reconstruction-visualization components of the robotic teleoperation pipeline with online Radiance Fields, offering highly maneuverable scenes with photorealistic quality.

We present three main contributions to the state-of-the-art:

  1. online training of Radiance Fields using live data from multiple cameras
  2. support for a variety of radiance methods including NeRF and 3DGS
  3. visualization suite for these methods including a virtual reality scene

Furthermore, to enable seamless integration with existing setups, these components were tested with multiple robots in multiple configurations and were displayed using traditional tools as well as the VR headset. The results across methods and robots were compared quantitatively to a baseline of mesh reconstruction, and a user study was conducted to compare the different visualization methods.

Video

Pipeline

Extend the existing ROS teleoperation pipeline to allow robotic data to feed into NerfStudio for Radiance generation, as well as a visualization suite to view the reconstructions.

Robots

We tested our pipeline with three robots: Panda, Anymal, and Alma. Each robot has different configurations and sensors, which allowed us to test the pipeline with a variety of data sources.

Static Arm

Panda

The Franka Panda arm is mounted to a table providing very accurate poses, but limited mobility when capturing a table top scene.

Mobile Base

anymal

The Anymal quadruped is a mobile base which was able to capture a room with a wooden pedistal in the middle, but had noisy poses during locomotion.

Mobile Arm

Alma

The ALMA system has utilizes and arm on a mobile base, providing a good balance of mobility and accuracy to capture a cabinet.

Reconstructions

We compared the reconstructions of the Radiance Fields with a baseline of mesh reconstruction. The Radiance Fields were generated using NeRF and 3D Gaussian Splatting. Three different datasets compare reconstruction of a table top scene, a room captured by the mobile Anymal, and a cabinet captured using the arm on a mobile manipulator.

Ground Truth Voxblox NeRF 3D Gaussian Splat
Panda GT Panda Voxblox Panda NeRF Panda 3DGS
Anymal GT Anymal Voxblox Anymal NeRF Anymal 3DGS
Alma GT Alma Voxblox Alma NeRF Alma 3DGS

Visualization

The Radiance Fields can be visualized in a variety of ways, from traditional RViz to a VR scene.

RViz

For integration with existing ROS teleoperation systems, an RViz plugin can display Radiance Fields along side robot data.

VR Scene

For more immersive interaction, a VR visualization suite can be used to view the Radiance Fields and robots in 3D. The VR scene also displays

BibTeX

@article{wildersmith2024rfteleoperation,
  author    = {Maximum Wilder-Smith, Vaishakh Patil, Marco Hutter},
  title     = {Radiance Fields for Robotic Teleoperation},
  journal   = {arXiv},
  year      = {2024},
}