Terminal
Terminal/Skill
Covers headless MuJoCo image rendering with cameras, RGB, depth and segmentation, for datasets and visual observations.
By Google
Works with
npx skills add google-deepmind/mujoco --skill mujoco-rendering[!TIP]
Related skills:
- mujoco-python — Core physics simulation, stepping, and state management.
- mujoco-studio — Interactive simulation, Studio viewer apps, and plugin architecture.
- mujoco-gui — Dear ImGui UI design and custom controls.
MuJoCo provides two distinct graphics workflows:
mujoco.Renderer / mjrContext):
Focused on capturing image buffers (RGB, depth, object segmentation) without
opening a desktop window. Ideal for batch dataset collection, reinforcement
learning observations, and automated visual regression testing.mujoco.experimental.studio / mujoco.viewer):
Interactive desktop or web-streamed applications designed for real-time
inspection, camera manipulation, and Dear ImGui control panels.MUJOCO_GLOn headless Linux systems (servers, containers), configure the OpenGL context
provider via the MUJOCO_GL environment variable before importing mujoco:
import os
os.environ['MUJOCO_GL'] = 'egl' # Hardware-accelerated offscreen on NVIDIA GPUs
# os.environ['MUJOCO_GL'] = 'osmesa' # Portable CPU software rasterization
import mujoco # Must import AFTER setting MUJOCO_GL
Or from the command line:
MUJOCO_GL=egl python3 my_script.py
mujoco.Renderer manages GL context creation, scene geometry synchronization,
and buffer readback. Always use it as a context manager:
import mujoco
import numpy as np
model = mujoco.MjModel.from_xml_path('scene.xml')
data = mujoco.MjData(model)
with mujoco.Renderer(model, height=480, width=640) as renderer:
mujoco.mj_forward(model, data)
# 1. RGB Rendering
renderer.update_scene(data, camera='overhead_cam')
rgb = renderer.render() # Shape: (480, 640, 3), dtype: uint8
# 2. Metric Depth Rendering (distance in meters)
renderer.enable_depth_rendering()
renderer.update_scene(data, camera='overhead_cam')
depth = renderer.render() # Shape: (480, 640), dtype: float32
renderer.disable_depth_rendering()
# 3. Semantic Segmentation Rendering
renderer.enable_segmentation_rendering()
renderer.update_scene(data, camera='overhead_cam')
seg = renderer.render() # Shape: (480, 640, 2), dtype: int32
# seg[:, :, 0] = object ID (-1 for background)
# seg[:, :, 1] = object type (mjtObj enum, e.g. mjOBJ_GEOM)
renderer.disable_segmentation_rendering()
Creating a Renderer allocates GPU framebuffers. In RL or evaluation loops,
create the renderer once outside the loop:
with mujoco.Renderer(model, height=84, width=84) as renderer:
for step in range(1000):
mujoco.mj_step(model, data)
if step % render_interval == 0:
renderer.update_scene(data, camera='agent_cam')
# Call .copy() because the renderer's internal buffer is reused
frame = renderer.render().copy()
In C++, offscreen rendering is performed using the mjvScene and mjrContext
APIs:
#include <mujoco/mujoco.h>
// 1. Initialize visualization structures
mjvCamera cam;
mjv_defaultCamera(&cam);
cam.type = mjCAMERA_FIXED;
cam.fixedcamid = mj_name2id(m, mjOBJ_CAMERA, "overhead_cam");
mjvOption opt;
mjv_defaultOption(&opt);
mjvScene scn;
mjv_makeScene(m, &scn, 2000); // Buffer up to 2000 geoms
// 2. Initialize offscreen rendering context
mjrContext con;
mjr_defaultContext(&con);
mjr_makeContext(m, &con, mjFONTSCALE_100);
// 3. Render frame
mjrRect viewport = {0, 0, 640, 480};
mjv_updateScene(m, d, &opt, nullptr, &cam, mjCAT_ALL, &scn);
mjr_render(viewport, &scn, &con);
// 4. Read pixels into user buffers
unsigned char rgb[640 * 480 * 3];
float depth[640 * 480];
mjr_readPixels(rgb, depth, viewport, &con);
// 5. Clean up
mjr_freeContext(&con);
mjv_freeScene(&scn);
MuJoCo cameras look down their negative Z-axis:
# Fixed camera pointing down at ground plane
cam = spec.worldbody.add_camera(
name='overhead',
pos=[0, 0, 3.0],
quat=[0.707, 0.707, 0, 0], # 90 deg rotation around X
fovy=60,
)
# Camera attached to robot wrist
wrist_cam = wrist_body.add_camera(
name='wrist_cam',
pos=[0.05, 0, 0],
xyaxes=[0, -1, 0, 0, 0, -1],
fovy=75,
)
MjvCamera)For custom interactive or orbital viewpoints:
cam = mujoco.MjvCamera()
cam.type = mujoco.mjtCamera.mjCAMERA_FREE
cam.lookat[:] = [0, 0, 0.5]
cam.distance = 2.5
cam.azimuth = 90.0
cam.elevation = -20.0
renderer.update_scene(data, camera=cam)
The offscreen rendering resolution cannot exceed the dimensions pre-allocated in the model. Adjust the buffer limit in the spec before compilation:
spec.visual.global_.offwidth = 1920
spec.visual.global_.offheight = 1080
model = spec.compile()
<visual> Block RuleMuJoCo uses only the last <visual> block defined in an XML file. Multiple
visual blocks will silently overwrite earlier settings:
<!-- ❌ WRONG — second visual block silently resets offwidth/offheight to default -->
<visual><global offwidth="1920" offheight="1080"/></visual>
<visual><headlight ambient="0.3 0.3 0.3"/></visual>
<!-- ✅ RIGHT — consolidate into one visual block -->
<visual>
<global offwidth="1920" offheight="1080"/>
<headlight ambient="0.3 0.3 0.3"/>
</visual>
Control rendering layers via MjvOption:
opt = mujoco.MjvOption()
opt.geomgroup[:] = False # Hide all groups
opt.geomgroup[0] = True # Show primary geometry (default group)
opt.geomgroup[1] = True # Show secondary visuals
# Toggle visual debugging flags
opt.flags[mujoco.mjtVisFlag.mjVIS_CONTACTFORCE] = True
opt.flags[mujoco.mjtVisFlag.mjVIS_TRANSPARENT] = True
renderer.update_scene(data, camera='overhead', scene_option=opt)
MuJoCo supports the Filament physically-based rendering (PBR) engine
experimentally for advanced lighting, soft shadows, and materials. In Python,
Filament renderer bindings are available under mujoco.rendering.filament.
with block. Failing
to free renderers in loops exhausts GL contexts.mjr_freeContext must be called before the
underlying OpenGL context is destroyed.renderer.render() returns an internal buffer view. Call
.copy() if storing frames in a list.camera='camera_name' or the integer ID.
Avoid passing raw string IDs that are not present in the model.