Terminal
Terminal/Skill
Covers MuJoCo Studio viewers, including sim and viewer threads, messaging, and writing viewer plugins.
By Google
Works with
npx skills add google-deepmind/mujoco --skill mujoco-studio"Studio" is the term used to refer to all the components used to build
visualizations that replace the legacy simulate tools. The default user
interface is known as the Studio Viewer App, which can be customized and
extended with Studio Plugins.
Under the hood, Studio modernizes the interactive visualization stack using Dear ImGui for UI, Google Filament (PBR with Vulkan, OpenGL, Metal, and WebGL backends) for rendering, and browser-based remote streaming out of the box via WebViewer (NetImgui WASM).
A core goal of Studio is to make plugins composable and reusable across workflows:
[!TIP]
Related skills:
- mujoco-gui — Dear ImGui agent development skill with best practices for building custom UI plugins across C++ and Python.
- mujoco-python — Core simulation lifecycle (
MjSpec→MjModel→MjData), named access, and spatial math.
MuJoCo Studio modernizes the interactive visualization stack:
| Feature | Simulate / mujoco.viewer (Legacy) | Studio / mujoco.experimental.studio |
|---|---|---|
| GUI Framework | Custom fixed MuJoCo UI (mjui) | Dear ImGui |
| Data Plotting | Built-in 2D line graphs (mjr_figure) | ImPlot |
| Render Backend | Classic OpenGL only (mjrContext) | Filament (PBR, Vulkan, OpenGL, Metal, WebGL) |
| Web & Remote | None | Web viewer with NetImgui (ui/sim/render data streaming over HTTP/WS to a WASM client) |
| Extensibility | Hardcoded UI panels and keybinds | Plugin Architecture in both C++ and Python |
| Threading | Sim/Viewer on separate thread | Python: Decoupled Sim/Viewer threads; C++ App: Single-threaded |
| Synchronization | Sim/Viewer share model/data with locking | Python: Message passing (Snapshots & Events); C++: Shared in main loop |
The Python Studio API introduced in viewer_protocol.py and viewer_handle.py
replaces the monolithic, synchronous viewer loop with a thread-decoupled,
message-driven plugin architecture:
┌────────────────────────────────────────────────────────┐
│ Simulation Thread (Caller) │
│ - Owns MjModel and MjData │
│ - Runs physics loop, RL policies, or MPC planners │
│ - Executes sim_plugins (e.g. StepControl, reset) │
│ - Drives simulation via handle.sync(model, data) │
└────────────────────────────────────────────────────────┘
▲
│ Queued Events & Snapshots
│ (SimEndpoint <-> ViewerEndpoint)
▼
┌────────────────────────────────────────────────────────┐
│ Viewer Thread (or Web Client) │
│ - Owns Window, Renderer, Camera, VisOptions, Perturb │
│ - Owns local copy of MjModel and MjData for rendering │
│ - Executes viewer_plugins (ViewerApp, custom UI) │
│ - Dispatches lifecycle events (UpdateEvent, BuildGuiEvent) │
└────────────────────────────────────────────────────────┘
launch_passive): The simulation thread
and the viewer thread run independently. The simulation communicates with
the viewer non-blockingly through endpoints (SimEndpoint ↔ ViewerEndpoint)
using queued Events and latest-value Snapshots.plugin_registry.py): State and UI
behaviors are decoupled into standalone plugin classes decorated with
@messages.handler(priority=...).
sim_plugins): Handle simulation-side events
(e.g. physics reset, step control, perturbations, model reloading,
policy changes).viewer_plugins): Handle rendering and UI lifecycle
events (e.g. viewer initialization, model reloading, per-frame visual
updates, GUI construction).python/mujoco/experimental/studio/):
python/mujoco/experimental/studio/viewer.py —
Minimal baseline with standard GUI (viewer_app.ViewerApp).python/mujoco/experimental/studio/sample/ghost.py —
Demonstrates a viewer plugin (GhostRenderer) injecting extra geoms via
extra_geoms and custom Dear ImGui controls on BuildGuiEvent.python/mujoco/experimental/studio/sample/implot.py —
Demonstrates real-time responsive charting with Dear ImGui and ImPlot
via BodyInspector.messages.py)All communication between the simulation and viewer runs over two types of
channels. Convention: custom message classes derive from Snapshot or
Event and end in *Snapshot or *Event to indicate their delivery semantics.
Snapshot (Lossy, Latest-Wins): Each new snapshot replaces the previous
unread value of the same type. Ideal for high-frequency continuous state
where intermediate drops are harmless.
StateSnapshot — transports the high-frequency
MuJoCo simulation state vector (state, state_sig) from simulation to
viewer.Event (Lossless, Reliable FIFO Queue): Guaranteed in-order,
exactly-once delivery without dropping. Used for discrete actions,
structural reloads, and user inputs.
ModelEvent — transports a compiled MjModel (and
optional path) when a model is initially loaded or recompiled.Lifecycle events are standard framework events dispatched by the runner loops to plugins to hook into rendering, UI layout, and physics execution. Each list below is ordered chronologically from startup through per-iteration execution to shutdown:
run_viewer_loop to viewer_plugins):
ViewerInitEvent (viewer_protocol.ViewerInitEvent) — dispatched once
on viewer initialization so stateful plugins can cache the viewer
instance (event.viewer).ModelEvent — received when a model is initially loaded or hot-swapped;
Viewer deep-copies viewer.model, creates viewer.data, and runs
mj_forward.PostModelEvent — dispatched locally by Viewer immediately after
ModelEvent has initialized viewer.model and viewer.data. Subscribe
to PostModelEvent (rather than ModelEvent) when resetting
viewer-plugin state on model changes so viewer.model and viewer.data
are already up to date.UpdateEvent — dispatched every frame (after incoming sim events and
snapshots are applied) to update visuals, camera, perturbations, and
transient geoms.BuildGuiEvent — dispatched every frame immediately after UpdateEvent
to construct and lay out Dear ImGui / ImPlot widgets.ExitEvent — dispatched once when the viewer shuts down, whether the
exit came from the sim side or from the window being closed. Handle it
to release resources (threads, pools, GPU handles).sim_plugins):
SimInitEvent (viewer_handle.SimInitEvent) — dispatched once by
ViewerHandle.__init__ so stateful plugins can cache the handle
(event.handle) instead of the launcher having to hand it to them.ModelEvent — dispatched when the initial model is set in
ViewerHandle.__init__ or when a model is hot-swapped via
handle.set_model() (or received from the viewer); ViewerHandle
installs handle.model and handle.data, runs mj_forward, and
forwards the model to the viewer.PostModelEvent — dispatched locally by ViewerHandle immediately
after ModelEvent has installed handle.model and handle.data and
run mj_forward. Subscribe to PostModelEvent (rather than
ModelEvent) when resetting sim-plugin state on model changes.PreStepEvent — dispatched by ViewerHandle.sync on every call (after
incoming viewer events and snapshots are applied) so plugins can prepare
the simulation before physics advances (e.g. writing actuator controls,
applying forces, or hot-swapping models via handle.set_model).StepEvent — dispatched by ViewerHandle.sync immediately after
PreStepEvent so sim-side stepping plugins (such as StepControl) can
advance the physics.PostStepEvent — dispatched by ViewerHandle.sync immediately after
StepEvent (before StateSnapshot is broadcast) so observer/recorder
plugins can inspect or record the resulting simulation state.ExitEvent — dispatched once by ViewerHandle.close(), which the
with block calls on every exit path (normal exit, KeyboardInterrupt,
an exception, or the viewer dying). Handle it to release resources.[!IMPORTANT]
Do not return
Truefrom anExitEventhandler: teardown must reach every registered plugin, so no handler should consume the event.
ViewerAppInitEvent (viewer_app.ViewerAppInitEvent) — dispatched by
the ViewerApp viewer plugin on startup so other viewer plugins can
cache application state and window handles (event.viewer_app).@messages.handler DecoratorPlugins subscribe to messages by annotating methods with @messages.handler:
from mujoco.experimental.studio import messages
class MyPlugin:
@messages.handler
def on_build_gui(self, event: messages.BuildGuiEvent) -> bool | None:
# Build Dear ImGui elements here
return False # Return True to consume and halt lower-priority handlers
messages.Priority)CRITICAL (1000) — Core framework state synchronization and dockspace setup.USER (100, default) — User plugins and custom UI tools.LIBRARY (10) — Standard framework extensions and helpers.INTERNAL (1) — Core framework operations and baseline fallbacks.There is an intentional distinction between lifecycle phase events (Pre* /
Post*) and handler priorities (messages.Priority):
Priority order, it is tempting to sequence work before or
after a framework action by subscribing to the main action event and
fiddling with priority=... — for example, subscribing to StepEvent with
a priority higher or lower than StepControl (Priority.INTERNAL) to run
code before or after physics stepping, or subscribing to ModelEvent with a
custom priority to run before or after ViewerHandle / Viewer swaps the
model.Pre* / Post* events instead: Using priorities
to order distinct lifecycle stages couples your plugin to internal priority
choices of other plugins and can expose stale references (for instance, a
high-priority ModelEvent handler runs before ViewerHandle or Viewer
has created the new MjData and run mj_forward, and a StepEvent handler
that calls handle.set_model() mid-dispatch leaves the in-flight
StepEvent holding the old model and data).@messages.handler decorator:
ModelEvent installs the new model
and runs mj_forward, subscribe to PostModelEvent so model and
data are guaranteed to be initialized and forwarded.PreStepEvent
(to write controls, apply forces, or hot-swap models via
handle.set_model), StepEvent (only when your plugin is itself the
physics stepper), or PostStepEvent (to observe or record the stepped
state).UpdateEvent for
per-frame state updates and BuildGuiEvent for Dear ImGui widget
submission.messages.Priority: Reserve priority=... strictly for
cases where multiple handlers target the exact same event and phase and
one handler needs to override or consume (return True) the event before a
fallback handler runs.[!IMPORTANT]
If a handler returns
True, the message is marked as consumed and will not be delivered to lower-priority handlers. ReturnFalseorNoneif downstream plugins or default handlers also need the message.
launch_web, launch_native & launch_passiveStudio provides three runner modules that launch the viewer and execute
the simulation sync loop via .run(...):
launch_web.run
(python/mujoco/experimental/studio/launch_web.py):
Spawns a WebViewer in a background daemon thread and runs the simulation
loop. This is usually the most practical choice. Its major advantage is
that the host Python simulation script has no native graphics
dependencies (e.g., no desktop display server, X11/Wayland, or Filament
libraries required on the host): the host executes physics and computes Dear
ImGui layout in a headless context, while streaming draw commands (via
NetImgui) and state snapshots over WebSockets to a WebAssembly client
running in the user's browser. This makes it ideal for headless servers,
containers, and remote development.launch_native.run
(python/mujoco/experimental/studio/launch_native.py):
Spawns a local desktop window with native Filament graphics (Vulkan or
OpenGL) and runs the simulation loop.launch_passive.run
(python/mujoco/experimental/studio/launch_passive.py):
A unified wrapper that inspects config.gfx and automatically delegates to
launch_web (if config.gfx in ('web', 'webgl')) or launch_native.Unless you specifically need your application to switch dynamically between
native desktop and web streaming, use just one of these directly. In most
workflows, launch_web.run is recommended.
For a full reference implementation, see
python/mujoco/experimental/studio/viewer.py.
All three modules share the same .run(...) interface:
from mujoco.experimental.studio import launch_web # Or launch_native / launch_passive
from mujoco.experimental.studio import step_control
from mujoco.experimental.studio import viewer_app
from mujoco.experimental.studio import viewer_protocol
config = viewer_protocol.ViewerConfig(title='Studio')
launch_web.run(
config,
model=model,
data=data,
viewer_plugins=[viewer_app.ViewerApp()],
sim_plugins=[step_control.StepControl()],
)
[!WARNING]
Stepping and pacing are plugin responsibilities. Including
step_control.StepControl()insim_pluginsprovides standard real-time-paced CPU stepping. If you omitStepControl()(for example, when running GPU rollouts or custom RL loops),handle.sync()will not advance physics or sleep, so your stepping plugin must manage its own pacing to avoid busy-spinning.
launch_web.launch, launch_native.launch, launch_passive.launch)When attaching a viewer to an external loop that you do not want to invert into
a StepEvent plugin (such as an existing RL training loop, trajectory replay
script, or unit test), each module also provides a non-blocking .launch(...)
context manager returning a ViewerHandle:
with launch_web.launch(
config,
viewer_plugins=[viewer_app.ViewerApp()],
sim_plugins=[step_control.StepControl()],
) as handle:
while handle.is_running():
model, data = handle.sync(model, data)
Explore python/mujoco/experimental/studio/sample/
for actively maintained reference implementations.
Viewer UI Plugins (ImGui & ImPlot):
See python/mujoco/experimental/studio/sample/implot.py
for a complete implementation of BodyInspector, demonstrating how to
receive ViewerAppInitEvent to cache app references, draw responsive ImPlot
charts on BuildGuiEvent, and use persistent window IDs (###) to preserve
docking and window state. Implement plugins as one or two ImGui windows so
they can dock directly into the Studio Viewer App dockspaces. (See the
mujoco-gui skill for Studio GUI plugin best practices
across C++ and Python using ImGui and extensions like ImPlot.)
Visual Overlays & Geoms:
See python/mujoco/experimental/studio/sample/ghost.py
for GhostRenderer, demonstrating how to subscribe to UpdateEvent and
populate viewer.extra_geoms with transient mjvGeom instances without
modifying the compiled model.
[!WARNING]
Upcoming API Migration for Extra Geoms: The
viewer.extra_geomsmechanism insample/ghost.pyis an interim solution and will soon be replaced by the direct scene/geom API frompython/mujoco/rendering/filament/renderer.py. Prefer referencing that upcoming renderer API when architecting new custom visual overlays.
[!WARNING]
Split Sim/Viewer Plugin Architecture (Asset Browser Example): Avoid monolithic designs where viewer-side UI directly accesses or mutates mutable model specifications (
MjSpec).In an interactive asset browser plugin, responsibilities should be cleanly split:
AssetBrowserGui(viewer plugin): Renders the asset browser UI and emits a custom event (e.g.AssetDropEvent) carrying asset data and target locations.AssetBrowserSim(sim plugin): ReceivesAssetDropEventon the simulation side (which owns theMjSpec), performs spec attachment, recompiles the model, and broadcastsModelEvent(model=new_model)back to the viewer.
In C++, MuJoCo Studio is structured around mujoco::studio::App and launched
via mujoco::studio::LaunchStudio.
[!NOTE]
Architecture Convergence Note: Unlike the Python Studio API, the current C++ Studio application is single-threaded (the main loop runs UI, rendering, and physics sequentially on a single thread). The Python and C++ plugin architectures will converge in future updates to share a unified message-driven and lifecycle design.
mujoco::platform::RegisterPlugin)C++ extensions define plugins using structs declared in
src/experimental/studio/ux/plugin.h:
| Plugin Struct | Role | Key Callbacks |
|---|---|---|
GuiPlugin | Creates an ImGui window listed in the "Plugins" menu | update(GuiPlugin* self) |
ScenePlugin | Adds custom mjvGeom elements to the mjvScene | enhance_scene(self, model, data, scene) |
ModelPlugin | Hooks into model loading and physics updates | get_model_to_load, pre_step, post_step, do_update |
KeyHandlerPlugin | Binds keyboard shortcuts via ImGui key chords | key_chord, on_key_pressed(self) |
SpecEditorPlugin | Edits mjSpec with pre/post-recompile triggers | pre_compile(self, spec, m, d, cam), post_compile |
Plugins can be registered in two ways:
mjPLUGIN_LIB_INIT): Plugins can be
authored in self-contained translation units using the mjPLUGIN_LIB_INIT
macro (from #include <mujoco/mjplugin.h>). This defines a static
constructor that calls mujoco::platform::RegisterPlugin on library load
before main() runs. Simply linking the plugin's library into the Studio
binary registers the plugin without having to modify main().#include "imgui.h"
#include <mujoco/mjplugin.h>
#include "src/experimental/studio/ux/plugin.h"
namespace {
void MyGuiUpdate(mujoco::platform::GuiPlugin* self) {
ImGui::TextUnformatted("Custom Tool Window");
}
} // namespace
mjPLUGIN_LIB_INIT(my_tool_plugin) {
mujoco::platform::GuiPlugin gui;
gui.name = "My Tool";
gui.active = true;
gui.update = MyGuiUpdate;
mujoco::platform::RegisterPlugin(gui);
}
main(): When writing a standalone
application or custom launcher entrypoint, plugins can also be registered
explicitly before calling LaunchStudio:#include "absl/flags/parse.h"
#include "imgui.h"
#include "src/experimental/studio/launcher.h"
#include "src/experimental/studio/ux/plugin.h"
void MyGuiUpdate(mujoco::platform::GuiPlugin* self) {
ImGui::TextUnformatted("Custom Tool Window");
}
int main(int argc, char** argv) {
absl::ParseCommandLine(argc, argv);
// Register plugins explicitly before LaunchStudio
mujoco::platform::GuiPlugin gui;
gui.name = "My Tool";
gui.active = true;
gui.update = MyGuiUpdate;
mujoco::platform::RegisterPlugin(gui);
mujoco::studio::LauncherConfig config;
config.title = "My Project Studio";
config.model_file = "path/to/robot.xml";
return mujoco::studio::LaunchStudio(argc, argv, config);
}
Graphics backends are configured via ViewerConfig.gfx using values from the
GFX_MODES tuple in viewer_protocol.py (matching platform::GraphicsMode in
C++):
NativeViewer modes: opengl (default Filament OpenGL), vulkan
(Filament Vulkan), classic (legacy OpenGL), and software/headless modes
(classic_headless, opengl_headless, opengl_software,
vulkan_software).WebViewer modes: web / webgl.[!TIP]
Decoupling Visualization Dependencies with WebViewer: Users running simulations on remote workstations, cloud VMs, or Docker containers may prefer launching via
WebViewer(gfx='web'). This completely prevents the simulation binary from linking or loading heavy graphical and windowing dependencies (Filament, SDL2, Vulkan, X11): the host only executes physics and a headless Dear ImGui context, while all 3D scene rendering and GPU rasterization happen inside the client's web browser via WebAssembly (WASM).
┌────────────────────────────────────────────────────────┐
│ Simulation Thread (Host) │
│ - Runs physics loop & sim_plugins │
│ - Pushes state snapshots & receives events │
└────────────────────────────────────────────────────────┘
▲ Endpoints (IPC / Queues)
│ (SimEndpoint ↔ ViewerEndpoint)
▼
┌────────────────────────────────────────────────────────┐
│ WebViewer Thread (Host Server) │
│ - Runs viewer_plugins in headless Dear ImGui context │
│ - Serves single HTTP port (default 8080) │
│ - Streams UI via NetImgui (/ui) & state (/state) │
└────────────────────────────────────────────────────────┘
▲ WebSocket (HTTP / Port 8080)
│ Streamed draw commands & input
▼
┌────────────────────────────────────────────────────────┐
│ Browser Client (WASM) │
│ - WebAssembly client running in user's browser │
│ - Renders 3D scene with Filament WebGL / WebGPU │
│ - Overlays streamed ImGui draw commands │
│ - Forwards user mouse & keyboard events to server │
└────────────────────────────────────────────────────────┘
/ui)./state) is transmitted primarily
via the compiled binary .mjb for the model and the raw MuJoCo state,
accompanied by a small payload of extra data covering visual elements
not represented in model/state (such as camera transforms, perturb
forces, and transient geoms).8080): the HTML/WASM client application, the /ui NetImgui stream, and
the /state WebSocket stream.In Python, MuJoCo Studio uses a strictly decoupled, multi-threaded architecture
(whereas the C++ Studio app is single-threaded). The simulation thread and the
viewer thread communicate exclusively via thread-safe message channels
(Snapshot and Event, via handle.send_to_viewer(...) and
viewer.send_to_sim(...)). Direct concurrent access causes data races and
rendering artifacts.
As a direct consequence of this principle:
viewer.camera, viewer.vis_options, or viewer-side model
copies).The outer simulation runner loop:
while handle.is_running():
model, data = handle.sync(model, data)
is standardized boilerplate intended to remain identical across all Studio
launchers (launch_passive, web streaming, native C++). Do not insert
domain-specific simulation logic, direct physics stepping, or ad-hoc state
mutations into this loop. If your application needs custom behavior (e.g. custom
stepping, perturbation forces, sensor logging, or RL resets):
StepEvent or custom messages.Event to the simulation thread rather than
mutating state directly in the outer loop.model, data = handle.sync(model, data): Unlike legacy
mujoco.viewer.Handle.sync() (which took no arguments and returned None),
Studio's ViewerHandle.sync(model, data) expects the current model/data and
returns an updated (model, data) tuple. When plugins or user actions
dynamically reload models or attach specs (e.g. via ModelEvent or asset
drops), handle.sync() updates internal references and returns the newly
instantiated model and data. All Studio callers in the codebase reassign
model, data = handle.sync(model, data); failing to capture the return
values leaves caller loops stepping a stale physics model while the viewer
renders the new one.[!NOTE] Future Evolution of Sim Plugins & Handle Access: Strictly keeping the runner loop boilerplate-free might not be fully achievable in all situations right now (for example, if sim plugins need direct access to the handle or mechanisms not yet exposed via existing lifecycle events). We will monitor how launcher boilerplate propagates and evolve the event and plugin APIs accordingly. In the meantime, try to avoid inserting simulation logic into the loop and consider carefully whether it should be handled as an event or plugin instead.
NumPy arrays returned by MuJoCo bindings are views into mutable C buffers. If
your plugin caches history across timesteps (e.g., position trajectories, Euler
angles), always call .copy():
# ❌ WRONG — reference updates in place, all history items point to latest state
self.history.append(app.data.xpos[body_id])
# ✅ RIGHT — stores a distinct copy
self.history.append(app.data.xpos[body_id].copy())
NullSimulation for Pure ViewersIf you do not pass step_control.StepControl() to sim_plugins,
handle.sync() will not pace itself against real wall-clock time. A tight
while handle.is_running(): handle.sync(...) loop without a stepping plugin
will spin at 100% CPU. Ensure you either include StepControl() or insert a
sleep/pacing mechanism.
To build a pure viewer (e.g., an interactive mesh browser, asset inspector,
or model visualizer where physics is not simulated), implement a
NullSimulation plugin that handles pacing (or sleeps/throttles each iteration)
without calling mj_step.
True only when your handler has fully satisfied the event and wants
to block all remaining handlers from seeing it.False or None if the event should continue propagating down the
priority chain.When authoring Studio plugin user interfaces in Dear ImGui and extensions like
ImPlot (in either C++ or Python), refer to the mujoco-gui
skill for comprehensive architecture, styling, and design best practices,
including:
imgui.Begin(...)). This ensures plugin panels can be cleanly docked into
the Studio Viewer App dockspaces alongside standard built-in tools.imgui_widgets.h/cc): Useful, shared
custom ImGui widget functionality lives in the imgui_widgets.h/cc
files in src/experimental/studio/ux/.
[!NOTE] Future Shared Widgets: Consider introducing a similar shared widgets module for widgets authored by Python users under
python/mujoco/experimental/studio/.
web/ folder) does
run a small amount of local C++ ImGui code for client-side chrome (such
as connection banners and disconnect dialogs), rich plugin widgets
rarely need to run inside the WebViewer's browser client itself.build_gui() or build_*_gui() for layout and
widget submission; reserve render / draw for low-level graphics
rasterization.## vs ###): Use ## for disambiguating duplicate
labels without changing displayed text, and ### to maintain persistent IDs
when labels contain dynamic text (preserving window docking, focus, and
collapse states) or to assign semantic names for test automation.ImGuiListClipper): Always virtualize large lists
and tables to avoid saturating WebSocket bandwidth in WebViewer / NetImgui.In WebViewer mode (launch_web), the host HTTP/WebSocket server and browser
client maintain continuous heartbeat ping-pongs over /ui and /state.
DISCONNECTED / "viewer server not
reachable" overlay banner.Event messages.