Training algorithms#

--algo ppo|wm — World-model training.

Training algorithms of a run: ppo (default) or wm (world model, wbc_mjlab.rl.wm).

Every algorithm trains the same tasks and presets (env, motion options) and exports the same policy; it brings its own runner config (params/agent.yaml, which Play reads back) and may adapt the env. ALGORITHMS describes them for the CLI and the Studio (whose recipes, wbc_mjlab.studio.recipes, are an algorithm and its settings).

wbc_mjlab.rl.algos.ALGORITHMS: dict[Literal['ppo', 'wm'], AlgoInfo]#

Every training algorithm (what --algo accepts), by id.

class wbc_mjlab.rl.algos.AlgoInfo[source]#

Bases: object

What an algorithm is, for listings and the Studio’s Train form.

id: Literal['ppo', 'wm']#
label: str#
summary: str#
settings: tuple[str, ...] = ()#

Runner config keys worth setting per run (algorithm.imagined_rollouts …).

iterations: tuple[int, ...] = (1000, 5000, 30000, 100000)#

Typical max_iterations budgets (an iteration’s length differs per algorithm).

notes: str = ''#
wbc_mjlab.rl.algos.rl_cfg(task_id: str, algo: Literal['ppo', 'wm'] = 'ppo')[source]#

Runner config of task_id for algo.

wbc_mjlab.rl.algos.runner_cfg(ppo_cfg, algo: Literal['ppo', 'wm'])[source]#

algo’s runner config derived from a task’s PPO runner config ppo_cfg (the policy network, experiment and logger carry over).

wbc_mjlab.rl.algos.env_cfg(cfg, algo: Literal['ppo', 'wm'])[source]#

The env algo trains (and plays) cfg in: wm turns off external forces.

wbc_mjlab.rl.algos.check_env(task_id: str, algo: Literal['ppo', 'wm'], motion_file: str, *, num_envs: int = 2) → dict[source]#

Build task_id’s env for algo (num_envs envs on motion_file) and check that algo can train it: {"ok", "error", "details"}. For wm, details names the rewards computed exactly in imagination, those learned, and the randomization the model is given as context.

wbc_mjlab.rl.algos.run_algo(run_dir: Path | str) → Literal['ppo', 'wm'][source]#

Algorithm a run was trained with (params/agent.yaml; PPO when absent).

wbc_mjlab.rl.algos.run_rl_cfg(task_id: str, run_dir: Path | str)[source]#

Rebuild a run’s architecture for Play/evaluation, including optional encoders.

Apply known dataclass fields only: YAML is data, never imported or evaluated.

Recipes (Studio)#

Training recipes: how a preset learns — the method (PPO or the world model) and its settings — saved by name, independent of any preset.

A preset is the task (what the policy sees, how it is rewarded, where episodes start); a recipe is how it is trained. Train picks one of each; world-model recipes share the trainer’s observation, action and environment support constraints. An unedited fresh job passes --recipe <name> to resolve defaults in the worker; custom edits use --agent.<key> <value>. Resolved configs are saved in the run for reproducibility and resume.

Built-in recipes: ppo and wm (task defaults), wm-24gb (2048 imagined rollouts), wm-encoder, wm-blink and wm-party (optional experiments). Yours live in <config dir>/recipes/<name>.yaml (wbc_mjlab.studio.presets.config_dir()).

wbc_mjlab.studio.recipes.fields(task: str, algo: Literal['ppo', 'wm']) → list[dict][source]#

Every setting of algo for task: {key, kind, value, section, main, help} with the task’s value as value (main: the method’s settings shown first).

wbc_mjlab.studio.recipes.list_recipes() → list[dict][source]#

Built-in recipes, then yours: {name, algo, label, description, settings, builtin}.

wbc_mjlab.studio.recipes.save(name: str, algo: str, settings: dict, description: str = '') → dict[source]#
wbc_mjlab.studio.recipes.architecture_of(run_dir: Path | str) → dict[source]#

The ARCHITECTURE settings run run_dir was trained with (those its params/agent.yaml holds).

Shared policy, saved runs and evaluation#

class wbc_mjlab.rl.policy.PolicyCfg[source]#

Bases: RslRlModelCfg

Policy architecture, independent of the optimizer used to train it.

activation: Literal['elu', 'selu', 'relu', 'crelu', 'lrelu', 'tanh', 'sigmoid', 'softplus', 'gelu', 'swish', 'silu', 'mish', 'identity'] = 'elu'#
reference_residual: bool = False#
keypoint_skip: bool = False#
keypoint_skip_hidden: tuple[int, ...] = (1024, 1024, 512)#
keypoint_skip_steps: int = 20000#
architecture: Literal['mlp', 'motion_encoder', 'xattn'] = 'mlp'#
xattn_dim: int = 128#
xattn_heads: int = 4#
encoder_hidden_dims: tuple[int, ...] = (2048, 1024, 512, 256)#
latent_dim: int = 32#
latent_aux: bool = False#
goal_terms: tuple[str, ...] = ()#
wbc_mjlab.run_params.load_run_config(run_dir: Path | str, task_id: str | None = None, *, play: bool = True) → RunConfig[source]#

Shared saved-task reconstruction for CLI/Studio Play, evaluation and resume.

Build registered Python code first, restore saved data second, then apply explicit Play overrides. Evaluation applies its protocol after this function returns.

wbc_mjlab.env.mdp.sampling.sampling_state(command: MotionCommand) → dict[source]#

One state schema for PPO/WM checkpoints and exported RSI sidecars.

wbc_mjlab.env.mdp.sampling.restore_sampling_state(command: MotionCommand, state, *, strict=False) → bool[source]#

Validate identity before copying anything; accept older schemas when compatible.

class wbc_mjlab.rl.evaluation.EvaluationConfig[source]#

Bases: object

checkpoint_file: Path | str#
task: str | None = None#
dataset: str | None = None#
dataset_path: str | None = None#
num_envs: int = 128#
device: str = 'cuda:0'#
clips: list[str] | None = None#
max_step_reward: float | None = None#
out: Path | str | None = None#
wbc_mjlab.rl.evaluation.evaluate_checkpoint(config: EvaluationConfig)[source]#

Shared CLI/Studio backend: restore a run, apply the protocol, evaluate, write JSON.

wbc_mjlab.rl.evaluation.deterministic_cfg(cfg)[source]#

Keep preset terms/weights, disable noise, randomization and external events.

wbc_mjlab.rl.evaluation.run_benchmark(env, policy, *, trajectories=None, max_step_reward=None, clip_actions=None)[source]#

Play each requested trajectory once. Returns serializable summary and clip rows.

env must use deterministic_cfg; the caller owns and closes it. Reward means include the terminal step. Drift is measured before stepping to avoid auto-reset contamination. A failure is any configured non-timeout termination.

World model (wm)#

class wbc_mjlab.rl.wm.config.WmAlgorithmCfg[source]#

Bases: object

World model fit to simulator steps; actor trained by backpropagation through it.

window: int = 8#

Steps of the autoregressive windows the world model is fit on (k).

model_hidden_dims: tuple[int, ...] = (2048, 2048, 2048)#

Dynamics network.

heads_hidden_dims: tuple[int, ...] = (512, 256)#

Step heads (termination, learned reward terms).

dynamics_mode: Literal['increment', 'integrated', 'velocity'] = 'increment'#

Full increments, integrated pose residuals, or velocity-only integration.

velocity_blend: float | None = 1.0#

new-velocity share; None derives (n+1)/(2n) from decimation.

Type:

Integrated pose prior

dynamics_loss_scale: Literal['output', 'increment', 'physical'] = 'output'#

output statistics, full increment statistics, or fixed physical tolerances.

Type:

Loss units

physical_error_scales: tuple[float, ...] = (0.01, 0.02, 0.5, 1.0, 0.1, 3.14)#

root m/rad/m/s/rad/s, joint rad/rad/s; physical loss only.

Pose/joint tolerances follow official SuperMimic; explicit root velocity supervision is our extension. Scales never change with replay statistics.

Type:

Fixed tolerances

critic_inputs: Literal['actor', 'structured'] = 'actor'#

physical state/tracking/context, clean goals and policy history.

Type:

Structured

critic_layer_norm: bool = False#
critic_activation: Literal['task', 'elu', 'gelu', 'silu'] = 'task'#
model_optimizer: Literal['adam', 'muon'] = 'adam'#
actor_optimizer: Literal['adam', 'muon'] = 'adam'#
muon_weight_decay: float = 0.0#

0.1.

Type:

Decoupled weight decay of the Muon groups (world model and actor); PartyMimic

actor_muon_io: bool = False#

Muon also on the first and last Linear of the actor’s MLPs (PartyMimic default).

fused_optimizer: bool = False#
skip_nonfinite_updates: bool = False#
critic_norm_eps: float = 0.0001#

Minimum critic input scale; Blink uses 0.01 to limit near-constant-feature gain.

bound_critic: bool = False#

Cap bootstrap values at the task reward upper bound / (1-gamma).

termination_mode: Literal['learned', 'hybrid'] = 'learned'#

Hybrid uses task tracking thresholds as a lower bound on imagined termination.

termination_margin_weight: float = 0.0#

Reward rate subtracted per tracking term at its failure threshold; zero disables.

termination_margin_start: float = 0.5#

Fraction of the configured threshold where the margin cost starts, in [0, 1).

policy_ema: float = 0.0#

Deployment policy exponential average; zero disables. Live weights resume training.

mirror_probability: float = 0.0#

Share of dynamics windows reflected using robot symmetry; step heads remain unmirrored.

actor_lr_multiplier: float = 1.0#
actor_lr_decay_start: int = 1400#
actor_lr_decay_updates: int = 300#
actor_lr_return_threshold: float | None = None#

Start multiplier decay at this normalized simulator reward, or the update deadline.

recent_failure_share: float | None = None#

Override the task’s recent-failure reset mixture when set.

probe_share: float = 0.15#
collection_cuda_graphs: bool = False#
probe_cuda_graphs: bool = False#

Capture policy inference; physics graphs remain owned by mjlab.

probe_interval: int = 0#

Actor updates between curriculum probe batches; zero disables.

probe_start: int = 1000#
probe_num_envs: int = 128#
probe_tile_s: float = 6.0#
probe_lead_s: tuple[float, float] = (4.0, 9.0)#
probe_steps_per_iteration: int = 20#

Bounded probe work per runner iteration; zero completes a sweep synchronously.

probe_async: bool = False#

Run probe chunks on a CUDA side stream with explicit snapshot/event ownership.

compile_collection: bool = False#

Compile the collection actor and critic loss; simulation uses mjlab’s own graphs.

separate_action_noise: bool = False#

Exploration only in execution; observations and action-rate reward use clean actions.

self_imitation: float = 0.0#

regress the actor onto the recorded (exploring) action where the return the simulator recorded beats V(s₀), weighted by the advantage (PartyMimic --sil: 1.0). Zero disables.

Type:

Weight of self-imitation at imagination starts

self_imitation_max_weight: float = 5.0#

Cap of the advantage weight, in units of the batch’s mean absolute advantage.

value_gradient: float = 0.0#

Critic loss weight on matching ∇ₛV to the backpropagated imagined-return gradient at imagined states (PartyMimic --v_gcons: 0.5). Zero disables.

value_gradient_states: int = 4096#

Imagined states per critic minibatch in the value-gradient loss.

value_gradient_epochs: int = 1#

Critic epochs whose minibatches include the value-gradient loss.

bf16: bool = False#

Run the networks’ matmuls (world model, critic, policy head) in bfloat16 autocast; integration, kinematics, rewards and normalization stay float32 (PartyMimic --amp bf16). CUDA only; exported policies stay float32.

latent_aux: float = 0.0#

Weight of predicting the goal frames’ joint positions from the actor’s reference latent (actor.latent_aux head; PartyMimic --kp_aux: 1.0 with keypoint goals). Zero disables.

scale_lr_by_world_size: bool = True#

Distributed runs use sqrt(world_size) times the configured learning rate.

diagnostic_interval: int = 100#

Updates between independent replay rollout diagnostics; zero disables them.

diagnostic_batch: int = 256#
model_batch: int = 2048#

Windows per world-model gradient step.

model_steps: int = 10#

World-model gradient steps per update.

horizon: int = 32#

Imagined steps per rollout (H); the critic bootstraps beyond.

imagined_rollouts: int = 4096#

Rollouts per policy update, started from replay states.

updates_per_iteration: int = 2#

2 per 5 steps is the paper’s update-to-data ratio of 0.4.

Type:

Updates per runner iteration (num_steps_per_env env steps)

gamma: float = 0.99#
lam: float = 0.95#

GAE λ of the critic targets.

action_noise: float = 0.1#

Std of the Gaussian noise on actions in the simulator (fixed).

learning_rate: float = 0.0002#

World model, actor and critic.

lr_final_ratio: float = 0.03#

Learning rates decay (cosine) to this fraction…

lr_decay_updates: int = 250000#

…over this many updates, then stay constant.

max_grad_norm: float = 1.0#
actor_update: Literal['pathwise', 'action_targets'] = 'pathwise'#

Direct return gradient, or regress onto gradient-improved actions (Nabla-inspired).

action_target_steps: int = 8#
action_target_windows: int = 1#
action_target_batch: int = 4096#
action_target_delta: float = 0.03#

Initial RMS target displacement in joint-setpoint radians.

action_target_delta_floor: float = 0.3#

Cosine annealing floor relative to delta, using lr_decay_updates.

action_gradient_clip_ratio: float = 0.0#

3); 0 disables.

Type:

Per-step action cotangent limit / median live norm (NablaMimic

actor_grad_clip: float = 0.1#

Element-wise clip of actor gradients, after the norm clip.

critic_epochs: int = 5#
critic_mini_batches: int = 4#
replay_steps: int = 1000#

Steps of every env kept in the replay.

warmup_steps: int = 32#

Env steps collected before the first update.

compile: bool = True#

torch.compile the imagination step and the world-model losses (first update compiles for a few minutes; off: slower, for debugging).

cuda_graphs: bool = True#

Run each world-model and policy update as one captured CUDA graph (thousands of small kernels launched at once); with compile only.

class_name: str = 'wbc_mjlab.rl.wm.algorithm:WorldModelRL'#
wbc_mjlab.rl.wm.config.wm_env_cfg(cfg: mjlab.envs.ManagerBasedRlEnvCfg) → mjlab.envs.ManagerBasedRlEnvCfg[source]#

Remove unobserved ongoing forces; offer optional, explicitly recorded velocity kicks.

Kicks preserve a pre-disturbance target and cut model windows (perturbations.py).

class wbc_mjlab.rl.wm.algorithm.WorldModelRL[source]#

Bases: object

rsl_rl algorithm: replay → world model → actor by BPTT, critic by GAE.

static construct_algorithm(obs: TensorDict, env: VecEnv, cfg: dict, device: str) → WorldModelRL[source]#
act(obs: TensorDict) → Tensor[source]#
process_env_step(obs: TensorDict, rewards: Tensor, dones: Tensor, extras: dict) → None[source]#
compute_returns(obs: TensorDict) → None[source]#
update() → dict[str, float][source]#
close()[source]#
train_mode() → None[source]#
eval_mode() → None[source]#
get_policy() → MLPModel[source]#
save() → dict[source]#
load(loaded_dict: dict, load_cfg: dict | None, strict: bool) → bool[source]#
broadcast_parameters() → None[source]#
rollout_diagnostics(batch: int = 256) → dict[str, float][source]#

Independent replay windows, fixed physical scales; never update normalization.

After SuperMimic Fig. 3c and SuperMimic-Reproduced’s wm_trainer.online_error (ba61053). These are in-replay diagnostics, not held-out generalization scores.

class wbc_mjlab.rl.wm.imagined.Imagination[source]#

Bases: object

Everything the world-model algorithm needs from the env, read once at construction.

Raises ValueError when the env has something imagination cannot follow: an actor observation that is not a function of the robot state and the reference, observation delay, an unsupported action term, or domain randomization outside the context (context()).

summary() → str[source]#
goal() → Goal[source]#

The env’s current goal (window included when the actor looks ahead).

env_history() → Tensor[source]#

The env’s observation history of the history terms, (N, H, history_dim), oldest first ((N, 1, 0) without history).

context(env_ids: Tensor) → Tensor[source]#
robot_state() → RobotState[source]#

The simulator’s robot state in the env frame (env origin subtracted).

view(s, bodies, env_ids, goal: Goal, action, prev_action) → ImaginedEnv[source]#
site_sensor(sensor_name: str) → tuple[bool, int, Tensor, Tensor][source]#

(is gyro, else velocimeter; robot body; site pos and quat in the body).

actor_obs(env: ImaginedEnv, past: Tensor | None = None, *, corruption: bool = True) → tuple[Tensor, Tensor][source]#

The actor observation group (noise as configured, clip, scale) and the history.

past: the history terms’ previous H - 1 values, (n, H - 1, history_dim) oldest first. Returns the observation and the history including this step’s values.

analytic_reward(env: ImaginedEnv) → Tensor[source]#

Sum of the reward terms that are functions of the imagined state (× dt).

learned_reward_targets() → Tensor[source]#

The env’s last step values of the learned reward terms (weighted, before × dt).

tracking_features(env: ImaginedEnv) → Tensor[source]#
joint_setpoint(action: Tensor, ref: MotionFrame, env_ids: Tensor)[source]#

PD position targets of a raw action (the action term’s processing, encoder bias).

class wbc_mjlab.rl.wm.model.WorldModel[source]#

Bases: Module

Dynamics f(s, u, ξ) → s' plus step heads h(s, u, g, ξ) → (termination, rewards).

u are the PD targets relative to the joint positions, g the tracking features of the state against the reference, ξ the domain-randomization context. The step heads predict whether the step ends the episode and the reward terms that are not functions of the state (actuator forces, contacts, …).

step(s: RobotState, setpoint: Tensor, context: Tensor) → RobotState[source]#
target_change(s: RobotState, next_state: RobotState) → Tensor[source]#
step_heads(s: RobotState, setpoint: Tensor, tracking: Tensor, context: Tensor) → tuple[Tensor, Tensor][source]#

(termination logit, learned reward terms) of the step from s.

update_dynamics_normalization(s: RobotState, setpoint: Tensor, context: Tensor, change: Tensor, weight: Tensor) → None[source]#

From logged steps (weight: those that are real transitions).

update_heads_normalization(s: RobotState, setpoint: Tensor, tracking: Tensor, context: Tensor, rewards: Tensor) → None[source]#
reward_error(pred: Tensor, target: Tensor) → Tensor[source]#

Squared error of the learned reward terms in units of their spread.

change_error(pred: RobotState, true: RobotState) → Tensor[source]#

Squared error of pred vs true in units of the typical one-step change.

class wbc_mjlab.rl.wm.kinematics.RobotKinematics[source]#

Bases: object

Forward kinematics for one robot entity of a compiled MuJoCo model.

Parameters:
  • mj_model – The scene model.

  • body_ids – Global body ids of the robot, in entity order (root first).

  • joint_qpos_adr – qpos address of each robot hinge joint, in entity joint order.

  • device – Torch device.

site_frame(site_id: int) → tuple[int, Tensor, Tensor][source]#

(entity body index, local pos, local quat) of a site on the robot.

bodies(s: RobotState) → BodyKinematics[source]#

Poses and origin velocities of every body (entity order).