Training algorithms#
--algo ppo|wm — World-model training.
Training algorithms of a run: ppo (default) or wm (world model, wbc_mjlab.rl.wm).
Every algorithm trains the same tasks and presets (env, motion options) and exports the same
policy; it brings its own runner config (params/agent.yaml, which Play reads back) and
may adapt the env. ALGORITHMS describes them for the CLI and the Studio (whose
recipes, wbc_mjlab.studio.recipes, are an algorithm and its settings).
- wbc_mjlab.rl.algos.ALGORITHMS: dict[Literal['ppo', 'wm'], AlgoInfo]#
Every training algorithm (what
--algoaccepts), by id.
- class wbc_mjlab.rl.algos.AlgoInfo[source]#
Bases:
objectWhat an algorithm is, for listings and the Studio’s Train form.
- settings: tuple[str, ...] = ()#
Runner config keys worth setting per run (
algorithm.imagined_rollouts…).
- wbc_mjlab.rl.algos.rl_cfg(task_id: str, algo: Literal['ppo', 'wm'] = 'ppo')[source]#
Runner config of
task_idforalgo.
- wbc_mjlab.rl.algos.runner_cfg(ppo_cfg, algo: Literal['ppo', 'wm'])[source]#
algo’s runner config derived from a task’s PPO runner configppo_cfg(the policy network, experiment and logger carry over).
- wbc_mjlab.rl.algos.env_cfg(cfg, algo: Literal['ppo', 'wm'])[source]#
The env
algotrains (and plays)cfgin:wmturns off external forces.
- wbc_mjlab.rl.algos.check_env(task_id: str, algo: Literal['ppo', 'wm'], motion_file: str, *, num_envs: int = 2) dict[source]#
Build
task_id’s env foralgo(num_envsenvs onmotion_file) and check thatalgocan train it:{"ok", "error", "details"}. Forwm,detailsnames the rewards computed exactly in imagination, those learned, and the randomization the model is given as context.
- wbc_mjlab.rl.algos.run_algo(run_dir: Path | str) Literal['ppo', 'wm'][source]#
Algorithm a run was trained with (
params/agent.yaml; PPO when absent).
- wbc_mjlab.rl.algos.run_rl_cfg(task_id: str, run_dir: Path | str)[source]#
Rebuild a run’s architecture for Play/evaluation, including optional encoders.
Apply known dataclass fields only: YAML is data, never imported or evaluated.
Recipes (Studio)#
Training recipes: how a preset learns — the method (PPO or the world model) and its settings — saved by name, independent of any preset.
A preset is the task (what the policy sees, how it is rewarded, where episodes start); a recipe
is how it is trained. Train picks one of each; world-model recipes share the
trainer’s observation, action and environment support constraints.
An unedited fresh job passes --recipe <name> to resolve defaults in the worker;
custom edits use --agent.<key> <value>. Resolved configs are saved in the run
for reproducibility and resume.
Built-in recipes: ppo and wm (task defaults), wm-24gb (2048 imagined
rollouts), wm-encoder, wm-blink and wm-party (optional experiments). Yours live in
<config dir>/recipes/<name>.yaml (wbc_mjlab.studio.presets.config_dir()).
- wbc_mjlab.studio.recipes.fields(task: str, algo: Literal['ppo', 'wm']) list[dict][source]#
Every setting of
algofortask:{key, kind, value, section, main, help}with the task’s value asvalue(main: the method’s settings shown first).
- wbc_mjlab.studio.recipes.list_recipes() list[dict][source]#
Built-in recipes, then yours:
{name, algo, label, description, settings, builtin}.
World model (wm)#
- class wbc_mjlab.rl.wm.config.WmAlgorithmCfg[source]#
Bases:
objectWorld model fit to simulator steps; actor trained by backpropagation through it.
Dynamics network.
Step heads (termination, learned reward terms).
- dynamics_mode: Literal['increment', 'integrated', 'velocity'] = 'increment'#
Full increments, integrated pose residuals, or velocity-only integration.
- velocity_blend: float | None = 1.0#
new-velocity share; None derives (n+1)/(2n) from decimation.
- Type:
Integrated pose prior
- dynamics_loss_scale: Literal['output', 'increment', 'physical'] = 'output'#
output statistics, full increment statistics, or fixed physical tolerances.
- Type:
Loss units
- physical_error_scales: tuple[float, ...] = (0.01, 0.02, 0.5, 1.0, 0.1, 3.14)#
root m/rad/m/s/rad/s, joint rad/rad/s; physical loss only.
Pose/joint tolerances follow official SuperMimic; explicit root velocity supervision is our extension. Scales never change with replay statistics.
- Type:
Fixed tolerances
- critic_inputs: Literal['actor', 'structured'] = 'actor'#
physical state/tracking/context, clean goals and policy history.
- Type:
Structured
- muon_weight_decay: float = 0.0#
0.1.
- Type:
Decoupled weight decay of the Muon groups (world model and actor); PartyMimic
- actor_muon_io: bool = False#
Muon also on the first and last Linear of the actor’s MLPs (PartyMimic default).
- critic_norm_eps: float = 0.0001#
Minimum critic input scale; Blink uses 0.01 to limit near-constant-feature gain.
- termination_mode: Literal['learned', 'hybrid'] = 'learned'#
Hybrid uses task tracking thresholds as a lower bound on imagined termination.
- termination_margin_weight: float = 0.0#
Reward rate subtracted per tracking term at its failure threshold; zero disables.
- termination_margin_start: float = 0.5#
Fraction of the configured threshold where the margin cost starts, in [0, 1).
- policy_ema: float = 0.0#
Deployment policy exponential average; zero disables. Live weights resume training.
- mirror_probability: float = 0.0#
Share of dynamics windows reflected using robot symmetry; step heads remain unmirrored.
- actor_lr_return_threshold: float | None = None#
Start multiplier decay at this normalized simulator reward, or the update deadline.
Override the task’s recent-failure reset mixture when set.
- probe_steps_per_iteration: int = 20#
Bounded probe work per runner iteration; zero completes a sweep synchronously.
- probe_async: bool = False#
Run probe chunks on a CUDA side stream with explicit snapshot/event ownership.
- compile_collection: bool = False#
Compile the collection actor and critic loss; simulation uses mjlab’s own graphs.
- separate_action_noise: bool = False#
Exploration only in execution; observations and action-rate reward use clean actions.
- self_imitation: float = 0.0#
regress the actor onto the recorded (exploring) action where the return the simulator recorded beats V(s₀), weighted by the advantage (PartyMimic
--sil: 1.0). Zero disables.- Type:
Weight of self-imitation at imagination starts
- self_imitation_max_weight: float = 5.0#
Cap of the advantage weight, in units of the batch’s mean absolute advantage.
- value_gradient: float = 0.0#
Critic loss weight on matching ∇ₛV to the backpropagated imagined-return gradient at imagined states (PartyMimic
--v_gcons: 0.5). Zero disables.
- bf16: bool = False#
Run the networks’ matmuls (world model, critic, policy head) in bfloat16 autocast; integration, kinematics, rewards and normalization stay float32 (PartyMimic
--amp bf16). CUDA only; exported policies stay float32.
- latent_aux: float = 0.0#
Weight of predicting the goal frames’ joint positions from the actor’s reference latent (
actor.latent_auxhead; PartyMimic--kp_aux: 1.0 with keypoint goals). Zero disables.
- scale_lr_by_world_size: bool = True#
Distributed runs use sqrt(world_size) times the configured learning rate.
- diagnostic_interval: int = 100#
Updates between independent replay rollout diagnostics; zero disables them.
- updates_per_iteration: int = 2#
2 per 5 steps is the paper’s update-to-data ratio of 0.4.
- Type:
Updates per runner iteration (
num_steps_per_envenv steps)
- actor_update: Literal['pathwise', 'action_targets'] = 'pathwise'#
Direct return gradient, or regress onto gradient-improved actions (Nabla-inspired).
- action_target_delta_floor: float = 0.3#
Cosine annealing floor relative to delta, using lr_decay_updates.
- action_gradient_clip_ratio: float = 0.0#
3); 0 disables.
- Type:
Per-step action cotangent limit / median live norm (NablaMimic
- compile: bool = True#
torch.compilethe imagination step and the world-model losses (first update compiles for a few minutes; off: slower, for debugging).
- wbc_mjlab.rl.wm.config.wm_env_cfg(cfg: mjlab.envs.ManagerBasedRlEnvCfg) mjlab.envs.ManagerBasedRlEnvCfg[source]#
Remove unobserved ongoing forces; offer optional, explicitly recorded velocity kicks.
Kicks preserve a pre-disturbance target and cut model windows (perturbations.py).
- class wbc_mjlab.rl.wm.algorithm.WorldModelRL[source]#
Bases:
objectrsl_rl algorithm: replay → world model → actor by BPTT, critic by GAE.
- static construct_algorithm(obs: TensorDict, env: VecEnv, cfg: dict, device: str) WorldModelRL[source]#
- class wbc_mjlab.rl.wm.imagined.Imagination[source]#
Bases:
objectEverything the world-model algorithm needs from the env, read once at construction.
Raises
ValueErrorwhen the env has something imagination cannot follow: an actor observation that is not a function of the robot state and the reference, observation delay, an unsupported action term, or domain randomization outside the context (context()).- env_history() Tensor[source]#
The env’s observation history of the history terms,
(N, H, history_dim), oldest first ((N, 1, 0)without history).
- robot_state() RobotState[source]#
The simulator’s robot state in the env frame (env origin subtracted).
- site_sensor(sensor_name: str) tuple[bool, int, Tensor, Tensor][source]#
(is gyro, else velocimeter; robot body; site pos and quat in the body).
- actor_obs(env: ImaginedEnv, past: Tensor | None = None, *, corruption: bool = True) tuple[Tensor, Tensor][source]#
The actor observation group (noise as configured, clip, scale) and the history.
past: the history terms’ previousH - 1values,(n, H - 1, history_dim)oldest first. Returns the observation and the history including this step’s values.
- analytic_reward(env: ImaginedEnv) Tensor[source]#
Sum of the reward terms that are functions of the imagined state (× dt).
- class wbc_mjlab.rl.wm.model.WorldModel[source]#
Bases:
ModuleDynamics
f(s, u, ξ) → s'plus step headsh(s, u, g, ξ) → (termination, rewards).uare the PD targets relative to the joint positions,gthe tracking features of the state against the reference,ξthe domain-randomization context. The step heads predict whether the step ends the episode and the reward terms that are not functions of the state (actuator forces, contacts, …).- step_heads(s: RobotState, setpoint: Tensor, tracking: Tensor, context: Tensor) tuple[Tensor, Tensor][source]#
(termination logit, learned reward terms) of the step from
s.
- update_dynamics_normalization(s: RobotState, setpoint: Tensor, context: Tensor, change: Tensor, weight: Tensor) None[source]#
From logged steps (
weight: those that are real transitions).
- update_heads_normalization(s: RobotState, setpoint: Tensor, tracking: Tensor, context: Tensor, rewards: Tensor) None[source]#
- class wbc_mjlab.rl.wm.kinematics.RobotKinematics[source]#
Bases:
objectForward kinematics for one robot entity of a compiled MuJoCo model.
- Parameters:
mj_model – The scene model.
body_ids – Global body ids of the robot, in entity order (root first).
joint_qpos_adr –
qposaddress of each robot hinge joint, in entity joint order.device – Torch device.