Core Concepts

What the words in a config, a log line, and a dashboard panel refer to. Components are described in Block Design, variables in Configuration.

Trial and rollout

The same attempt has two names, kept distinct throughout:

• Trial — one sandboxed execution of a task. The agent reasons, edits files, and runs commands in a fresh sandbox until it submits or hits the turn or timeout limit.
• Rollout — the token sequence captured from that trial: prompt context, sampled assistant tokens, behavior-policy log-probabilities, loss mask.

One trial produces one rollout. A step runs TRAIN_BSZ × N_RESP trials — by default 64 × 8 = 512, that is 64 tasks with 8 attempts each.

A task is an instruction, a repository snapshot, and a hidden test suite. The trainer never authors the prompt: the agent runs for as many turns as it needs and rebuilds the context each turn from its own template. Turn count and response length are therefore properties of the agent, measured per run, not settings in the config.

Reward

Each task carries an executable verifier — its own test script, in the SWE-bench sense (Jimenez et al., 2024). It runs in the sandbox after the trial and returns 1 if every fail-to-pass test now passes and every pass-to-pass test still passes, 0 otherwise.

The reward is binary and produced by execution, not by a learned reward model, which removes reward-model drift as a failure mode. What remains is protecting the verifier itself — see the anti-hacking defenses in Environment. Logged as critic/score/mean.

Advantage

GRPO (Shao et al., 2024) scores each rollout against the other attempts at the same task, so no critic is needed: the advantage of rollout $i$ in a group of $G$ is its reward minus the group mean, over the group standard deviation.

$$\hat{A}_i=\frac{r_i-\operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}$$

With a binary reward the consequence is concrete: a task solved in every attempt, or in none, has zero standard deviation and produces zero gradient — however many GPU hours its trials cost. A group carries signal only when it lands between the two extremes. Pool composition, not batch size, decides how much of a step trains anything; this is what the task filter selects for.

Policy loss

Set by adv_estimator (advantage) and policy_loss_mode (loss). The default profile is adv_estimator=grpo, policy_loss_mode=gspo.

GSPO (Qwen Team, 2025) clips on a sequence-level importance ratio, normalized by response length:

$$s_i=\left(\frac{\pi_\theta(y_i\mid x)}{\pi_{\theta_{\text{old}}}(y_i\mid x)}\right)^{1/\lvert y_i\rvert}$$

The $1/\lvert y_i \rvert$ exponent puts the clip bounds in units of average per-token log-ratio. That is why clip_ratio_low / clip_ratio_high are of order $10^{-4}$ here, and why they need no retuning as responses grow from 43k to 90k tokens.

PPO (Schulman et al., 2017) uses the same clipped form on a per-token ratio instead, with a critic supplying the advantage.

Both optimize over full multi-turn rollouts, not single completions.

Rollout fidelity

The loss is computed on a rollout the trainer did not generate. It is only valid if, for every token that carries gradient, the log-probability the trainer recomputes equals the one the serving engine emitted at generation time.

Meeting that requires the same token ids, the same mask, the same weights, and — for a sparse model — the same expert routing. Retokenizing a transcript, reconstructing a prompt, or resampling routing each break it silently: the loss still looks reasonable, and the gradient refers to a policy that never ran. The in-process proxy records log-probabilities at generation time for this reason.

The check is the correlation between the two sets of log-probabilities over a batch. Healthy is ≥ 0.99, and a drop is the first symptom of a broken path — see Training Startup.

R3 (routing replay) covers the sparse-model case: the experts the router picked at generation are recorded and replayed during the training forward pass, so both sides route each token identically. Enabled with ENABLE_R3=1 (requires USE_NEW_VERL=1); cf. Ma et al., 2025.

Importance correction

When rollout and training log-probabilities differ by construction — a different engine, or stale weights under async — ROLLOUT_IS corrects the mismatch instead of assuming it away, at token or sequence level. A sequence weight is the product of its per-token ratios, so its variance grows with response length.

The diagnostic is the effective sample size: how many of the batch's rollouts the weights actually leave in play. On the long rollouts this stack produces, ROLLOUT_IS=sequence has been measured at ESS ≈ 0.06 — the batch collapses onto a handful of samples and the gradient starves. Token-level correction is the default for that reason.

Training step

verl — the open-source implementation of HybridFlow (Sheng et al., 2025) — drives one step: sample tasks → run the trials in Harbor sandboxes, serving the policy from vLLM (Kwon et al., 2023) → collect rewards → compute advantages → update the actor → sync weights.

Rollouts whose trial ended in an infrastructure failure are dropped by the trajectory filter before the loss, so the batch reaching the optimizer is smaller than the batch that was launched.

Metrics

verl writes one step:N - key:value - … line per step to the launch log and to wandb.

The dashboard renders these as live charts.

Defined elsewhere

References

• Schulman et al., Proximal Policy Optimization Algorithms, 2017. arXiv:1707.06347
• Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. arXiv:2402.03300
• Qwen Team, Group Sequence Policy Optimization, 2025. arXiv:2507.18071
• Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024. arXiv:2310.06770
• Sheng et al., HybridFlow: A Flexible and Efficient RLHF Framework, EuroSys 2025. arXiv:2409.19256
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. arXiv:2309.06180
• Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019. arXiv:1909.08053
• Ma et al., Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers, 2025. arXiv:2510.11370