Introduction

SWE-Lego-RL is a framework for training coding agents with online reinforcement learning on real software-engineering tasks. It connects unmodified agent harnesses — Claude Code, OpenHands, OpenCode, Terminus — to verl-based policy optimization: every trial runs in a fresh Harbor sandbox, the task's own test suite produces the reward, and training data is captured at token level directly from the serving path.

Key features

• Unmodified agents — the agent that produces a rollout is the deployed agent. Each scaffold is a thin adapter, not a fork. Claude Code, OpenHands, OpenCode, and Terminus are validated end to end; any scaffold speaking the OpenAI or Anthropic API can be added as a custom adapter.
• Token-level trajectory capture — an in-process proxy serves one unified OpenAI/Anthropic API and records token ids, masks, and log-probabilities at generation time. No transcript re-tokenization, and history rewriting by the agent does not corrupt the signal.
• MoE routing replay (R3) — rollout-time expert routing is recorded and replayed during the update, keeping rollout and training log-probabilities aligned (pearson ≈ 0.999) for sparse policies.
• Verified rewards in real sandboxes — each task's verifier runs its test suite in a fresh Kubernetes or Docker container, with image caching, lazy pull, and anti-hacking defenses. Broken environments are filtered out of the loss rather than polluting the gradient.
• Scalable training — PPO / GRPO / GSPO over full multi-turn rollouts on FSDP, VeOmni, or Megatron, from a synchronous single-node loop to a fully-asynchronous multi-node cluster with partial rollout. See Scaling Up.
• Operations included — preflight config validation, an agent plugin for guided launches, a self-contained dashboard, and symptom-indexed troubleshooting.

Training Infrastructure

Each layer has its own page in Block Design:

1. Environment: each task runs in a fresh sandbox (a Kubernetes pod or a local/remote Docker container — setup and permissions in Backends) with sandboxed execution plus the task's own verifier. A coding-agent scaffold — Claude Code, OpenCode, OpenHands, Terminus 2 — drives the model through the task.
2. AgentLoopWorkers: run the scaffolds concurrently and capture each rollout.
3. In-Process Proxy: fronts every worker with a single, unified OpenAI/Anthropic API — it captures the trajectory (token-in / token-out) and forwards each call onward. No standalone LiteLLM.
4. Global Load Balancer: spreads requests across inference replicas with sticky, per-session routing.
5. Rollouter: a pool of vLLM inference servers serves the policy, streams completed rollouts into a Data Buffer, and receives Weight Sync updates.
6. Trainer: verl consumes the buffer, computes advantages, and updates the actor (PPO / GRPO / GSPO), then syncs new weights back to the rollouter for the next round.

Documentation map

• Getting Started: prerequisites and your first training run
• Data Preparation: raw issues → Harbor tasks, sandbox images, and a filtered task index
• Example Usages: launch and monitor training, evaluation, and batch inference
• Core Concepts: the terms used in configs, logs, and the dashboard
• Block Design: the architecture, one page per block of the diagram above
• Agent Plugin: validate, launch, and diagnose runs through the /rl:* skills
• Dashboard: serve the training board in one command, locally or on a public URL
• Reference: the runner configuration surface