SWE-Lego-RL

Example Usages

Launch and monitor training, evaluation, and batch-inference runs

Three workloads share one contract: train a policy, eval a checkpoint on a held-out set, infer — batch-generate trajectories over a task index. Each has a runner and a set of templates. A run is one .env config copied from a template, and launches either by hand or through the agent plugin. Shown for train; eval and infer are identical in form:

# form the config once
cp scripts/train/templates/sync_k8s_ohsdk_qwen35a3b.env scripts/train/configs/my_run.env

# manual path
PREFLIGHT_ONLY=1 bash scripts/train/train.sh train/configs/my_run.env   # checks only, no launch
bash scripts/train/train.sh train/configs/my_run.env                    # launch

# agent path (same runner, same validation, plus live-machine checks)
/rl:check train/configs/my_run.env
/rl:run   train/configs/my_run.env
KindRunnerGuide
trainscripts/train/train.shthis section
evalscripts/eval/eval.shEvaluation
inferscripts/infer/infer.shBatch Inference

See Configuration for the config layers and Getting Started for the first-run walkthrough.

Launch in the background

A training run lasts hours. /rl:run backgrounds it by default; a manual launch needs tmux or nohup setsid to survive a shell disconnect. Follow it with tail -F logs/<exp>.log.

Boot sequence

A train run serves the policy from vLLM, fronts it with the in-process proxy, and lets verl drive the RL loop while the agent rolls out across tasks in Harbor sandboxes:

  1. Config — the runner sources lib/site.env + your config, applies the model preset, and resolves the scaffold and backend.
  2. Preflight — nine classes of fatal assertion; the run stops here on any ✗.
  3. venv — built once by scripts/setup_env.sh (or reused via VENV_PATH).
  4. Ray — the cluster comes up and waits for NNODES nodes to join.
  5. vLLM — verl launches DP × TP replicas; CUDA-graph capture on a 30B MoE (TP=4) is ~10–20 min before the first replica registers.
  6. In-process proxy — started by the verl agent loop; it advertises a per-session URL per trial (no standalone LiteLLM, no fixed port).
  7. verl loop — generate rollouts → run trials in Harbor → reward → update → checkpoint every SAVE_FREQ steps.

In this section

  • Preflight: the nine assertion classes PREFLIGHT_ONLY=1 runs before any launch
  • Inference Stack: the agent → in-process proxy → vLLM chain and its health checks
  • Backends: Kubernetes vs Docker sandboxes
  • Results & Artifacts: logs, checkpoints, trajectories
  • Evaluation: score a checkpoint on a held-out set
  • Batch Inference: trajectory generation and difficulty filtering
  • Scaling Up: advanced: multi-node, fully-async, VeOmni/MoE, partial rollout, R3

Start single-node

A sync_* template on one 8×GPU host is the minimal-cost configuration. Move to an async_* template and more machines only when it is no longer sufficient; see Scaling Up.

Stop cleanly between runs

Between runs, bash scripts/cleanup_before_run.sh stops Ray, frees ports, and reaps stale vLLM/verl processes. Checkpoints and wandb runs are preserved; set KEEP_TRIALS=0 to also clear the trials dir. It never kills running GPU tasks.

Data Preparation

Raw issues → Harbor task instances → sandbox images → a filtered task index

Evaluation

Score a checkpoint on a held-out task set with the eval runner

On this page

Boot sequenceIn this section