Training a terminal agent with reinforcement learning
October 2026 · Experiment notes
I trained Qwen3.5-9B to solve terminal tasks. The run finished, but the adapter passed fewer held-out tasks than the base model. Here is what I built and what went wrong.
TL;DR
- One trainer, two inference GPUs. I trained Qwen3.5-9B with LoRA for ten updates using Prime RL on three H100s in Modal.
- 128 concurrent rollouts. I configured 64 per inference GPU, with eight attempts per task and 64 accepted trajectories per training update.
- Tasks with a 25–75% screening pass rate. I kept tasks with two to six passes out of eight attempts: 103 total, split into 80 training and 23 held-out tasks.
- About 89% lower sandbox request floor. I reduced requests from 1 CPU and 2 GiB to 0.125 CPU and 0.125 GiB. Sandboxes could still burst above those requests, so this is not an 89% saving on the total bill.
- 60 seconds per Bash command. Commands that exceeded the limit returned partial output so the agent could continue. Compaction also kept the original task alongside its latest summary.
Held-out result: 91/184 passes for the base model, 83/184 after training. More attempts reached turn or output limits. This run did not demonstrate an improvement.
I wanted to let the model work in a terminal, grade its files, and train on those attempts. First I had to get that loop working.
The setup
I used Qwen3.5-9B with thinking enabled, Bash, and an edit tool. Prime RL handled training and asynchronous rollouts; Verifiers ran the agent loop; Harbor graded the resulting files in a separate environment; Modal supplied the GPUs and sandboxes.
Following TermGrade’s selection approach, I screened 216 tasks with eight attempts each. The 103 tasks with two to six passes became an 80/23 training/held-out split. Difficulty depends on the model and harness: TermGrade evaluated with Terminus-2, while I used Bash.
I pinned Git commits and file hashes, checked that a no-op failed and the reference solution passed, and fixed Harbor packaging before using GPUs. Twelve retained Dockerfiles had interfered with native environment upload; removing those active files restored the intended setup path.
- Adapter
- LoRA rank 16, alpha 32, learning rate 1e-5.
- Precision
- FP8 inference; BF16 training with FP32 optimization and reduction.
- Sampling
- Temperature 0.6, top-p 0.95, top-k 20; 8,192 output tokens per request.
- Context
- 50,000 tokens, compaction at 30,000, maximum 50 turns.
Getting inference and training to agree
My first setup generated answers I could not reliably train on. I moved to Prime’s integrated Bash path for tool execution, compaction, and token capture. I had no evidence that Bash solved more tasks.
I broke a migration by replacing bundled dependencies with older releases. The client could no longer decode the packed log-probabilities and sampling masks. Replaying one response through both decoders exposed the mismatch. I restored compatible versions. I checked a full tool-call exchange for matching token prefixes, finite log-probabilities, and sampled tokens inside their recorded support.
A full conversation rendered, but a system-only prefix failed with “No user query found in messages.” I used Prime’s Qwen3.5 renderer and checked prefixes and masks as the conversation grew.
Two screening workers later failed discovery because I supplied the served model name as a string instead of a list. Another router path omitted authentication internally. With one engine per worker, I used supported direct-engine mode, kept authentication enabled, and checked the advertised name, wrong-key rejection, and generation.
Commands that kept waiting
Some terminal commands and sandboxes had stayed stuck for hours. When I checked the executor, it allowed 3,600 seconds per command. Shortening the HTTP timeout would still leave the process running inside the sandbox.
My Bash patch closes stdin, starts a process group, and kills that group after 60 seconds, returning partial output. Killing only Bash can leave children running or holding output pipes open, so the follow-up read is bounded too.
The Linux check stopped at 60.02 seconds, passed six process tests, and let the next command run. Legitimate long builds get stopped too. File edits remain, and detached processes still need sandbox cleanup. I could not trace every earlier wait to the same cause.
Paying less while sandboxes wait
Sandboxes often sat idle while the model generated tokens. I reduced each request from one CPU and 2 GiB RAM to 0.125 CPU and 0.125 GiB. These are uncapped requests: one earlier sandbox reached 9.32 GiB despite requesting 2 GiB.
- Old floor
- 1 core + 2 GiB: $0.1899 per sandbox-hour.
- New floor
- 0.125 core + 0.125 GiB: $0.02074 per sandbox-hour.
Modal bills the higher of requested or actual CPU/memory usage. The request floor fell about 89%; busy tasks still pay for their usage, and lower requests guarantee less capacity on a busy host. GPU time and other services are additional.
These estimates use sandbox rates, three times function rates: $0.00003942/core-second and $0.00000667/GiB-second, checked against pricing on October 31, 2026. They are not measured total-bill savings.
Keeping the task, then getting back to work
Compaction rebuilt the prompt from system instructions and a generated summary, potentially dropping the exact task. I changed Verifiers to retain the original task plus the latest summary. Repeated compaction replaces the previous summary instead of accumulating summaries. CPU checks confirmed that the task survived. A bad summary or an earlier coding mistake can still carry through.
Some attempts spent all 8,192 output tokens thinking without calling a tool. I kept thinking enabled and added one corrective request per episode, using mini-SWE-agent’s output-limit feedback. The unfinished response stays in the trace but is omitted from the next prompt. I kept Bash’s timeout and compaction behavior rather than switching adapters.
The recovery comparison finished 9/16 with recovery and 9/16 without, while recovery used 62% more output tokens. Two final-evaluation attempts still consumed two full responses without taking any tool action.
Choosing inference concurrency
I tested 16, 32, 64, 98, and 128 concurrent requests on one H100 with two fixed workloads: long uncached prefill and shared-prefix long output.
Long prefill: throughput
Output tokens / second
Shared-prefix long output: throughput
Output tokens / second
Long prefill: request latency
Mean seconds per request
Shared-prefix long output: request latency
Mean seconds per request
Long prefill: elapsed time
Seconds to finish the workload
Shared-prefix long output: elapsed time
Seconds to finish the workload
Average GPU utilization
Percent · monitored phase
Observed KV-cache preemptions
Count · monitored phase
One H100, frozen weights. At each concurrency, both workloads ran together with 64 requests each. Long prefill used about 16k input and 2,048 output tokens. Long output used a shared 24k prefix plus 4k input and 8,192 output tokens. These are synthetic serving measurements.
Concurrency 128 won synthetic long-output throughput, but requests took longer and it recorded 25 KV-cache preemptions. The separate 98 run was slower than 64 despite zero preemptions. I chose 64 per inference GPU for real rollouts, 128 globally. I chose for usable training batches, not token throughput alone.
What reached the trainer
Prime keeps collecting attempts while the trainer updates the model and sends out new weights. An attempt can start before an update and finish after it. I allowed one off-policy step. For GRPO, I grouped eight attempts on the same task. If all eight get the same reward, that group gives no reward difference to learn from.
What happened to the returned attempts?
Episodes, grouped by policy version at group start
The stream returned 1,394 episodes; ten batches accepted 640, including 365 passes and 275 failures. All 80 training tasks appeared, but only 55 contributed accepted trajectories. Prime dropped 306 completed stale traces; other extra or cancelled work had different causes.
Accepted training reward
Pass rate (%)
Gradient norm
L2 norm
Token entropy
Logged mean entropy
Rollout / trainer mismatch KL
Logged mean KL
Where the trainer spent time
Minutes per update
Completed stale traces dropped
Count per update
The trainer waited 141.5 minutes for batches and spent 52.5 minutes on forward/backward passes during a roughly 3-hour-23-minute training window. Gradients and entropy showed no obvious numerical collapse. Mismatch KL compares rollout and training probabilities, not distance from the base model. Batch reward also depends on which tasks make it into the batch, so I used held-out tasks to check whether training helped.
Fixing my own checks
My check rejected trainer step 5 beside orchestrator step 6. I had read the convention wrong: Prime’s interval save records the orchestrator’s next step. Both were at 10 in the final save. After fixing my check, I verified the metadata, model and optimizer state, scheduler progress, and shard extents. The adapter had 256 finite tensors, though I did not test a full training resume.
After 184 base evaluations completed, my audit failed because it ran outside Prime’s virtual environment without zstandard and searched the wrong directory. I switched to the native environment and official indexed-trace reader, tested that path on CPU, and continued only the adapter evaluation. The completed base results stayed untouched.
The model passed fewer held-out tasks
Both policies received eight attempts on each of 23 held-out tasks. All 368 attempts were graded, with no infrastructure errors or timeouts.
Held-out pass rate
Percent · 184 attempts per model
Passes on each held-out task
Passes out of eight · anonymized task labels
Eleven tasks declined, six improved, and six tied. With eight attempts per task, the paired-task bootstrap interval was wide: approximately −13.0 to +4.35 percentage points. The actual base/final GPUs were H200/H100 despite matching requests. I cannot claim an improvement from this run.
How attempts ended
Attempts out of 184 · categories overlap
Generated tokens per attempt
Mean token count
The adapter spent more tokens reasoning and hit limits more often, while the number of tool results barely changed. Zero-tool attempts rose from four to ten, while empty Bash calls fell from 32 to 18. That gave me no reason to blame the parser alone.
The traces also showed ordinary coding failures. One log-parser edit collapsed message whitespace:
" ".join("retry after timeout".split())
# 'retry after timeout'Another attempt saw an empty CSV return no quantiles instead of three zeros, then claimed completion without fixing it.
Why did it keep going?
Of 640 accepted training trajectories, 452 were truncated, including 253 of the 365 passing trajectories. The grader checks the workspace, so correct files can earn reward even if the agent keeps going until a limit.
I used GRPO advantages with Prime’s IPO loss, token aggregation, no GRPO length penalty, and no separate completion reward. I suspect the model had too little incentive to finish once the files were correct. That would explain some of the continued reasoning, but these traces do not prove it caused the regression.
What the reward actually checked
I used TermGrade correctness tests through Harbor, with local packaging and dependency-error checks. The separate grader received selected files from the solver. Its running processes did not carry over. A pass still only tells me that those tests passed.
All 103 tasks used separate grading and cleared stale reward files. However, the shared image defaulted to root without unprivileged Rewardkit workers, leaving submitted code able to tamper with the grader. Harbor’s guidance helped identify that hardening gap. I found no evidence it caused this regression.
References
- TermGrade: task pool and model-specific difficulty screening.
- Prime: RL at 1T Scale: asynchronous rollouts and policy lag.
- Prime training docs and the pinned source: batches, checkpoints, and native audits.
- Qwen3.5-9B: thinking, formatting, and sampling.
- Modal sandbox resources: requests, bursts, and billing.
- Harbor reward-hacking guidance: grading isolation and privilege gaps.
- mini-SWE-agent recovery and Verifiers’ Bash executor: output-limit feedback and command handling.