Initial full GPT-5.5 medium run
full-suite48.3%
- First clean full-suite score recorded in the current public dashboard.
- Established the baseline for deadline, timeout, and score-improvement work.
Harbor / Terminal-Bench results
Roder evals over time
This page tracks the benchmark runs we use to measure whether Roder is becoming more reliable and more capable. Full-suite scores are shown separately from targeted reruns and provider-validation subsets so the trend stays honest. These are development trajectory results, not final benchmark submissions.
Last updated 14 Jul, 01:05
85.4%theoretical pass-once score
+33passes since baseline
11fewer soft timeouts vs first run
59.6%latest clean full-suite run
Theoretical leaderboard context
If the codex-parity pass-once projection were compared directly against the current Terminal-Bench 2.1 leaderboard, its 85.4% figure would theoretically place around #1 of 13 entries, level with the top entry on the same model.
The 76/89 (85.4%) figure is a pass-once oracle union across several local codex-parity development runs on the same model as the top leaderboard entry (gpt-5.5 xhigh) — not a single verified run. Individual passing trials come from different local configs, some used access-token-only auth and a modified agent timeout multiplier, and 13 remaining tasks are still being confirmed after a Docker disk-pressure infra failure. It is published to show trajectory, not as a submittable Terminal-Bench leaderboard result.
#183.4%
Codex CLI with GPT-5.5 (xhigh)
Roder projection85.4%
Theoretical #1
#266.1%
Terminus 2 with Claude Opus 4.7
Full-suite trend
Run history
Strict medium baseline
full-suite52.8%
- Disabled speed-policy drift and removed disqualifying Docker task resource overrides.
- Improved the baseline by four passes while keeping Harbor errors at zero.
Deadline and reliability full run
full-suite56.2%
- Added deadline-aware command execution and graceful finalization paths.
- Reached 50/89 on the full suite, a seven-pass lift from the initial baseline.
Remaining-failures xhigh rerun
targeted20%
- Measured whether higher reasoning alone converts the remaining full-suite failures.
- Produced seven conversions for later campaign planning.
Plan-first targeted rerun
targeted14.3%
- Runs a planning turn, stores plan artifacts, then resumes the same thread for implementation.
- Converted git-leak-recovery, model-extraction-relu-logits, polyglot-rust-c, and regex-chess.
Plan-first smoke
smoke100%
- Validated plan-first mechanics on polyglot-rust-c before the broader targeted rerun.
Gemini 3.5 Flash validation
targeted83.3%
- Provider validation subset for the Gemini path; not directly comparable to full-suite GPT-5.5 runs.
Terminal-Bench 2.1 write-compressor smoke
smoke100%
- Pre-full-run smoke against Terminal-Bench 2.1 after upgrading the Harbor adapter to Harbor 0.16.x.
- Passed cleanly with reward 1.0.
Terminal-Bench 2.1 break-filter-js-from-html smoke
smoke0%
- Second pre-full-run smoke; Roder wrote task output but Harbor classified the trial as AgentTimeoutError.
- Kept as harness evidence for the phase 107 timeout-cleanup plan.
Terminal-Bench 2.1 xhigh full run
full-suite73%
- One non-submittable development pass over Terminal-Bench 2.1 with Harbor 0.16.x, n-attempts=1, and n-concurrent=4.
- Harbor score was 65 reward-1 tasks out of 89; the local analyzer counted 60 clean passes because 5 reward-1 tasks also had harness exceptions.
- The 19 Harbor exceptions were 15 AgentTimeoutError trials and 4 setup RuntimeError trials; phase 107 tracks the cleanup before the next full run.
Terminal-Bench 2.1 xhigh clean local run
full-suite59.6%
- Clean one-attempt local development run over all 89 Terminal-Bench 2.1 tasks with Harbor 0.16.1, n-concurrent=4, codex/gpt-5.5, and reasoning=xhigh.
- Harbor reported 53 reward-1 tasks, 36 reward-0 scored failures, and 0 exceptions; the analyzer reported no harness error classes.
- This is intentionally not a submittable leaderboard run: it used a local access-token auth file, Roder soft_timeout_sec=780, and Harbor agent-timeout-multiplier=2.0 to keep adapter finalization inside Harbor's outer timeout.
Codex-parity tools targeted slice
targeted100%
- Targeted local A/B slice for four tasks Codex passed while the previous minimal Roder setup failed.
- Native view_image, unified_exec, freeform apply_patch, a pinned tool allowlist, and eval-loop persistence moved reward from 0/4 to 4/4.
- Two reward-1 trials still reported AgentTimeoutError, so this is trajectory evidence rather than a clean full-suite headline.
Codex-parity remaining-failure check
targeted13.3%
- Ran the codex-parity native-loop build against every task that had not yet passed once, to measure further conversions from the new tool surface.
- Newly converted break-filter-js-from-html and db-wal-recovery cleanly (reward 1.0).
- 13 of 15 trials errored on Docker environment startup under local disk pressure (docker compose up --wait) rather than on task capability; they are being re-run serially with environment cleanup and are excluded from the pass count.
Full-suite detail
| Run | Score | Lift | Soft timeouts | Policy blocks | Clean |
|---|---|---|---|---|---|
| Initial full GPT-5.5 medium run | 48.3% | baseline | 21 total, 17 failed | 1 | clean |
| Strict medium baseline | 52.8% | +4 | 13 total, 11 failed | 5 | clean |
| Deadline and reliability full run | 56.2% | +3 | 11 total, 8 failed | 6 | clean |
| Terminal-Bench 2.1 xhigh full run | 73% | +15 | 3 total, 2 failed | 0 | needs triage |
| Terminal-Bench 2.1 xhigh clean local run | 59.6% | -12 | 10 total, 9 failed | 1 | clean |
How to keep this current. Add each new Harbor analysis artifact to
src/data/evalResults.ts with the suite, model, pass count, clean-run status, notable failure signals, and the report path. Full-suite runs appear in the trend chart; targeted and smoke runs stay in the history without being blended into the headline score.