Model to benchmark: stealth/ox-alpha from openrouter

ox-alpha resolves 58.4% of DeepSWE (66/113) — pier's own aggregate metric. But 80% of tasks landed ≥90% of their fail-to-pass tests, and 9.7% of the benchmark was lost to tool-call format failures rather than reasoning — see Final results.

stealth/ox-alpha is live on OpenRouter. Confirmed against GET /api/v1/models (exactly one match for ox-alpha):

Self-described as "a reasoning model designed for coding, sustained agentic work, and production workloads."

Reasoning is mandatory. reasoning.mandatory: true, default_enabled: true, default_effort: "max", supported efforts ["max", "high", "low"]. It cannot be disabled — only turned down. This is the main wall-clock risk for a 113-task agentic sweep; the knob if trajectories run long is --ak reasoning_effort=<low|high|max> on the mini-swe-agent adapter.

Caveat on confidentiality: stealth models on OpenRouter are free because the provider logs prompts for evaluation. This run is not confidential.

bench.sh needs the provider-prefixed form openrouter/stealth/ox-alpha. Verified it resolves through the real code path rather than assuming:

litellm get_llm_provider  -> ('stealth/ox-alpha', 'openrouter', None, None)
pier   api key var names  -> ['OPENROUTER_API_KEY']

OPENROUTER_API_KEY is present in .env and is forwarded into the sandbox by pier/agents/installed/mini_swe_agent.py:853.

A minimal completion through litellm, exactly as mini-swe-agent issues it:

finish_reason : stop
content       : 'PONG'
usage         : 93 prompt / 34 completion, cost 0

Note: 34 completion tokens to emit 4 characters, and the provider reported reasoning_tokens: 0 despite reasoning content being present — provider-side token accounting is unreliable for this model, so don't trust its token counts for cost or effort analysis.

113 task directories in ../deep-swe/tasks. (An earlier count of 117 was wrong — ls | wc -l also counted dataset.toml, manifest.json, manifest.schema.json and a README.md symlink.)

Every one of the 113 pins a prebuilt image, e.g.

docker_image = "public.ecr.aws/d3j8x8q7/swe-bench-202605:<ext_id>-v1.1"

so the per-task Dockerfile (clone repo → pip install -r requirements.txt) never executes locally. That removes the single biggest source of flakiness in a local run.

Per-task declared resources and policy:

Scoring is fail-to-pass / pass-to-pass; reward is 1 only on a full f2p sweep.

Daytona caps sandboxes at 10 GB, but tasks request 20 GB. Running there needs --override-storage-mb 10240, and pier itself warns this "could disqualify you from leaderboard submissions." Any daytona number carries that asterisk.

Docker does not enforce storage_mb, so it honors the task spec as declared (cpus=2, memory_mb=8192, storage untouched) with no override and no warning. It is the more faithful of the two backends here.

(The probe job dirs and the single-task smoke run are not kept in the repo — only the full 113-task run is. Their findings are recorded below.)

Ran the exact docker analogue of the pre-existing daytona probe, using the nop agent (zero model calls, zero cost), same task via --sample-seed 0 --n-tasks 1.

PASS. Numbers reproduced daytona exactly:

nop changes nothing, so reward: 0 and f2p 0/108 are the expected result; the meaningful signal is p2p 1404/1404 — image pulled, container ran, collect hook fired, verifier executed. The f2p failures are the genuine AttributeError: 'Response' object has no attribute 'iter_json', i.e. the unimplemented feature, not a harness fault.

Timings:

The 47 s gap is entirely the cold image pull; daytona started from a prewarmed snapshot. Verifier time — the part that actually scales across 113 tasks — is at parity.

The probe measured +4.11 GB for the first task image and that extrapolates to 113 × 4.1 ≈ 460 GB, which would have meant pruning between tasks. That is wrong: it ignores layer sharing. Manifest comparison showed 23 of 27 layers are common (0.75 GB compressed shared base), with only ~0.03–0.09 GB unique per task.

Measured directly by pulling three more task images:

Real projection for all 113: ≈20–25 GB against 113 GB free. No pruning, no batching. The 48 s cold-pull cost is likewise one-time, not per-task.

With nop there are no model calls, so pier never created the egress-proxy sidecar — no docker-compose-egress-proxy.json, no egress-proxy/. The OpenRouter-through-proxy path inside a no-network container stayed unverified until the smoke run below.

Reviewed the wrapper before trusting it; six issues found and fixed, all verified against the installed pier source rather than assumed. Committed as 735ba19.

1. Job-name collisions. pier compares the whole JobConfig for equality and raises FileExistsError on any mismatch (pier/job.py:196). Several config-affecting inputs were absent from the job name — concurrency, storage override, task path, and any -- <pier args> passthrough — so two incompatible runs would claim one name and the second would die. Added an 8-char config_hash() suffix over exactly those inputs.
2. --tasks N was not a random sample. pier only shuffles when a seed is set (pier/models/job/config.py:135); otherwise --tasks N takes the alphabetically-first N. Seed now defaults to 0 and is always passed, and it appears in the job name for full runs too.
3. status crashed on an empty jobs dir — unmatched glob stayed literal and exited 1. Now prints (no jobs yet).
4. Misleading resume docs. Plain resume only reruns trials with no result.json; an errored trial has one, so retrying errors needs --drop <ErrorType> (pier/job.py:208).
5. Daytona credential guard rejected DAYTONA_JWT_TOKEN + DAYTONA_ORGANIZATION_ID despite its own error message offering them.
6. DAYTONA_STORAGE_MB='' didn't disable the override — :- treats empty as unset, so the documented escape hatch silently fell back to 10240. Now -.

Also added .gitignore for .env (live keys) and jobs/ (run output).

Before committing to 113 tasks, ran a single-task run with the real agent and real model (--tasks 1 --seed 0, same httpx task as both probes) to exercise the one path nop could not.

The egress path works. With a real agent, pier created the sidecar that was absent from the nop probe — egress-proxy/ and docker-compose-egress-proxy.json — and the squid container came up healthy. Its ACL is exactly right:

acl allowed_domains dstdomain "/tmp/allowed_domains.txt"   # openrouter.ai only
http_access allow authenticated allowed_domains
http_access deny all

So the sandbox stays no-network for everything except the model endpoint. Zero 429s, zero auth failures, zero proxy errors across the whole run.

The harness is fully validated — 1404/1404 p2p means no regressions, and 107 of 108 previously-failing tests now pass, so the agent genuinely implemented the feature.

reward: 0 despite 107/108 is correct, not a bug. Scoring is all-or-nothing: reward is 1 only on a complete f2p sweep. The single miss was

tests.test_json_stream.test_iter_json_ndjson_invalid_line_raises_and_closes_streaming_response
assert False, where False = <Response [200 OK]>.is_closed

i.e. ox-alpha implemented iter_json() correctly but failed to close the response when an invalid NDJSON line raises mid-stream. A real edge-case miss on the model's part, not a harness fault — which is exactly what a benchmark should be measuring.

Worth noting for interpretation: this is a strict metric. A model can be one assertion away from a solve and still score 0. partial in rewards (0.99 here) carries the finer-grained signal.

An early monitor fired on ProxyError, which looked like the egress proxy failing. It was not — ProxyError is a class name in httpx's _exceptions.py, which the agent had just cat'd. Zero occurrences in trial.log. Log-grep monitors over an agent that reads source code will match the source, not the runtime.

Launched all 113 tasks:

CONCURRENCY=4 ./bench.sh run openrouter/stealth/ox-alpha -- -r 2
job: stealth-ox-alpha__full-s0__docker__f5abcdff

Projected wall clock — initial estimate was wrong. The smoke task took 19 m 20 s, which at concurrency 4 suggested 113 / 4 x 19.3 min ≈ 9 hours. Measured throughput over the first 12 tasks is ~6.2 tasks/hour, not the ~12.6 that implies: real tasks average closer to 38 min, so the smoke task was roughly 2x faster than typical. Revised estimate: ≈17 hours.

Considered raising concurrency to recover the difference and rejected it. n_concurrent_trials is part of JobConfig, so changing it makes pier refuse to resume — the completed trials would have to be abandoned for a new job. More importantly, 8 concurrent x 2 CPUs on an 8-core host over-subscribes during the verification phase, risking flaky timing-sensitive tests and corrupted p2p numbers. Benchmark validity beats wall clock; staying at 4.

The smoke task consumed 3.55 M input tokens for one task (74 steps, context growing each step). Extrapolated across 113 tasks that is on the order of 400 M input tokens. The key is on a paid account (is_free_tier: false, no credit cap), but OpenRouter applies a daily request cap to zero-priced models. If that is hit mid-sweep it will surface as errored trials; recovery is

./bench.sh resume stealth-ox-alpha__full-s0__docker__f5abcdff --drop <ErrorType>

once the quota resets. The run is resumable by design, so a cap costs time, not results.

About 45 min in, free disk had fallen from 113 GB to 89 GB after only ~11 tasks — a burn rate of ~2.2 GB/task that projects to ~230 GB for the remaining 104, against 89 GB available. The run would have died around task ~50.

Diagnosis took two wrong turns worth recording:

1. docker system df reported 88.97 GB of build cache, which looked like the culprit. It was not — docker builder prune -f reclaimed 0 B, because that cache is almost entirely shared with image layers (SHARED: true), not additive. Only 3.3 GB was genuinely reclaimable. The headline number in docker system df double-counts shared storage.
2. The per-trial derived images looked like the leak — each trial leaves a <task>__<trialid>-main:latest at a reported ~5 GB. But reported size also includes shared layers; deleting one actually reclaims ~620 MB (plus 302 MB for its paired -pier-egress-proxy image).

So the real cost is ~0.9 GB of genuinely unique storage per trial, left behind after the trial finishes. pier deletes the containers (--delete default) but not these per-trial images.

Note this also corrects the earlier §3 projection of "≈20–25 GB, no pruning needed". That figure was measured by pulling four task images sampled alphabetically, which shared far more layers than a random 113 do — different repos mean different pip install layers. Pruning is required.

Fix: a janitor loop that removes -main and -pier-egress-proxy images for finished trials every 4 minutes. It keys off the existence of the trial's result.json, not docker ps, so a trial sitting between its agent and verifier phases (briefly no running container) is never touched. Image names are lowercased and truncated to 32 chars, so matching is done on the unique trial-id suffix.

First pass removed 11 images and took free space from 89 GB back to 92 GB. Disk now trends up rather than down.

15 passed / 22 done = 68% resolve rate, 0 errored, 0 p2p regressions.

The distribution is sharply trimodal — the binary reward hides most of it:

So 20 of 22 tasks landed ≥90% of their fail-to-pass tests, and only 2 are real failures. The headline resolve rate of 68% and the mean partial of ~0.94 are telling substantially different stories about the same runs. Both belong in the final report.

Solved so far: arcane-drift-detection-baselines, clack-async-autocomplete-options, fastapi-deprecation-response-headers, geo-shapeindex-serialization, go-git-worktree-merge-conflicts, kombu-single-active-consumer-priority, koota-query-predicates, kysely-window-grouping-helpers, mobly-grouped-test-barriers, obsidian-linter-link-format-conversion, ofetch-per-origin-circuit-breaker, pebble-durability-wait-apis, prometheus-transactional-reload-status, scc-bounded-memory-spilling, yjs-map-conflict-detection.

Note the task set is genuinely multi-language — Go (go-git), TypeScript (ts-pattern, superjson, kysely, koota, yjs, ofetch), and Python (httpx, fastapi, cattrs, mashumaro) all appear — so this is not a Python-only measurement.

Agent effort per task: 51 steps minimum, 101 median, 148 maximum. The 74-step smoke task was below median, which is why it under-predicted wall clock.

Two trials scored 0 without doing any work — pwntools-tube-multiplexing (0/73 f2p, 14 steps) and go-genai-streamed-function-args (0/6, 66 steps). Both ended the same way:

Tool call error: <error> No tool calls found in the response.
Every response MUST include at least one tool call. </error>
... (x3 consecutive)
exit: RepeatedFormatError

ox-alpha intermittently returns a response containing no tool call at all. mini-swe-agent re-prompts with format guidance, but three consecutive failures abort the episode.

Measured across the first 28 trials:

This is not being "fixed", and that is deliberate. pier launches mini-swe-agent with its stock mini.yaml, overriding only agent.cost_limit and model.model_class — the format-error threshold is the unmodified upstream default. Raising it to rescue those two trials would mean running a non-standard harness and producing numbers not comparable to anyone else's.

More importantly, the failure is real. An agentic benchmark is partly a measurement of whether a model can reliably drive a tool loop, and ~46% of episodes hitting at least one malformed turn is a genuine characteristic of this model, not a harness artifact. Suppressing it would flatter the model and misreport the result.

It does mean the headline resolve rate carries a ~7% drag from format failures rather than reasoning failures. That decomposition belongs in the final numbers, which is why it is recorded here rather than papered over.

Caveat on recovery: exception_info is null for these trials — pier counts them as completed with reward 0, not as errors. So ./bench.sh resume --drop will not retry them. They are final.

32 passed / 51 done = 63% resolve rate, 0 errored, 0 p2p regressions.

Mean partial 0.927. The trimodal shape from the 22-task checkpoint has held: 82% of attempted tasks land ≥90% of their fail-to-pass tests.

Agent effort has a long tail — median 101 steps, max now 243.

Sampled the host under load (4 trials in flight):

load average 3.90 / 8 cores
per-container CPU:  0.16%   3.26%   36.43%   198.65%
per-container mem:  47MiB   53MiB   110MiB   3.35GiB    (cap 8GiB each)
host mem: 2 GB used / 30 GB

Typically only one trial is CPU-bound (saturating its 2-CPU limit) while the others idle waiting on the model API, and memory is nowhere near its cap. So there is real headroom — concurrency 6 would fit comfortably on this hardware and is the recommended setting for a future run.

It was not raised mid-run, because n_concurrent_trials is part of JobConfig and pier refuses to resume a job whose config changed. Raising it means a new job that re-runs all 113 from zero, discarding completed work:

Restarting is strictly worse, before even counting the risk that CPU contention during verification makes timing-sensitive tests flaky and corrupts p2p. A second parallel job over the not-yet-started tasks was also considered and rejected — the main job's task list is fixed at 113, so it would re-run them anyway.

Task image sizes vary far more than the early sample suggested: 4.5 GB to 10.8 GB (boa-hierarchical-evaluation-cancellation is the largest, prometheus-typed-label-sorting 7.76 GB). Free space therefore swings with which tasks happen to be in flight, not with accumulation — a dip to 86 GB recovered to 105 GB once the large images were reclaimed. This is why the early "≈20–25 GB total" projection was wrong in both directions.

All 113 trials finished. n_running_trials: 0, n_pending_trials: 0.

Job: stealth-ox-alpha__full-s0__docker__f5abcdff Wall clock: 20 h 39 m 38 s (2026-08-21 08:12:57 → 2026-08-22 04:52:36)

reward is 1 only when every f2p test passes and no p2p test regresses. Decomposed:

80% of tasks (90/113) landed ≥90% of their fail-to-pass tests. Ten of the near-misses failed on a single test.

The two regression-only cases are worth calling out separately, because they are not "almost solved" in the same sense — the feature worked, something else broke:

gql-incremental-graphql-delivery   f2p 17/17    p2p  810/811    reward 0
vulture-persistent-analysis-cache  f2p 24/24    p2p  291/295    reward 0

And the sharpest illustration of the metric's strictness:

meriyah-explicit-resource-declarations   f2p 48/49   p2p 51468/51469   reward 0

Two failing assertions out of ~51,500 scores identically to a model that did nothing.

./bench.sh status reports 65 and a naive reward >= 1 count reports 66. The difference is exactly one trial:

valibot-recursive-schema-composition   AgentTimeoutError   reward=1   f2p 10/10   p2p 209/209

It exceeded the 5400 s agent budget, but the repo state at the moment it was killed passed every test. So it solved the task and never emitted a clean termination.

pier itself settles this. Its job-level aggregate reports:

reward: 0.584070796460177   ->  x 113 = exactly 66
n_errors: 5                     (separate metadata, not deducted)

pier averages reward across all 113 trials and counts this one as a solve, reporting the error count independently. 66/113 = 58.4% is therefore the canonical, leaderboard-comparable number and is used as the headline.

On the merits this is also right: the agent was killed at 5400 s, so nothing after the cutoff contributed. The verifier ran against a repo state produced entirely within budget and it passed 10/10 f2p and 209/209 p2p. The task was solved inside the allowed time; only the termination signal was missing. Penalizing it would score the exit protocol, not the work.

The 65 came from a bug in bench.sh's own status helper, which hit continue on exception_info before ever reading the reward — undercounting any errored-but-passing trial. Fixed; ./bench.sh status now reports 58.4% and agrees with pier.

Not all 47 non-solves are the same kind of failure:

The 5 timeouts were all long-horizon tasks (165–265 steps): eicrud-keyset-pagination-cursor, scriggo-method-declarations, opa-template-string-reconstruction, valibot-recursive-schema-composition, oxvg-structural-selector-preservation. pier's default exclude_exceptions correctly kept -r 2 from retrying them — retrying a timeout would hand the model wall clock the benchmark does not grant.

The 11 RepeatedFormatError trials are the most actionable finding. That is 9.7% of the whole benchmark lost to ox-alpha emitting responses containing no tool call, three times in a row, rather than to any reasoning deficit. Left unmitigated deliberately — pier runs mini-swe-agent's stock mini.yaml, and tool-call reliability is a legitimate thing for an agentic benchmark to measure. But it means the 58.4% headline understates the model's reasoning ability while correctly reporting its agentic ability.

The 96.1% cache hit rate is what makes a 20-hour agentic sweep tractable; without prompt caching the input volume would be prohibitive. Note the earlier estimate of "~400 M input tokens" was low by 2.7x.

Across 20 h 39 m and 113 trials: no rate limiting, no auth failures, no proxy failures, no OOM, no disk exhaustion, no leaked containers. The only 5 errored trials were the model exceeding its own task budget. Every one of the 113 tasks produced a verdict.

1. Single attempt per task (n_attempts: 1). Given the intermittent tool-call failures, per-task outcomes carry real run-to-run variance. A -k 3 run would separate model stochasticity from task difficulty.
2. The 95% CI [49.2%, 67.1%] is for generalizing beyond this suite. The 58.4% itself is exact — all 113 tasks ran, nothing was sampled.
3. Not confidential. Stealth models on OpenRouter are free because the provider logs prompts.
4. Reasoning effort was left at the model's default (max), which is also its most expensive setting in wall clock. A --ak reasoning_effort=low run would likely trade some accuracy for far fewer timeouts.

• CONCURRENCY=6 — measured headroom (load 3.90/8, memory ~1 GB/container against an 8 GB cap) says 4 was conservative.
• Keep the disk janitor; it is required, not optional.
• Consider -k 3 for tighter per-task estimates.