Repository navigation
devlore-test run fails on a quarter of the fixtures the Go harness passes #664
Description
devlore-test run <fixture> cannot execute roughly a quarter of the fixture suite — the same fixtures pass under the Go harness.
Measured 2026-08-24 by running every fixture through the built binary:
ran=104 mmap-failures=12 other-failures=12
24 of 104. Every one of them passes via make test, which drives the same files through runScript in cmd/devlore-test/devloretest/runner_test.go.
Symptom 1 — the content store is absent (12)
Error: writing receipt: op.Graph: pack tag:…:sha256:0947b60f…#…/provider/function.Resource:
mmap …/.devlore/function/resource/sha256/09/0947b60f…: no such file or directory
Affected: test_choose_exists, test_choose_lambdas, test_choose_literals, test_choose_not_exists, test_choose_predicates, test_choose_unchosen_branch, test_function_call_walk_tree, test_is_dir, test_is_file, test_judgment_scoped_claims, test_wait_until, test_walk_tree_planned.
The common factor is a function resource in the graph. Its bytecode belongs in the content store, and the store the binary sets up does not have it by the time the receipt is written.
Symptom 2 — callables rejected outright (3 of the other 12)
test_walk_tree, test_walk_tree_closure, test_walk_tree_gitignore:
Error: executing script: param fn
Immediate mode this time, but the same ingredient: a Starlark callable passed as a parameter.
Symptom 3 — star extensions unresolved (6)
test_imm_shell, test_imm_staranalysis, test_imm_starcode, test_imm_starcomplexity, test_imm_starindex, test_imm_starstats fail to execute at all. Plausibly a different cause — extension resolution rather than callables — and worth confirming separately before treating it as one bug.
Why it matters
The binary is the documented way to run a fixture by hand. A contributor debugging test_choose_lambdas runs it, gets an mmap error naming a sha256 path, and has no reason to suspect their fixture is fine. It also cost real time here: I hit the mmap failure while sealing providers, spent effort deciding whether I had broken something, and only settled it by running an untouched fixture the same way.
The failure should either not happen — the binary sets up what the harness sets up — or it should say what is missing rather than surfacing as an mmap error on a hashed path.
Not a regression
Present before any of the #625 sealing work; confirmed by running test_walk_tree_planned (untouched) and watching it fail identically.
Activity
David-Noble-at-work commentedon Sep 10, 2026
Absorbing #876, which I filed as a new bug without first checking the star report — it duplicates this
issue. Closing #876 in favour of this one. Two things from it worth keeping here:
A gate, not just a fix. Beyond making
devlore-test runsucceed on every fixture (this issue), nothingruns the
data/*.starcorpus automatically — the Go suite references only a handful by name, so even thefixtures that pass today are ungated. #875 proved the cost: three fixtures asserted text the runtime had
stopped producing, all three failed when finally run by hand, and CI stayed green throughout. Whatever
fixes the run failures here should also leave behind a gate that runs the whole corpus and fails on any
failing expectation.
Relationship to the Starlark linter (#721 / feature #837). That linter is static — dead API calls,
wrong arity, wrong parameter names. It would not have caught #875's three: all were valid Starlark with
valid (or deliberately-invalid) calls, and the rot was in the runtime assertion. So the linter and this
issue are complementary axes: #721 checks that a fixture resolves; this checks that it passes when run.