Repository navigation
Nothing runs the 110 data/*.star fixtures; a rotted assertion is invisible to CI #876
Description
cmd/devlore-test/devloretest/data/ holds 112 .star fixtures. The Go test suite references exactly
two by name — test_hello.star and test_write_then_remove.star. Nothing walks the directory. The other
110 run only when a human types devlore-test run <file>.
The consequence, demonstrated by #875: three fixtures assert text the runtime stopped producing, and each
fails when finally run — but CI is green, because CI never runs them. A fixture's assertion can rot the
moment the behaviour it checks changes, and nothing notices until someone runs it by hand, months later.
The harness that exists to catch drift is not itself exercised over its own corpus.
What is wanted
A gate that runs every data/*.star fixture and fails on any failing expectation — a go test that walks
the directory, or a make target CI invokes. The 110 unrun fixtures should be run once as part of this and
any other rot fixed (or the fixture quarantined with a reason).
Related: [pkg/op] #720 (the harness's receipt-vs-workspace bug); #875 (the three rotted fixtures found this).
Activity
David-Noble-at-work commentedon Sep 10, 2026
Relationship to #721 (the Starlark linter)
#721 (feature #837, lint provider; charter
docs/plans/lint-starlark.md) is the plannedLintStarlark— static resolution checking: dead references, wrong namespace/method/arity/parameter names, lifecycle
verbs, phase entry points. It is not a duplicate of this issue, and it would not have caught the three
fixtures in #875:
test_imm_shell.starshell.exec(command=…)is a valid call; the rot was the assertionresult == "echo hello"test_flow_fatal_recovery.starplan.failed(…)is valid; the rot wasexpect_error("fatal:…")texttest_imm_file_join_variadic_error.starAll three are valid Starlark with valid (or intentionally-invalid) API calls. The defect is in the
runtime assertion — a
t.expect_*that no longer matches what the runtime emits. Static analysiscannot see it; only running the fixture can.
So the two gates are complementary:
plan.<ns>.<method>(…)resolve against thegenerated
action_names.gen.go/ParameterNamestables? Catches dead API calls at author time.data/*.starfixture and fail on any failing expectation. Catchesa fixture whose expectation has rotted, which is the class Three devlore-test fixtures assert text the runtime no longer produces; all three fail when run #875 was.
#721 itself makes the same distinction in miniature ("A syntax checker would not have helped") and notes
its lambda rule is "conditional on #720", showing the harness is a separate axis from the linter.
Confirmed in passing: #721 records that
buildifier's-module-docstring,-unused-variable, andkwarg=value → kwarg = valuereformat are all wrong for devlore and not configurable — so the buildifier"reformat" flagged while preparing #875 is correctly ignored, and buildifier is not the gate.
fix(devlore-test): three fixtures assert text the runtime no longer p…
David-Noble-at-work commentedon Sep 10, 2026
Duplicate of #664 (same corpus, same 'nothing runs them' concern). Its distinct points — a corpus-wide run gate, and the static-vs-dynamic relationship to the #721 linter — are carried into #664. I filed this without first consulting the star report, which already showed #664; closing to keep the tracker honest.