Repository navigation
Codegen runs only when Make thinks it is stale, so CI validations fire nondeterministically #702
Description
Epic: #452 — Code quality, coverage, and the lint gates
Codegen runs when Make thinks a target is stale. On a fresh CI checkout that is not a deterministic
question, so whether a given provider regenerates on a given CI leg is luck — and when it does not
regenerate, every validation the generator carries is silently skipped.
The mechanism
Each provider's generation is a grouped target gated on source mtime:
$(SP)/goast/gen/receiver_type.gen_test.go \
$(SP)/goast/gen/module.gen_test.go \
$(SP)/goast/gen/provider.gen.go &: $(SP)/goast/provider.go | $(STAR)
$(STAR) devlore actions generate --source=$(SP)/goast --gen=true --write=true --output=$(SP)/goastgit clone stamps every file at checkout time. Whether provider.go reads as newer than
gen/provider.gen.go comes down to write order and timestamp resolution, and git's parallel checkout
makes that order nondeterministic. Nothing about the source decides it.
It has already produced a false negative
On #697, goast declared LoadSourceFile twice — a *Provider method and a package-level function.
Action-name consts live in the package root, so emitting one for the method collides with the function
and does not compile. validate_action_name_consts exists to catch exactly that.
| Leg | Regenerated goast? |
Result |
|---|---|---|
| darwin-arm64 | yes | failed, correctly — generate.star:357 in validate_action_name_consts |
| linux-amd64 | no | passed — the validation never ran |
Neither declaration is build-tagged. Same source, same defect, opposite verdicts, and the green one was
wrong. Forcing staleness across every provider locally confirmed the defect was real and unique: 158
codegen invocations, no other collisions.
The larger hole: nothing checks that committed generated files are current
Generated files are committed — .gitignore excludes none of them. CI never runs generate explicitly,
only as a dependency of build / test / test-race. And no step asserts that generation produced no
diff. Two consequences:
- A stale committed
.gen.gocan pass CI. Edit aprovider.go, forget to regenerate, commit — if
the mtimes fall the wrong way, nothing notices. - When codegen does run, CI tests the regenerated output, not the committed output. The artifact
under test is not the artifact in the repository, and the divergence is invisible because the rewrite
happens inside the build.
The fix already exists in the same workflow
go.mod is guarded exactly the way generated files are not (.github/workflows/ci.yaml:52):
- name: Check go.mod tidy
run: |
go mod tidy
git diff --exit-code go.mod go.sumRun the generator, then assert nothing changed. Applied here it makes the question deterministic by
construction — generation always runs, and the committed output must match it:
- Make codegen unconditional in CI rather than mtime-gated, so generator validations always fire
git diff --exit-codeover the generated paths afterward, so committed output must equal generated
output- Keep the mtime-gated path for local iteration, where skipping unchanged providers is the point
Acceptance
- Every provider's generator runs on every CI leg, regardless of checkout mtimes
- A stale committed
.gen.gofails CI - A generator validation failure fails CI on every leg, not on whichever leg happened to regenerate
- Local
make generatekeeps skipping unchanged providers
Activity
David-Noble-at-work commentedon Aug 26, 2026
Mechanism decided — the issue named the shape but not the how, which left the real decision hidden. Plan:
docs/plans/deterministic-codegen.md.Rejected —
make -B generate.--always-makepropagates into thebuild/star: FORCEsub-make and rebuilds targets unrelated to codegen.Rejected — a
FORCEprerequisite on the 30 grouped rules. They would then regenerate always, including locally, which costs the local skip outright.Chosen — delete the outputs, then generate.
NEW_OP_INVENTORYalready enumerates one output per provider, and each is the head of a grouped rule, so removing the heads makes all 30 rules unconditionally stale and each re-emits its full group. The existing variable does the enumeration; no second list to keep in sync.Then in
quality-gate, beside thego.modcheck it mirrors:One leg is the right scope. Generated output is platform-independent — it must be, since the repo holds one copy — so currency is not a per-platform question. Forcing regeneration on
quality-gatemakes validations fire deterministically there, and the diff check makes every other leg's mtime-gated build safe, because the committed files are then known current.A deletion the generator does not recreate also fails the check — the #572 class of bug, caught for free.
The step that matters is proving it fails. A check that cannot fail is not a check: before trusting it, edit a
provider.go, skip regeneration, and confirm the diff check catches it. Second guard: count generator invocations and expect 30, so arm -fthat misses outputs cannot leave the nondeterminism in place in a form that now looks handled.One open question: does anything generated legitimately differ by platform? If some output embeds a path separator or a GOOS-dependent list, the diff check fails on the quality-gate runner for a legitimate reason and the scope widens.
David-Noble-at-work commentedon Aug 27, 2026
Measured on run 33024633546 — one commit, one run, six jobs, counting
[star] [✔] Wrotelines per job:quality-gatetest (darwin-arm64)test (linux-amd64)test (linux-arm64)test (windows-amd64)test (windows-arm64)Codegen coverage ranged from 0 to 30 providers with nothing varying but which runner picked up the job. Half the jobs ran no provider generator at all — their entire generate phase was:
inventoryruns because it is a plain recipe with no mtime gate. The 30 grouped rules are gated, and on three of six jobs not one of them fired.This is stronger evidence than the darwin/linux split the issue was opened on. That looked like a platform difference; it is not.
validate_action_name_constsand every other generator validation ran for 30 providers on one job, 1 provider on two jobs, and 0 providers on three. Whether a defect is caught depends on which job happens to regenerate the provider that carries it.Two further consequences worth recording:
quality-gateis not exempt. It regenerated exactly one provider. Putting the currency check there without forcing regeneration first would check almost nothing — which is why step 2 of the plan counts invocations and expects 30 rather than trusting thatmake regeneratedid its job.The current PR passes on trust, not verification. darwin happened to regenerate all 30 and the tree stayed green, so the committed generated files do appear current. Nothing asserted that; it was luck, and on a run where darwin drew 0 the same tree would have passed identically with stale files.
Also corrected from the earlier writeup:
grep "devlore actions generate"returns 0 on every job and is not evidence of anything — make does not echo the recipe. The generator announces itself as[star] [✔] Wrote <path>, which is what the counts above use.ui takes the root namespace and replaces print and fail (#708)