Skip to content

Codegen runs every time, and its output must match what is committed - #705

Merged
David-Noble-at-work merged 4 commits into
developfrom
deterministic-codegen
Aug 27, 2026
Merged

David-Noble-at-work merged 4 commits into
developfrom
deterministic-codegen

Conversation

@David-Noble-at-work

Copy link
Copy Markdown
Contributor

Closes #702.

Codegen is mtime-gated, and a fresh clone stamps every file at checkout — so whether a provider looks
stale is decided by write order rather than by its source. Measured on run
33024633546, one commit across
six jobs
:

Job Files regenerated Providers
test (darwin-arm64) 159 30
quality-gate 9 1
test (linux-amd64) 9 1
test (linux-arm64) 0 0
test (windows-amd64) 0 0
test (windows-arm64) 0 0

Nothing varied but which runner took the job. Half ran no provider generator at all, and every
generator validation rides along with that — so which defects get caught was luck.

#697 has the receipt: darwin regenerated goast, caught a duplicate declaration through
validate_action_name_consts, and failed. Linux skipped it and passed. Neither declaration was
build-tagged.

Underneath it, a second hole

Generated files are committed, CI never ran generate explicitly, and nothing asserted that
generation produced no diff
. So a stale committed .gen.go passed outright — and when codegen did
run, CI tested the regenerated output rather than the committed output. The artifact under test was not
the artifact in the repository, and the rewrite happened inside the build where nobody saw it.

The fix already existed in the same workflow file

go.mod is guarded the way generated files were not: run the tool, assert nothing changed.

- name: Generated files are current
  run: |
    make regenerate
    git diff --exit-code

make regenerate forces every rule; the diff holds the committed tree to it.

Touching sources, not deleting outputs

The obvious lever was rm -f $(NEW_OP_INVENTORY) — one output per provider, each the head of a grouped
rule, list already maintained. It cannot work. The generated files hold the op.Announce*() call
sites devlore-inventory scans, inventory is a prerequisite of the star binary, and
build/star: FORCE rebuilds star every run. The outputs are also inputs, so deleting them destroys the
bootstrap that would regenerate them. Confirmed by doing it: inventory found 8 packages instead of 19,
then failed with no op.Announce*() call sites found.

Deletion would be safe with an LKG present — STAR resolves to build/star.lkg when it exists and
build/star: FORCE is never consulted. Not taken: CI has no LKG, and a check that depends on an
artifact CI lacks is not a check.

Proved it can fail before trusting it

A check that has only ever passed is not a check. Removing one +devlore:claim directive from ui.Note
changed pkg/op/provider/ui/gen/provider.gen.go, which the diff catches. Restored byte-identical after.

Worth recording: the first probe edited a method doc comment and changed nothing — doc comments do
not reach the generated artifacts. Stopping there would have concluded the check was broken.

One leg is the right scope

Generated output is platform-independent — it has to be, since the repository holds one copy. Confirmed
rather than assumed: regenerating all 159 files on darwin reproduced the committed tree byte for byte.

A deletion the generator no longer emits also fails this check — the #572 class of bug, caught for free.

Verified

make regenerate 159 files, 30 providers — matches the darwin leg exactly
make generate 0 files — the mtime gate still skips unchanged providers
make check pass
make test-race pass
make test-scenario pass

Also included

One unrelated commit that rode along from a branch switch: the CheckGOOS doc comment still argued for
the four-platform sweep #697 removed, and misdescribed CI as running one platform.

David Noble added 4 commits August 26, 2026 17:44
…at was removed

Narrowing the test to one platform per CI leg left the doc comment on CheckGOOS making
the case for the four-platform sweep it replaced:

  Verifying only the developer's platform would leave the others to CI, and CI runs
  one of them.

That is now both stale and an argument against the design around it -- a reader looking
for why the sweep is gone finds a paragraph explaining why it should not be. It also
misdescribes CI, which runs five legs spanning three platforms rather than one of them.

Rewritten to say where the coverage actually comes from: each leg checks its own host,
the legs span darwin, linux, and windows between them, and an explicit goos is for
sweeping the rest from one machine.
Codegen is mtime-gated, and a fresh clone stamps every file at checkout -- so whether a
provider looks stale is decided by write order, not by its source. Measured on CI run
33024633546, ONE commit across six jobs:

  test (darwin-arm64)    159 files   30 providers
  quality-gate             9 files    1 provider
  test (linux-amd64)       9 files    1 provider
  test (linux-arm64)       0 files    0 providers
  test (windows-amd64)     0 files    0 providers
  test (windows-arm64)     0 files    0 providers

Nothing varied but which runner took the job. Half of them ran no provider generator at
all, and every generator validation rides along with that -- so which defects get caught
was luck. #697 has the receipt: darwin regenerated goast, caught a duplicate declaration
through validate_action_name_consts, and failed; linux skipped it and passed. Neither
declaration was build-tagged.

Touching sources, not deleting outputs
  The obvious lever is `rm -f $(NEW_OP_INVENTORY)` -- one output per provider, each the
  head of a grouped rule, list already maintained. It cannot work. The generated files
  hold the op.Announce*() call sites devlore-inventory scans, inventory is a prerequisite
  of the star binary, and `build/star: FORCE` rebuilds star every run. THE OUTPUTS ARE
  ALSO INPUTS, so deleting them destroys the bootstrap that would regenerate them. Tried
  it: inventory found 8 packages instead of 19, then failed with `no op.Announce*() call
  sites found`.

  Deletion would be safe with an LKG present -- STAR resolves to build/star.lkg when it
  exists and build/star: FORCE is never consulted. Not taken: CI has no LKG, and a check
  that depends on an artifact CI lacks is not a check.

  `make -B generate` also forces the rules and forces far more than the question needs,
  propagating into the build/star sub-make.

Verified
  make regenerate   159 files, 30 providers -- matches the darwin leg exactly
  make generate       0 files -- the mtime gate still skips unchanged providers
  check / test-race / test-scenario   all pass
…duces

Generated files are committed, CI never ran `generate` explicitly, and no step asserted
that generation produced no diff. Two consequences, both live until now:

  A stale committed .gen.go passed CI. Edit a provider.go, forget to regenerate, commit
  -- if the mtimes fell the wrong way, nothing noticed.

  When codegen DID run, CI tested the regenerated output rather than the committed
  output. The artifact under test was not the artifact in the repository, and the
  rewrite happened inside the build where nobody saw it.

The fix already existed in this file, applied to go.mod: run the tool, then assert
nothing changed. `make regenerate` forces every rule; `git diff --exit-code` holds the
committed tree to it.

One leg is the right scope
  Generated output is platform-independent -- it has to be, since the repository holds
  one copy. Confirmed rather than assumed: regenerating all 159 files on darwin
  reproduced the committed tree byte for byte. Forcing regeneration on quality-gate
  makes the generator validations fire deterministically there, and the diff makes every
  other leg's mtime-gated build safe, because the committed files are then known current.

A deletion the generator no longer emits also fails this check -- the #572 class of bug,
caught for free.

Proved it can fail before trusting it
  A check that has only ever passed is not a check. Removing one +devlore:claim directive
  from ui.Note changed pkg/op/provider/ui/gen/provider.gen.go, which the diff catches.
  Restored byte-identical afterward.

  Worth recording: the first probe edited a method doc comment and changed NOTHING --
  doc comments do not reach the generated artifacts. Stopping there would have concluded
  the check was broken.
@David-Noble-at-work David-Noble-at-work self-assigned this Aug 27, 2026
@David-Noble-at-work
David-Noble-at-work merged commit 0583e1e into develop Aug 27, 2026
12 checks passed
@David-Noble-at-work
David-Noble-at-work deleted the deterministic-codegen branch August 27, 2026 01:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codegen runs only when Make thinks it is stale, so CI validations fire nondeterministically

1 participant