Repository navigation
ci: run the unit suite on every platform - #375
Merged
Merged
Conversation
Phase 1 of issue #373. The unit suite ran in quality-gate on ubuntu alone — 121 packages there — while macOS and Windows ran only `go test -run TestWritDeployScenario ./cmd/writ`, a single test in a single package. A green scenario leg attested to one writ deploy, not to the suite. A new `test` job carries a three-platform matrix running `make test-race`, and the Test step leaves quality-gate; the matrix's ubuntu leg covers it. Lint, go mod tidy, frontmatter validation, shell lint, and code metrics stay on the one ubuntu runner — they are platform-invariant and should not run three times. CGO_ENABLED is set explicitly: -race requires cgo and a C toolchain, the default varies by platform and by whether a compiler is discoverable, and the windows runner ships mingw-w64. The plan specified continue-on-error for this phase. That was corrected during implementation and the plan is updated in the same commit. The develop ruleset (12426847) requires exactly one status check, quality-gate, so a job absent from that list cannot block a merge whatever it reports — continue-on-error would add nothing except suppressing the red, and the red IS the phase 2 triage data. The job therefore lands without it. Phase 4 reduces to a ruleset change with no further edit to this file. Recorded from the same query: the existing scenario macos and windows legs are not required checks either, so a red Windows scenario does not block a merge today. Two facts make this cheaper than it looks. test-scenario depends on build, which depends on generate, so the existing scenario job already runs the whole codegen chain on all three platforms and is green — test-race inherits a prerequisite chain that is already proven, including make and bash working on the windows runner. And TestWritDeployScenario is symlink-based deployment passing on windows-latest, so symlink creation works there. Expect the first runs to be red on macOS and Windows. Those are pre-existing defects being revealed, not regressions introduced here; phase 2 classifies every one of them and phase 3 fixes them. A t.Skip added to turn a leg green is not an outcome this plan permits. Verified: actionlint clean on ci.yaml. Plan: docs/plans/platform-test-matrix.md (phase 1). Issue: #373
David-Noble-at-work
added a commit
that referenced
this pull request
Aug 12, 2026
) Phase 3 of issue #373, taking the largest cluster first. The first three-platform run (PR #375, commit f8b28c8) passed on macOS and ubuntu and failed on Windows with 63 --- FAIL lines. Roughly 37 of them were one bug. `go build -o` appends .exe on Windows, so the binary landed at devlore-test.exe while TestMain kept exec'ing devlore-test. Every invocation failed to start, reporting `exit code = -1` with empty output — which is why all 30 TestCLI_* cases failed at 0.00s, along with the 5 "no JSON summary" and 2 "missing Hello World!" assertions that depend on the same process running. The plan's phase 2 triage is recorded in the same commit. Two genuine product defects were found and filed rather than patched inline, both invisible to the previous ubuntu-only gate: #376 — Starlark extension loading breaks on any Windows path. C:\Users\... interpolated into Starlark source fails on \U, an invalid escape. #377 — fsroot.Path.Rel() returns OS-native separators, and Path is a serialized type, so documents and their checksums differ between platforms. The six failing assertions are correct; the product is wrong, and it contradicts this repository's own platform-stable digest rule. Remaining Windows failures are permission-bit assertions, a hardcoded /dev/null, and two needing a closer read. Those counts are a lower bound: ~37 masked failures may have been concealing others, and the next Windows run reveals what was behind them. No test is skipped and no build tag is added to dodge a platform. The fix is one conditional suffix in the test's own setup. Verified on Darwin: make vet, full make test green — zero FAIL or panic lines by count; gofmt clean. The changed path is Windows-only, so CI's windows leg is the real verification. Plan: docs/plans/platform-test-matrix.md (phase 2 recorded, phase 3 begun). Issue: #373 Co-authored-by: David Noble <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of issue #373.
What changes
A new
testjob runsmake test-raceon a three-platform matrix. TheTeststep leavesquality-gate— the matrix's ubuntu leg covers it. Lint,go mod tidy, frontmatter validation, shell lint, and code metrics stay on the one ubuntu runner; they are platform-invariant and should not triple.CGO_ENABLED: "1"is explicit:-raceneeds cgo and a C toolchain, the default varies by platform and by whether a compiler is discoverable, and the windows runner ships mingw-w64.A correction to the plan, in the same commit
The plan specified
continue-on-error: truefor this phase. That is both unnecessary and harmful, and the plan is updated here.Ruleset
12426847ondeveloprequires exactly one status check —quality-gate. A job absent from that list cannot block a merge whatever it reports. Socontinue-on-errorwould add nothing except suppressing the red — and the red is the phase-2 triage data. The job lands without it: failing legs report as failures, visibly, and still block nothing.Phase 4 therefore reduces to a ruleset change with no further edit to
ci.yaml.Recorded from the same query: the existing
scenario (macos-latest)andscenario (windows-latest)legs are not required checks either. A red Windows scenario does not block a merge today.Why this is cheaper than it looks
test-scenario→build→generate, so the existing scenario job already runs the whole codegen chain on all three platforms, green — includingmakeand bash working on the windows runner.test-race: generateinherits a proven chain. AndTestWritDeployScenariois symlink-based deployment passing on windows-latest, so symlink creation works there.Expect red
The first runs will likely fail on macOS and Windows. Those are pre-existing defects being revealed, not regressions introduced here. Measured signals across the test tree: 66 hardcoded
/tmp|/usr|/etc|/binreferences, 66 permission-bit assertions, 14 expected values containing/, 9 files callingos.Symlink, 6 shelling out — and zeroruntime.GOOS == "windows"branches in production code.Phase 2 classifies every failure as a genuine platform defect, a Unix-assuming test, or a test whose subject exists on one platform only. The third moves into a build-tagged
_linux_test.go, which is scoping. At.Skipadded to turn a leg green is not an outcome this plan permits.Verified:
actionlintclean onci.yaml.