Repository navigation
docs: plan for running every test on every platform - #374
Merged
Merged
Conversation
The plan for issue #373. The unit suite runs on ubuntu-latest only: 121 packages there, and exactly one test on macOS and Windows, since the three-platform scenario job runs `go test -run TestWritDeployScenario ./cmd/writ` and nothing else. Three rulings are recorded. The race detector runs on every leg, not Linux only. The matrix job lands non-blocking so triage gets real data without halting branches in flight, with a mandatory phase that removes continue-on-error and makes the legs required. quality-gate stays on ubuntu-latest — macOS bills at 10x and the shell-lint step installs shellcheck via apt. That last ruling exposed a second gap the plan now covers. go vet and golangci-lint both honor build constraints, so choosing a platform for quality-gate does not make it analyze the whole tree — it only changes which files are skipped. Verified with go list: in pkg/platform, ubuntu sees detect_linux.go and linux_managers_linux.go while detect_darwin.go, darwin_managers_darwin.go, detect_windows.go and windows_managers_windows.go are invisible. The real Darwin package manager is never linted on CI; its non-Darwin stub is. Ten platform-gated files are unvetted and unlinted on every pull request, and their only compilation anywhere is make dist's cross-compile on release tags. Phase 1b adds per-GOOS vet, lint, and build sweeps — three invocations on the one ubuntu runner, no additional cost. Two facts reduce the risk materially. test-scenario depends on build, which depends on generate, so the existing scenario job already runs the full codegen chain on all three platforms and is green; test-race inherits a prerequisite chain that is already proven. And TestWritDeployScenario is symlink-based deployment passing on windows-latest, so symlink creation works on that runner. Measured risk across the test tree: 66 hardcoded /tmp, /usr, /etc or /bin references; 66 permission-bit assertions; 14 expected values containing forward slashes; 9 test files calling os.Symlink; 6 shelling out; 2 already skipping on Windows; and zero runtime.GOOS == "windows" branches in production code. That zero matters most — the codebase carries no Windows runtime branching at all, so whatever the matrix surfaces is real behavior rather than an absent conditional. Phase 2 classifies every failure into exactly one of three buckets: a genuine platform defect, a Unix-assuming test, or a test whose subject only exists on one platform. The third moves into a build-tagged _linux_test.go file, which is scoping rather than skipping. A t.Skip added to turn a leg green is bucket one or two in disguise. Documentation only; no code changes. Issue: #373
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The plan for issue #373.
The gap
quality-gaterunsmake test-raceacross all 121 packages — on ubuntu-latest alone. The three-platformscenariomatrix runsgo test -run TestWritDeployScenario ./cmd/writ: one test, one package. macOS and Windows execute nothing else.Rulings
-raceon every leg, not Linux only. Windows needsCGO_ENABLED=1and the runner's mingw-w64; that cost is accepted.continue-on-error: trueso triage gets real data without halting branches in flight. Phase 4 removes it and makes the legs required; that phase is mandatory.quality-gatestays on ubuntu-latest — macOS bills at 10× and shell-lint installsshellcheckviaapt.What ruling 3 exposed
go vetandgolangci-linthonor build constraints, so choosing a platform forquality-gatedoes not make it analyze the tree — it only changes which files are skipped. Verified withgo listonpkg/platform:detect_linux.go,linux_managers_linux.godetect_darwin.go,darwin_managers_darwin.godetect_windows.go,windows_managers_windows.goThe package uses a real/stub pattern, so on CI today the Darwin package-manager stub is linted and the real Darwin package manager is not. Ten platform-gated files are unvetted and unlinted per PR; their only compilation anywhere is
make diston release tags.Phase 1b adds per-
GOOSvet,lint, andbuildsweeps — three invocations on the one ubuntu runner, no extra cost.What reduces the risk
test-scenario→build→generate, so the existing scenario job already runs the full codegen chain on all three platforms, green.test-race: generateinherits a proven prerequisite chain. AndTestWritDeployScenariois symlink-based deployment passing on windows-latest, so symlink creation works on that runner.Measured risk
/tmp,/usr,/etc,/bin/os.Symlinktest filesexec.Commandtest filesruntime.GOOS == "windows"in production codeThe zero is the most informative: no Windows runtime branching exists anywhere, so whatever the matrix surfaces is real behavior, not an absent conditional.
The rule phase 2 enforces
Every failure lands in exactly one bucket — a genuine platform defect, a Unix-assuming test, or a test whose subject only exists on one platform. The third moves into a build-tagged
_linux_test.go, which is scoping, not skipping. At.Skipadded to turn a leg green is bucket one or two in disguise.Documentation only; no code changes.