Skip to content

docs: plan for running every test on every platform - #374

Merged
David-Noble-at-work merged 1 commit into
developfrom
docs/platform-test-matrix
Aug 12, 2026
Merged

David-Noble-at-work merged 1 commit into
developfrom
docs/platform-test-matrix

Conversation

@David-Noble-at-work

Copy link
Copy Markdown
Contributor

The plan for issue #373.

The gap

quality-gate runs make test-race across all 121 packages — on ubuntu-latest alone. The three-platform scenario matrix runs go test -run TestWritDeployScenario ./cmd/writ: one test, one package. macOS and Windows execute nothing else.

Rulings

  1. -race on every leg, not Linux only. Windows needs CGO_ENABLED=1 and the runner's mingw-w64; that cost is accepted.
  2. Non-blocking first — the matrix job lands with continue-on-error: true so triage gets real data without halting branches in flight. Phase 4 removes it and makes the legs required; that phase is mandatory.
  3. quality-gate stays on ubuntu-latest — macOS bills at 10× and shell-lint installs shellcheck via apt.

What ruling 3 exposed

go vet and golangci-lint honor build constraints, so choosing a platform for quality-gate does not make it analyze the tree — it only changes which files are skipped. Verified with go list on pkg/platform:

GOOS Sees
linux detect_linux.go, linux_managers_linux.go
darwin detect_darwin.go, darwin_managers_darwin.go
windows detect_windows.go, windows_managers_windows.go

The package uses a real/stub pattern, so on CI today the Darwin package-manager stub is linted and the real Darwin package manager is not. Ten platform-gated files are unvetted and unlinted per PR; their only compilation anywhere is make dist on release tags.

Phase 1b adds per-GOOS vet, lint, and build sweeps — three invocations on the one ubuntu runner, no extra cost.

What reduces the risk

test-scenario → build → generate, so the existing scenario job already runs the full codegen chain on all three platforms, green. test-race: generate inherits a proven prerequisite chain. And TestWritDeployScenario is symlink-based deployment passing on windows-latest, so symlink creation works on that runner.

Measured risk

Signal Count
Hardcoded /tmp, /usr, /etc, /bin 66
Permission-bit assertions 66
Expected values containing / 14
os.Symlink test files 9
exec.Command test files 6
runtime.GOOS == "windows" in production code 0

The zero is the most informative: no Windows runtime branching exists anywhere, so whatever the matrix surfaces is real behavior, not an absent conditional.

The rule phase 2 enforces

Every failure lands in exactly one bucket — a genuine platform defect, a Unix-assuming test, or a test whose subject only exists on one platform. The third moves into a build-tagged _linux_test.go, which is scoping, not skipping. A t.Skip added to turn a leg green is bucket one or two in disguise.

Documentation only; no code changes.

The plan for issue #373. The unit suite runs on ubuntu-latest only: 121
packages there, and exactly one test on macOS and Windows, since the
three-platform scenario job runs `go test -run TestWritDeployScenario
./cmd/writ` and nothing else.

Three rulings are recorded. The race detector runs on every leg, not Linux
only. The matrix job lands non-blocking so triage gets real data without
halting branches in flight, with a mandatory phase that removes
continue-on-error and makes the legs required. quality-gate stays on
ubuntu-latest — macOS bills at 10x and the shell-lint step installs shellcheck
via apt.

That last ruling exposed a second gap the plan now covers. go vet and
golangci-lint both honor build constraints, so choosing a platform for
quality-gate does not make it analyze the whole tree — it only changes which
files are skipped. Verified with go list: in pkg/platform, ubuntu sees
detect_linux.go and linux_managers_linux.go while detect_darwin.go,
darwin_managers_darwin.go, detect_windows.go and windows_managers_windows.go
are invisible. The real Darwin package manager is never linted on CI; its
non-Darwin stub is. Ten platform-gated files are unvetted and unlinted on every
pull request, and their only compilation anywhere is make dist's cross-compile
on release tags. Phase 1b adds per-GOOS vet, lint, and build sweeps — three
invocations on the one ubuntu runner, no additional cost.

Two facts reduce the risk materially. test-scenario depends on build, which
depends on generate, so the existing scenario job already runs the full
codegen chain on all three platforms and is green; test-race inherits a
prerequisite chain that is already proven. And TestWritDeployScenario is
symlink-based deployment passing on windows-latest, so symlink creation works
on that runner.

Measured risk across the test tree: 66 hardcoded /tmp, /usr, /etc or /bin
references; 66 permission-bit assertions; 14 expected values containing forward
slashes; 9 test files calling os.Symlink; 6 shelling out; 2 already skipping on
Windows; and zero runtime.GOOS == "windows" branches in production code. That
zero matters most — the codebase carries no Windows runtime branching at all,
so whatever the matrix surfaces is real behavior rather than an absent
conditional.

Phase 2 classifies every failure into exactly one of three buckets: a genuine
platform defect, a Unix-assuming test, or a test whose subject only exists on
one platform. The third moves into a build-tagged _linux_test.go file, which is
scoping rather than skipping. A t.Skip added to turn a leg green is bucket one
or two in disguise.

Documentation only; no code changes.
Issue: #373
@David-Noble-at-work David-Noble-at-work self-assigned this Aug 12, 2026
@David-Noble-at-work
David-Noble-at-work merged commit 1980e22 into develop Aug 12, 2026
5 checks passed
@David-Noble-at-work
David-Noble-at-work deleted the docs/platform-test-matrix branch August 12, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant