Repository navigation
CI runs the unit suite on ubuntu only; macOS and Windows run a single test #373
Description
All tests must run on every supported platform. Today the unit suite runs on ubuntu-latest only, and macOS and Windows run exactly one test.
Current state
| Job | Runner(s) | Test command | Scope |
|---|---|---|---|
quality-gate |
ubuntu-latest | make test-race |
all 121 packages |
scenario |
ubuntu, macos, windows | make test-scenario |
1 test, 1 package |
knowledge-extract |
ubuntu-latest | make star |
— |
make test-scenario is go test -run TestWritDeployScenario -v -count=1 -timeout 600s ./cmd/writ (Makefile). It is the writ deploy end-to-end scenario, deliberately scoped, and it is the only thing macOS and Windows execute.
So a green scenario (windows-latest) says one writ deploy scenario passed on Windows. It says nothing about the other 120 packages.
Consequence: ten platform-gated files are unit-tested nowhere
Build constraints exclude these from the ubuntu run, and the platforms where they compile only run the scenario test:
pkg/op/default_funcs_windows.gopkg/op/provider/file/helpers_windows.gopkg/platform/detect_windows.gopkg/platform/windows_managers_windows.gopkg/platform/detect_darwin.gopkg/platform/darwin_managers_darwin.gocmd/writ/writ/snapshot/protect_darwin.gopkg/platform/detect_linux.gopkg/platform/linux_managers_linux.gopkg/platform/cross_distro_managers_linux.go
The _windows.go and _darwin.go files cannot even be compiled by quality-gate — go build on ubuntu excludes them. Their only compilation in CI is make dist, which cross-compiles via GOOS=$$os GOARCH=$$arch (Makefile:195-196) and runs on release tags only. Compiling is not testing, and it does not happen per pull request.
This is the test-side form of a blind spot this repository has already been bitten by: the lint recount had to add an explicit GOOS=linux pass because CI is ubuntu and platform-gated code was escaping the gate.
How it surfaced
PR #372 added characterization tests under cmd/lore/lore and cmd/lore/lore/onboard. All five checks went green, including scenario (windows-latest), and that was reported as the tests passing on Windows. They did not run on Windows — -run TestWritDeployScenario ./cmd/writ never reaches those packages. They ran on ubuntu only, via test-race in quality-gate.
Those tests assert exact byte sequences including newlines, which is precisely the class of assertion that can differ across platforms.
Required change
make test-race runs on every leg of the matrix, not just ubuntu.
Known risks, to be handled by the plan rather than assumed away
- Turning the suite on for Windows and macOS will likely surface pre-existing failures. Those are real defects being revealed, not regressions introduced by this change, and each needs triage rather than a skip.
- Some existing tests may carry Unix assumptions — path separators, file permission bits (
0o600behaves differently on Windows), symlink creation privileges, line endings. - CI cost rises: the full race-enabled suite on three runners instead of one.
quality-gatecurrently mixes concerns (build, tidy, frontmatter, vet, lint, test, shell lint, metrics). Only the test step needs to fan out across platforms; lint and shell lint arguably should not.
Activity
docs: plan for running every test on every platform (#374)
ci: run the unit suite on every platform (#375)
fix(test): give the devlore-test binary its .exe suffix on Windows (#378
David-Noble-at-work commentedon Aug 12, 2026
Phase 4 input, found on PR #386:
TestRegistry_SyncIntegration(internal/lorepackage/registry_test.go:196) performs a live git clone of devlore-registry from github.com inside the unit suite, and a runner DNS hiccup produced the first macOS red in six matrix runs (Could not resolve host: github.com, 30s of it). Windows held at the standing 34 — the red was pure flake.Consequence for phase 4: a network-dependent test in
make test-racemeans a requiredtest (…)leg can go red on infrastructure weather, which is exactly what erodes a required check's authority. Before the legs are made required, this test needs one of: a hermetic fixture (local bare repo instead of github.com), a guarded skip when the network is genuinely unreachable (a skip on unreachability is not a skip to dodge a failure — the subject cannot exist without the network), or relocation behind an explicit integration tag excluded from the required legs. The hermeticity-guarantees work already established the repo's preference here.ci: analyze every GOOS per pull request — vet, lint, and cross-compil…
docs: the canonical-Rel plan and the corrected Windows baseline (#391)
fix(file): make Find's glob matcher slash-native so ** patterns work …
fix(file): treat rooted patterns as absolute in Find's scope check; f…
docs: record the honest Windows baseline and the banked #393 ruling (#…
docs: charter complexity phase 8 and record the two gates disagreeing (…
test(cli): assert on what the CLI owns, not on what the OS says (#555)
test: stop four fixtures asserting Unix-shaped assumptions (#562)
doc(platform-test-matrix): phase 4 enforced, and it is not on the rul…