Skip to content

CI runs the unit suite on ubuntu only; macOS and Windows run a single test #373

Open

Description

@David-Noble-at-work

All tests must run on every supported platform. Today the unit suite runs on ubuntu-latest only, and macOS and Windows run exactly one test.

Current state

Job Runner(s) Test command Scope
quality-gate ubuntu-latest make test-race all 121 packages
scenario ubuntu, macos, windows make test-scenario 1 test, 1 package
knowledge-extract ubuntu-latest make star —

make test-scenario is go test -run TestWritDeployScenario -v -count=1 -timeout 600s ./cmd/writ (Makefile). It is the writ deploy end-to-end scenario, deliberately scoped, and it is the only thing macOS and Windows execute.

So a green scenario (windows-latest) says one writ deploy scenario passed on Windows. It says nothing about the other 120 packages.

Consequence: ten platform-gated files are unit-tested nowhere

Build constraints exclude these from the ubuntu run, and the platforms where they compile only run the scenario test:

  • pkg/op/default_funcs_windows.go
  • pkg/op/provider/file/helpers_windows.go
  • pkg/platform/detect_windows.go
  • pkg/platform/windows_managers_windows.go
  • pkg/platform/detect_darwin.go
  • pkg/platform/darwin_managers_darwin.go
  • cmd/writ/writ/snapshot/protect_darwin.go
  • pkg/platform/detect_linux.go
  • pkg/platform/linux_managers_linux.go
  • pkg/platform/cross_distro_managers_linux.go

The _windows.go and _darwin.go files cannot even be compiled by quality-gate — go build on ubuntu excludes them. Their only compilation in CI is make dist, which cross-compiles via GOOS=$$os GOARCH=$$arch (Makefile:195-196) and runs on release tags only. Compiling is not testing, and it does not happen per pull request.

This is the test-side form of a blind spot this repository has already been bitten by: the lint recount had to add an explicit GOOS=linux pass because CI is ubuntu and platform-gated code was escaping the gate.

How it surfaced

PR #372 added characterization tests under cmd/lore/lore and cmd/lore/lore/onboard. All five checks went green, including scenario (windows-latest), and that was reported as the tests passing on Windows. They did not run on Windows — -run TestWritDeployScenario ./cmd/writ never reaches those packages. They ran on ubuntu only, via test-race in quality-gate.

Those tests assert exact byte sequences including newlines, which is precisely the class of assertion that can differ across platforms.

Required change

make test-race runs on every leg of the matrix, not just ubuntu.

Known risks, to be handled by the plan rather than assumed away

  1. Turning the suite on for Windows and macOS will likely surface pre-existing failures. Those are real defects being revealed, not regressions introduced by this change, and each needs triage rather than a skip.
  2. Some existing tests may carry Unix assumptions — path separators, file permission bits (0o600 behaves differently on Windows), symlink creation privileges, line endings.
  3. CI cost rises: the full race-enabled suite on three runners instead of one.
  4. quality-gate currently mixes concerns (build, tidy, frontmatter, vet, lint, test, shell lint, metrics). Only the test step needs to fan out across platforms; lint and shell lint arguably should not.

Activity

  1. David-Noble-at-work commented on Aug 12, 2026

    @David-Noble-at-work
    ContributorAuthor

    Phase 4 input, found on PR #386: TestRegistry_SyncIntegration (internal/lorepackage/registry_test.go:196) performs a live git clone of devlore-registry from github.com inside the unit suite, and a runner DNS hiccup produced the first macOS red in six matrix runs (Could not resolve host: github.com, 30s of it). Windows held at the standing 34 — the red was pure flake.

    Consequence for phase 4: a network-dependent test in make test-race means a required test (…) leg can go red on infrastructure weather, which is exactly what erodes a required check's authority. Before the legs are made required, this test needs one of: a hermetic fixture (local bare repo instead of github.com), a guarded skip when the network is genuinely unreachable (a skip on unreachability is not a skip to dodge a failure — the subject cannot exist without the network), or relocation behind an explicit integration tag excluded from the required legs. The hermeticity-guarantees work already established the repo's preference here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Epic:WindowsCampaignThe Windows campaign epicfeatureDeliverable unit within an epic

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions