Skip to content

e2e: node-join diagnostics still describe the QEMU sandbox that no CI lane boots #4825

Description

The node-join diagnostics in hack/e2e-chainsaw/_lib/run-kubernetes.sh were written for the QEMU sandbox, where the three management nodes are QEMU guests started by hack/e2e-prepare-cluster.bats. Since #4020 the PR lanes run the nodes as Talos containers, and once nightly and the release-tag lane move too, no CI lane boots that sandbox. The diagnostics still run on every failed tenant node-join and still explain their output in QEMU terms.

Two of the legends now state something false in the report. The node CPU-time legend says a zero steal is meaningful because "the sandbox VMs are started with accel=kvm and -cpu host by hack/e2e-prepare-cluster.bats". On the container lane the node's /proc/stat is the runner kernel's, so that reasoning does not apply. The NO-KVM-DIR line says the same thing about accel=kvm and reads a missing kvm debugfs directory as evidence about how the sandbox VMs were run, while the only KVM guests left on this lane are the tenant workers.

The sandbox QEMU thread capture (cozy_capture_sandbox_qemu_thread_cpu) looks for srv<N>/qemu.pid files that the container lane never writes, so it can only report that it found nothing. It still runs on every tenant Kubernetes suite, once before the node-join wait and once after it, green runs included.

hack/run-kubernetes-kvm-exits_test.bats and hack/run-kubernetes-runner-cpu_test.bats pin parts of this wording, so changing it means changing the tests too. I think each capture should either be re-targeted at what the container lane actually has (tenant worker VMs under KubeVirt on the runner kernel) or removed together with the QEMU sandbox when it is retired (#4642).

Activity

  1. added
    priority/backlogGeneral backlog priority. Lower than priority/important-longterm
    kind/cleanupCategorizes issue or PR as related to cleanup of code, process, or technical debt
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    on Oct 9, 2026
  2. lexfrei commented on Oct 9, 2026

    @lexfrei
    ContributorAuthor

    Same thing in the header of hack/sandbox-runner-headroom.bats: it describes every lane as booting the QEMU guests from hack/e2e-prepare-cluster.bats. The note inside the test already explains that container lanes read the same figures, so only the header text is out of date.

  3. lexfrei commented on Oct 9, 2026

    @lexfrei
    ContributorAuthor

    Two more places still name the QEMU entry point as the way every lane gets its sandbox: hack/e2e-runner-identity.sh says every lane goes through make prepare-env, and docs/agents/overview.md presents make prepare-env as the e2e environment. Both still work by accident, since prepare-env-container also runs apply, but no CI lane calls prepare-env once nightly and the release-tag lane move.

  4. lexfrei commented on Oct 9, 2026

    @lexfrei
    ContributorAuthor

    One more for whoever removes the QEMU target: packages/core/testing/Makefile justifies override PLATFORM := linux/amd64 with the QEMU lane booting the amd64 disk, and docs/agents/overview.md says the same. Still true while prepare-cluster exists, but that text has to be rewritten in the same change that drops it.

  5. lexfrei commented on Oct 9, 2026

    @lexfrei
    ContributorAuthor

    The same "every e2e lane reaches the sandbox through make prepare-env" sentence is also in docs/agents/e2e-testing.md and in packages/core/testing/Makefile next to the runner-identity call, and the Makefile comment there says apply boots three QEMU guests, which it does not. The mechanism still works because prepare-env-container runs apply too, so only the wording is off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/cleanupCategorizes issue or PR as related to cleanup of code, process, or technical debtpriority/backlogGeneral backlog priority. Lower than priority/important-longtermtriage/needs-triageIndicates an issue needs triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions