e2e: the tenant backend readiness budget starts before the tenant can schedule, and absorbs a Docker Hub pull #3577

hack/e2e-chainsaw/_lib/run-kubernetes.sh:628 gives the tenant workload Deployment 300s to become Available:

kubectl wait deployment --kubeconfig "tenantkubeconfig-${test_name}" "${test_name}-backend" -n tenant-test --for=condition=Available --timeout=300s

Two things consume that budget before the container starts. In run 31020254620 on e80b8e370 they consumed all of it in both kubernetes-previous and kubernetes-latest, and both failed with error: timed out waiting for the condition on deployments/test-*-backend.

What the budget went on

Timings are from the tenant crust-gather the run collected, under snapshots/<suite>/tenantkubeconfig-<test>/namespaces/tenant-test/events.k8s.io-v1/.

kubernetes-previous, wait window 17:52:21 to 17:57:21:

The container was created one second before the deadline, so the readiness probe (initialDelaySeconds: 2, periodSeconds: 2) never ran once.

kubernetes-latest, window 18:22:26 to 18:27:26: FailedScheduling with both nodes tainted at 18:22:26, the same node.cilium.io/agent-not-ready named at 18:23:14, Scheduled at 18:24:23, Pulling at 18:25:42, and the budget gone with the pull still in flight.

Ready is not schedulable

What the suite establishes before this point is two tenant nodes Ready, through the 18m gate above it. Both instances show that a Ready node is not necessarily one this pod can land on. In the first, one node carried node.cilium.io/agent-not-ready and the other was SchedulingDisabled. In the second, both carried the taint. Scheduling cost 2m18s and 1m57s.

The image comes from Docker Hub every time

run-kubernetes.sh:599 uses nginx:alpine, unpinned, pulled by the tenant kubelet. Nothing in this environment caches container images: hack/e2e-talos-image-cache.yaml serves the Talos worker OS disk image over HTTP and is not a registry mirror, and no registry mirror is configured for tenant workers. So an external fetch of variable duration sits inside a fixed budget. It measured 1m42.435s in one instance and outlasted the remaining budget in the other.

Not the node-join failure

This is distinct from #3513 and #3229. There the tenant registers zero nodes and the 18m gate fails. Here both workers registered and reached Ready, the gate passed, and the failure is one step later. The job log separates them: node-join failed: fewer than 2 tenant nodes Ready within 18m against timed out waiting for the condition on deployments/test-*-backend.

Directions

Start the clock from a schedulable node rather than a Ready one. Or split the wait so scheduling and image pull carry their own budgets, which would also make the log say which of the two ran out. Or take the pull off the critical path, by pinning the image by digest and preloading it onto the workers instead of fetching it per run. They are not exclusive, and only the last removes the external dependency.