Repository navigation
e2e: diagnostics gaps — 35% of red runs are undiagnosable, 16% retain nothing #3368
Description
Activity
New gap: host-node Talos collection has never worked
Found while diagnosing run
29825884772(bothkubernetes-*Tests red). This is distinct from the tenant-worker item above — that one is "no talosconfig exists on the runner". Here the talosconfig does exist andcozyreport.shactively collects against it, but all three commands fail on every run, silently.hack/cozyreport.sh:345-347:talosctl --talosconfig /workspace/talosconfig -n "$node" dmesg --tail=200 > "$DIR/talos-$node-dmesg.txt" 2>&1 || true talosctl --talosconfig /workspace/talosconfig -n "$node" logs kubelet --tail=500 > "$DIR/talos-$node-kubelet.log" 2>&1 || true talosctl --talosconfig /workspace/talosconfig -n "$node" logs containerd --tail=500 > "$DIR/talos-$node-containerd.log" 2>&1 || true
Two independent defects.
2>&1 || truemeans each one writes its own error message into the artifact and returns 0, so nothing ever surfaced:-
dmesg --tailis a bool, not an int.--tail=200→invalid argument "200" for "--tail" flag: strconv.ParseBool: parsing "200": invalid syntax, then the usage dump. Fordmesg,--tailmeans "only new messages" and only makes sense with--follow; plaindmesgalready returns the whole ring buffer. (talosctl logsis the subcommand where--tailis an int32 — the flag was copied across two commands that don't share its type.) - No
-eendpoint.logs kubelet/logs containerdparse fine, then die aterror constructing client: failed to determine endpoints— the talosconfig carries no endpoints, so-nalone is not sufficient.hack/e2e-prepare-cluster.batsalready passes-e 192.168.123.10for exactly this reason.
Ordering matters: flag parsing precedes client construction, so fixing the
--tailtype alone just movesdmesgonto the same endpoint error as the other two. Both need fixing together.From this run's
cozyreport/sandbox-host/— 9 files, 9 errors, zero bytes of signal:1110 B talos-192.168.123.1{1,2,3}-dmesg.txt → invalid argument "200" for "--tail" flag + usage 57 B talos-192.168.123.1{1,2,3}-kubelet.log → error constructing client: failed to determine endpoints 57 B talos-192.168.123.1{1,2,3}-containerd.log → error constructing client: failed to determine endpointsThis has been true since the collector landed in #2553, which means no e2e failure has ever shipped host-node kernel, kubelet or containerd logs — including the node-join class where they are the first thing you'd want to read. Belongs in the "free, no budget cost" tier: add
-e, drop--tailfrom thedmesgcall.Confirmations from the same run
- Silently partial crust-gather trees are real, and they land on the test that matters.
kubernetes-latest's snapshot hashost/cluster/andhost/namespaces/but zero pod specs and zero logs; siblingkubernetes-previous— same run, same failure mode — is complete with logs. Consistent with the--durationdefect above (60s default, exit 1,finish()skipped). The run was only diagnosable because the other snapshot happened to survive. events-only onkubernetes-*cost the diagnosis directly. The failure wascapi-controller-managerunable to reach the CAPK validating webhook (dial tcp 10.96.246.186:443: connect: connection refused) → MachineDeployment never scaled (MachinesReady: NoReplicas,ScalingUp: InternalError) → zero worker VMs →WorkloadsReady=False→ timeout at 680s/566s. That error exists only in the CAPI controller log, whichkubernetes-*never captures.Ready∧WorkloadsReadywould have been the one-line answer. TheKubernetesCR readReady=True(InstallSucceeded) alongsideWorkloadsReady=False("One or more workloads are not operational") — precisely the discriminator proposed above.kubectl top nodeswould have closed out the CPU-starvation hypothesis on the spot. Host PSI for the full 110-minute run totalled 23.7s CPU / 30.5s IO / 0.03s memory stall, steal ~0.001%, no cgroup throttling — the runner was never the constraint, but proving that took an out-of-band session on the host.
-
Implementation progress
- test(e2e): capture previous container logs and stop killing the snapshot #3346 covers explicit
crust-gather --durationbudgets at all three call sites, the global Chainsaw catch timeout inversion, persisted collector verdicts, and local previous-instance container log capture. - test(e2e): capture per-node network diagnostics in cozyreport #3213 adds end-of-job per-node network diagnostics: iptables, conntrack, OVS flows, Hubble flows, and Cilium endpoint state.
- test(e2e): make crust-gather failure snapshots reliable and complete #2952 has been closed because its
hack/cozytest.shsnapshot handling overlaps test(e2e): capture previous container logs and stop killing the snapshot #3346. Its non-overlapping--disable-additional-logswork remains tracked here. - The next implementation block will fix the dataplane collector’s unreachable-LB ordering and empty-
podIPblind spots, render the Bucket catches as YAML, and repair host-node Talos dmesg/kubelet/containerd collection. - Tenant-worker Talos collection remains a separate design task because the runner currently has no tenant talosconfig.
#3213 and #3346 are still open, so their items should not be treated as merged until they land.
- test(e2e): capture previous container logs and stop killing the snapshot #3346 covers explicit
- added a commit that references this issue
on Jul 23, 2026 Host-node collection works now, and the
--tailvalues are doing something other than what they look likeThree findings from artifacts of failed runs on 07-28 and 07-29, all about
hack/cozyreport.sh:345-347.The three commands produce real output now. On four failed runs every node yields a kubelet log of exactly 500 lines of kubelet JSON and a dmesg of 1363 to 1474 lines with kernel timestamps. The "fails on every run, silently" state described earlier no longer holds, so that box can be checked.
--tail=200on dmesg does not bound anything. In talosctl v1.13.5, the version pinned inpackages/core/testing/images/e2e-sandbox/Dockerfile, the two flags have different types.talosctl logs --tailis an int32, "lines of log file to display", default -1 meaning show from the beginning.talosctl dmesg --tailis a boolean, "specify if only new messages should be sent (makes sense only when combined with --follow)". No--followis passed, so the dmesg call returns the whole ring buffer and the 200 is decoration. That matches the 1363 to 1474 lines observed.The 500-line kubelet bound is what hides the install phase. Measured on three nodes of one failed job, 500 lines of kubelet covers the last 15 to 27 minutes. The snapshot is taken at the end of the job, so on any run where install takes half an hour the install phase falls outside that window every time rather than occasionally. In one failure a zero read from those files meant nothing: two of the four lease-loss bursts in that run happened before the earliest line collected.
Since
--taildefaults to -1 and that already means "from the beginning", the fix is to drop the flag rather than raise it. There is no time-based option ontalosctl logs, so bounding by lines is the only knob available, and the useful setting is the default.etcd is the one service not collected, and it is the one that would close install-phase failures
Five install-phase failures over two days share a shape. Controller managers across several namespaces lose their leases in bursts, whichever bats step is waiting inside a burst fails, and the errors are exclusively deadline-exceeded with
connection refusedandno route to hostat zero.The apiserver static pod logs already retained, under
snapshots/*/host/namespaces/kube-system/v1/pod/kube-apiserver-srv*/, show what the apiserver was doing: on one run a single apiserver logged 48etcdserver: leader changedand 62etcdserver: request timed outover 36 minutes, and the same logs carry tens of thousands ofwatch chan error: etcdserver: no leaderconcentrated into bursts. Measured across four failed runs those bursts last 0.9 to 17.9 seconds, and on the two runs where the lease-loss bursts were also measured they fall between 0.9 and 5.6 seconds. So the apiserver was waiting on its backend rather than failing by itself.The backend's own log is the part never collected. The sandbox runs Talos, so control-plane etcd is a host service rather than a pod, and the
tenant-rootetcd entries in the snapshot belong to the tenant etcd application instead. etcd reports its own slow paths with measured durations:failed to send out heartbeat on time,apply request took too long,slow fdatasync,took too long to execute. Those durations are what separates a starved disk from a starved CPU. Without them the class stays diagnosable only through consumers, which all report the same deadline and none report a cause.One more line beside the three already there, same command and same talosconfig, with the flag left at its default so the whole run is covered:
talosctl --talosconfig /workspace/talosconfig -n "$node" logs etcd > "$DIR/talos-$node-etcd.log" 2>&1 || true
- added a commit that references this issue
on Aug 8, 2026 Another gap, found while working on #4374: the report never carries the CDI controller log, and that is the one log that could say why an importer pod was left behind.
hack/cozyreport.shat 8d37e0b collects operator, Flux, cert-manager, Kamaji and COSI logs, and nothing fromcozy-kubevirt-cdi. In #4374 CDI marked the PVC import as succeeded and then should have deleted the importer pod in the same reconcile. It did not, and the only record of why is thecdi-deploymentlog, which is gone once the sandbox is. The pod itself is also out of reach, because the per-pod walk skipsCompletedpods by design, so there is nopod.yamlto show whether it had adeletionTimestamp.A capped
kubectl logs deploy/cdi-deployment -n cozy-kubevirt-cdinext to the other controller logs would be enough for this case. The finally-step side of the same run is in #4408.State refresh, 2026-10-01
Most of the free tier landed. 8 rows in the body are ticked with their PRs (#3421, #3346, #3444, #3567, #3548), and #4414 added the CDI controller log lexfrei asked for on 2026-09-23. The headline 35% and 16% are from 2026-07-07 to 07-20 and I can't re-measure them cheaply, the body now says so.
What is still open is the run-level deadline. The job cap is 215 min and in the last two weeks of PR runs 2 of 880 E2E jobs ended as 220 min timeouts with no report, one more failure ended at 214.5 min. As a rough proxy, all 84 failed jobs in that window uploaded
cozyreport.tgz, and 5 of 880 jobs lost their evidence, against 20 of 124 bad jobs in July. This counts job conclusions and not log contents, so it is a different measurement from the headline.Nightly is not skipped any more, it runs and is red on most nights, so that bullet is updated. Narrower defects in collectors that exist already are filed separately, the body lists them. I keep this issue for what the report should contain.
Status as of 2026-10-01
Checked against
mainatbc400dff1. 8 rows below are ticked, each with the PR that did it. Three items raised in the comments are done too: host-node Talos collection (#3421), previous-instance and init-container logs (#3346) and the CDI controller log (#4414). The numbers in the opening section are from 2026-07-07 to 07-20 and predate all of those PRs, so read them as the starting point and not as today's state.Still open, and the most important one is the first: there is no run-level deadline around the Chainsaw step, so the job cap (215 min now, not 180) still destroys log and report when a job hangs. In PR runs from 2026-09-17 to 09-30, 2 of 880 E2E jobs ended as 220 min timeouts with no report uploaded, and one failure finished at 214.5 min. The other open rows are the event TTL, the LB probe sweep gate, and the signals from the "never collect" list (
WorkloadsReady,kubectl top,lastState, VAP status, shard ownership, VictoriaLogs). As a rough proxy for "retains nothing": all 84 failed E2E jobs in that window uploadedcozyreport.tgz, and 5 of 880 jobs lost their evidence, against 20 of 124 bad jobs in July. That is job-level, not the log-content "undiagnosable" metric of the Summary, so it is not the same measurement.The nightly is not skipped any more, it reaches the suite and is red on most nights, so the last adjacent bullet below is out of date. Narrower defects in the collectors that already exist are filed as separate issues (#4408, #3680, #3686, #3687, #3666, #3675, #3679, #3690, #3694, #3695, #3698, #3707, #3754, #4243), and I keep this issue for what the report should contain.
Summary
I sampled 400
pull-requests.yamlruns (2026-07-07 → 07-20, 12.9 days) and pulled every retrievable log for the 124 failed or cancelledE2E Testsjobs, then cross-checked what our collection points actually capture against what each failure class needs.Two numbers frame the rest of this issue:
cozyreport.Roughly half of red e2e jobs are infrastructure flake or lost evidence rather than product signal. So the problem is not mainly "collect more" - it is that we lose what we already collect, spend budget on collectors that answer nothing, and in two cases run diagnostics that are structurally blind to the failure they were written for.
Two structural losses
A 180m job timeout destroys the log and the report. When the job hits
timeout-minutes: 180it endsconclusion=cancelled. GitHub's post-cancellation grace measured at ~4m48s in both examined cases, and the earlierCollect reportstep consumes all of it, soUpload cozyreport.tgznever starts.Complete jobnever runs either, sogh api .../jobs/<id>/logsreturnsBlobNotFound.Examples: runs
29557003429,29568562337,28924357714,29722079370. The five slowest and most interesting jobs in the window are the five with zero evidence - anything whose failure mode is "hangs" is definitionally unobservable today.All chainsaw catch output is job-log-only.
chainsaw-report.xmlcarriesFailure.Messageand nothing else - noSystemOut. So everydescribe, everypodLogs, and every assertion diff dies with the log in the case above. There is no partial-upload path.Diagnostics that are blind to their own target
e2e-capture-dataplane.shexits before the LB section whenever all pods are Ready.capture_lb_datapathis called at line 737, but lines 250-253exit 0when no NotReady pod exists. The script's own header (lines 56-64, 472-474) states the LB section exists precisely for the case where there is no NotReady pod and is "independent of the NotReady-pod path above". It is not. An unreachable LB with all pods Ready - thekubernetes-*node-join / MetalLB signature - never reaches the capture. Fixed in fix(e2e): improve failure diagnostics #3421, the onlyexit 0is at the end of the script now.podIP. The ciliumendpointManagerleak manifests as pods that never receive an IP, so the one script written to diagnose it filters it out by construction. Fixed in fix(e2e): improve failure diagnostics #3421.HostLogscreates a privileged debug pod on every node even when no--systemd-unitis set, iterates an empty list, collects zero logs, and deletes the pod withgrace_period(0). We pass no units, so every call churns 3 privileged pods for nothing. Pod-delete-under-churn is the documented trigger for the cilium endpoint leak, so the collector can induce the flake it is collecting for.--disable-additional-logsis free. Done in fix(e2e): keep failure snapshots complete and gate tenant patch on aggregated-API auth #3444 at all three call sites.Budget defects
crust-gatherbudget in the repo is really 60s, not the 300s it reads as.--durationis not passed at any of the three call sites, so the 60s default applies. On elapse it exits 1 (not 124) and skipsfinish(), leaving a silently partial tree. Done in test(e2e): capture previous container logs and stop killing the snapshot #3346,--duration 180sat all three call sites.error.catchdeclares 960s of inner budget inside a 300s outer. The catchscript:carries no per-optimeout:, so it inheritstimeouts.exec: 5mwhile wrappingtimeout -k 30 300(crust-gather) plustimeout -k 30 600(dataplane). 17 jobs in the window showsignal: killedmid-collection having produced nothing. test(e2e): capture previous container logs and stop killing the snapshot #3346 fixed this specific inversion, e2e: the Chainsaw catch budget counts only the global legs, not the per-suite collectors #3754 is the leftover (the budget still counts only the two global legs).event-ttloverride in-tree, so end-of-jobkubectl get events -Ais largely vacuous: install-phase events have expired by minute 95. Warning events need capturing at failure time.Coverage gaps
catchblock at all - all ingateway/, six of them VAP/policy negative tests whose most useful single line would bedescribe validatingadmissionpolicy+...binding.events-only, includingkubernetes-latestandkubernetes-previous- the two longest (40m budget) and most flake-prone Tests in the suite.bucket(the COSI provisioner is the prime suspect and is never logged) andvmdisk/vminstance(the CDI importer surface, which is our single largest failure class).events: {}only reachestenant-test. Chainsaw rewrites it togetonv1/Eventscoped to$NAMESPACE, so thecozy-*namespaces where the operators run are never covered.bucketandharborget:catches emit NAME + AGE only - noformat:is set and COSIBucketCRDs ship no printer columns. These catches exist to show readiness and show nothing.cozyreport.sh:220-228already works around this withcustom-columns. Done in fix(e2e): improve failure diagnostics #3421, all fivegetcatch ops haveformat: yaml.cozyreport.shskips logs entirely when pod phase isPending, which also skips the init-container logs of a pod stuck atInit:0/2- readable, and usually the answer. Done in fix(e2e): collect diagnostics for unready pods and record every truncated capture #3567, non-Running pods are collected.=== (b) in-guest Talos/kubelet capture skipped: no tenant talosconfig on the runner ===- CI states in its own output that it cannot collect the evidence for the case it just hit. This is 14 of the 43 undiagnosable failures. Done in fix(e2e): capture Talos logs for tenant worker join failures #3548.Signals we never collect that would be cheap
Ready∧WorkloadsReadycondition pair on everyapps.cozystack.ioCR. Perpkg/registry/apps/application/rest.go:1436-1444,Readyis copied verbatim from the HelmRelease and deliberately never reflects workload health; real health isWorkloadsReady, synthesized fromWorkloadMonitor. A Postgres can reportReady=Truewith zero CNPG instances serving. One jsonpath, and it is the cheapest discriminator between "Helm applied" and "the product works". (Note: postgres/mongodb/redis/opensearch/kafka embed the monitor in the engine CR template, andvpchas none, so coverage is not uniform.)qdrant,nats,harbor,openbaoandbucketeach emit a second<name>-systemHelmRelease against a vendored chart; any probe checking only the app-named HR sees half the chain.kubectl top nodes/top pods -A- metrics-server is already deployed and this is collected nowhere. It is also the only cheap test of the CPU-starvation hypothesis behind the node-join and bare-kubectl waitclasses.lastState.terminated(exit code / reason / OOMKilled) projected across all pods - an OOMKilled process writes no farewell, so the proof is in pod status, not logs.Endpoints/linstor-controller .subsets[].addresses- reloader rolls the single-replica controller when cert-manager rewrites the API-TLS Secret aftercondition=availablepassed, so the Deployment condition is not the signal.ValidatingAdmissionPolicy/...Bindingand.status.typeChecking- never collected anywhere, despite six Tests existing solely to exercise them, and despite this being the object behind the KCM type-checker CrashLoop in fix(api): publish open spec as x-kubernetes-preserve-unknown-fields, not additionalProperties:true #2867.tenant-*namespace, does a controller exist whose watch selector matches itssharding.fluxcd.io/key? flux-aio controllers run--watch-label-selector=!sharding.fluxcd.io/key, and tenant HRs keepsharding.fluxcd.io/key=tenantsuntilhelm-controller-shard0relabels them, so if the shard operator never comes up no controller owns them and the HR.statusis simply empty. The proof is a selector mismatch; there is no error to grep for.The observability stack we already run and discard
isp-fulldeploys VictoriaMetrics (shortterm 3d, scraping cadvisor/kubelet/node-exporter/apiserver/kube-state-metrics), VictoriaLogs (1d, fed by cluster-wide fluent-bit), metrics-server and goldpinger. None of it is exported on failure.src/scanners/logs.rs:107iteratespod.spec.containersonly;logs.rs:127returns out of the whole function on a 400, and the apiserver returns 400 for a not-yet-started container).Best-effort caveat: both are HelmReleases and may themselves be down on a badly degraded cluster, so this supplements rather than replaces object collection.
Suggested ordering
Free, no budget cost, no design debate:
cancelled- maximum diagnostics collected, zero delivered - into an ordinary failure with full artifacts, and it protects every other item in this issue.--disable-additional-logson all three crust-gather call sites. Done, fix(e2e): keep failure snapshots complete and gate tenant patch on aggregated-API auth #3444.format: yamlon thebucketandharborget:catches. Done, fix(e2e): improve failure diagnostics #3421.podIPexclusion. Done, fix(e2e): improve failure diagnostics #3421.Then the cheap always-tier probes above, and only then bounded/on-signature collection with a run-level budget rather than per-test caps.
Upstream gaps
None of these have existing upstream issues, and there is no upgrade path - crust-gather v0.16.0 is 19 commits from v0.15.2 with 18 of them dependabot and
src/scanners/untouched, and chainsaw v0.2.15 is the current release withPodLogsunchanged onmain.--tail/--sincefor logs (~15 lines; logs are currently fetched unbounded, twice per container, and are the dominant cost term).join_allwith no QPS limit and infinite retries, so it hammers the degraded apiserver it is diagnosing).logs.rs:127return→continue- one word, and it is why multi-container pods are routinely truncated.previousonpodLogs(~5 lines). Low urgency since test(e2e): capture previous container logs and stop killing the snapshot #3346 works around it locally. Worth watching chainsaw [dx] automatically detect version for migrations in installer.sh #837, which proposes renamingpodLogs→logsand would touch every catch block we have.Adjacent findings - separate issues, not diagnostics
Surfacing these here so they are not lost; happy to split them out.
serviceexposure-delete/serviceexposure-externalipssuites failed identically on 10 unrelated branches from 2026-07-16T16:06 to 07-17T21:42 withnetwork.cozystack.io/v1alpha1: no matches, i.e. testing a CRD that does not exist. That is 13.7% of all e2e failures in the window, charged to individual PR authors. The suites were removed inbb1127c76on 2026-07-17. Nothing in CI notices that a suite is failing on every branch, that part is still true.nightly.yamlfailed at its first step for 11 consecutive nights in July (unsupported image-specific operation on artifact with type "application/vnd.cncf.flux.config.v1+json"), soE2E (full suite)was silentlyskipped. Fixed in fix(ci): resolve nightly packages digest without image-specific skopeo ops #3332. It runs now and is red on most nights, the last scheduled green was 2026-09-03 and the 2026-09-30 run failedkubernetes-latest,kubernetes-previousandopensearch. That is tracked as A18 in [Tracking] Remaining CI / build / release / E2E automation work #3278.missing go.sum entry, fanned across 8 Go packages on every rebase - and 113 runs never reached e2e at all as a result.