Repository navigation
e2e: a chainsaw assert fails its step on a transient etcd leader change #4720
Description
A chainsaw assert fails the step when an etcd leader change lands during one of its GETs. It retries only on NotFound. Any other API error ends the poll right away and prints a diff from the last snapshot it held, so the step is red long before its own timeout.
Latest hit: run 36981828596 on #4654, suite vmdisk, step wait-vmdisk-ready. The DataVolume assert has timeout: 5m and failed after 48s, at 10:59:22.900 UTC. Between 10:59:22.894 and 10:59:22.900 all three kube-apiservers log rpctypes.EtcdError, and kube-scheduler, kube-controller-manager and kube-ovn log etcdserver: leader changed. The DataVolume itself was mid-import (PVC Prime created successfully, importer pod scheduling), not stuck.
To recognise it: ASSERT ERROR far short of the assert's timeout, plus status: {} in the diff when the expected key is an expression. Then look for leader changed in the kube-apiserver logs of the cozyreport snapshot in the same second.
Any suite using a declarative assert can get hit, and which one does is down to timing. The imperative side already guards against this with kubectl_wait_retry in hack/e2e-chainsaw/_lib/run-kubernetes.sh. The declarative asserts have no such guard. The fix belongs upstream, in kyverno/chainsaw#2758. Here, bump chainsaw once a release carries it, and until then rerun the failed job when the signature matches.
Activity
lexfrei commentedon Oct 9, 2026
Four more hits today, on four different PRs, all with the same signature. Each assert failed far short of its timeout. In the cozyreport snapshot of each run, host controllers log
etcdserver: leader changeda few seconds before the snapshot was taken:leader changed(UTC)computeplanecomputeplaneopensearchcomputeplaneI don't have the exact
ASSERT ERRORtimestamps, because my copies of the job logs stop early. So the match is on the snapshot time, not on the failing GET.All three
computeplanefailures hit the same assert, the one waiting forcomputeplane-cluster-ingress-nginxto become Ready. My guess is that this assert polls the longest of all (up to 35m while the tenant worker joins), so it has the most chances to catch a leader change. I haven't checked that. I also don't know what triggers the leader changes themselves.lexfrei commentedon Oct 11, 2026
One more hit, on #4802:
leader changed(UTC)ASSERT ERROR(UTC)computeplanefix(e2e): keep chainsaw asserts polling through transient API errors
fix(e2e): keep chainsaw asserts polling through transient API errors (c…