Repository navigation
e2e: harbor suite flakes when rfs-harbor-test-redis Sentinel HA never reaches readiness #3445
Description
Activity
- added a commit that references this issue
on Jul 28, 2026 This flaked again twice after the fix landed, on runs whose base already carried it: job 90333030340 (2026-07-28 16:34:43 UTC) and job 90385337598 (2026-07-28 21:26:45 UTC). Same step, same body,
Ready: "Unknown"on the harbor-test HelmRelease.I pulled the duration of the
wait-helmrelease-readyassert out of 20 runs. Sixteen healthy ones span 1m42s to 3m12s. The failures are 0m25s and 0m27s on one side and 10m00s and 9m59s on the other. Nothing at all lands between 3m12s and 10m.That says the budget was never the binding constraint. The healthy maximum of 3m12s is 53% of the old 6m and 32% of the 10m I raised it to, so the value I changed had not limited a healthy run even once. The two short failures are a separate thing: an assert whose poll was cut short by a transient API error rather than a slow release (upstream behaviour, kyverno/chainsaw#2758).
The catch output adds one more fact: six minutes after the deadline expired, the dataplane capture reports no scheduled NotReady pods at all. The release does converge, somewhere between 10m and 16m, it just does not converge inside the window.
What is missing to take this further is a snapshot of the RedisFailover and its sentinels taken at the moment the deadline expires. The catch collects a host snapshot, which does not detail them, so a rare slow sentinel bootstrap and a release that stalls and then recovers on its own look identical in what we keep today.
A fresh occurrence contradicts the diagnosis in this issue, so posting what the artifacts of run 31173212908 (job 92852286215, 2026-08-07) actually show before anyone goes digging in the spotahome operator.
The sentinels are fine there. All three
rfs-harbor-test-redispods are 1/1 Running, each logs+monitor master mymaster ... quorum 2at 12:10:07 and sees the two others by 12:10:09. Quorum formed 85 seconds after the pods were created. "All three Sentinel replicas fail their readiness probe" does not hold on this run.What actually eats the budget is an uninstall-remediation cycle. First install: harbor-core waits for redis, redis waits for its PVC (linstor provisioning) and the master election, and
harbor-test-nginxhits the Deployment defaultprogressDeadlineSecondsof 600s. Flux reads that kstatusFailedas fail-early (failed early due to stalled resources), and withinstall.remediation.retries: -1(packages/apps/harbor/templates/harbor.yaml:92-94) it remediates by uninstall, which deletes everything including the redis PVC. The second install starts from scratch at 12:08:47, converges its sentinels by 12:10:09, and runs out of the chainsaw assert budget at 12:11:40 with harbor-core still in its startup probe.The arithmetic makes this a deterministic fork, not a busy-runner flake: the chainsaw wait budget is 10m and
progressDeadlineSecondsis 10m, so a single remediation cycle consumes the whole test budget by construction. If the first install beats the deadline the test is green; if it does not, the test is lost no matter how fast the second attempt converges.This is the install-remediation-by-uninstall family, the same shape the tenant csi and cni HelmReleases had before their retry fixes. The fix probably belongs there (retry in place instead of uninstall, or ordering redis before harbor-core, or a deadline that actually fits), not in the redis operator.
One verification limit: the July run this issue was filed from cannot be re-checked, its job log is expired and the report snapshot is taken after chainsaw wipes the namespace, so the harbor pods are already gone in it. The catch-block log is currently the only record of this mechanism, and it dies with log retention.
- added a commit that references this issue
on Aug 8, 2026 Not sentinels, and not the uninstall cycle from my last comment. Both recent harbor reds are the pod-IP collision from #3615 and #3736.
Runs 31338950312 (job 93311099197) and 31338954173 (job 93313889744), both 2026-08-09. In each one a harbor pod got an address the node already holds on
cilium_host, so kubelet's probe lands on the node's own stack and answersconnection refusedfor as long as the pod lives.harbor-test-nginx-c69459d47-rnkjson srv3, first run:=== host netns: ip route get 10.244.0.197 === local 10.244.0.197 dev lo table local src 10.244.0.197 === host netns: ip -o addr show === 10: cilium_host inet 10.244.0.197/32 scope global cilium_hostharbor-test-db-1on srv2, second run:=== host netns: ip route get 10.244.1.94 === local 10.244.1.94 dev lo table local src 10.244.1.94 === host netns: ip -o addr show === 10: cilium_host inet 10.244.1.94/32 scope global cilium_hostThe second run shows it is not a dead workload. kubelet on srv2 gets
connection refusedonhttps://10.244.1.94:8000/startupzwhile cilium conntrack on that same node has other pods talking to the address:TCP OUT 10.244.0.58:58282 -> 10.244.1.94:8000 RxFlagsSeen=0x1a TxFlagsSeen=0x1a [ SeenNonSyn ] TCP IN 10.244.0.83:51088 -> 10.244.1.94:9187 RxFlagsSeen=0x1a TxFlagsSeen=0x1a [ SeenNonSyn ]Pod to pod works, host to pod does not.
Everything else follows from that one pod. In the first run every harbor pod is Ready at the deadline except nginx, and the release dies on
failed early due to stalled resources: [Deployment/tenant-test/harbor-test-nginx status: 'Failed']. In the second the unreachable pod is postgres, with core, jobservice and exporter crash-looping behind it at 5, 6 and 5 restarts.Sentinels are healthy in both. All three
rfs-harbor-test-redis-*pods are Ready with zero restarts at the deadline in each run, and in the first one core logssentinel: new master="mymaster" addr="10.244.0.204:6379"83 seconds after the RedisFailover was created. #3621 works too:harbor-test-systemcarries a single install in its history in the first run,First Deployedequal toLast Deployed, and nothing is torn down in either. Bucket isbucketReady: truein the first run, so not the hairpin either.Same night, same batch of eight concurrent runs, three of the four suite failures were this bug. Harbor twice in the quiet form above, mongodb once in the loud one with
IP ipv4:10.244.1.12 is already in use. Harbor is not special, it just has the most pods and draws the unlucky address most often.The attribution only became readable that day. #3617 merged at 12:39 UTC and these runs started at 22:15, so they are among the first to record which interface holds the address. #3615 asked that question and the answer is
cilium_host.I would close this one in favour of #3736.
- added a commit that references this issue
on Aug 11, 2026
Summary
The
harborChainsaw suite intermittently fails atwait-helmrelease-readybecause harbor's Redis dependency, therfs-harbor-test-redisSentinel HA set, never reaches readiness within the 10m budget. This is separate from #3143 (BucketClaim readiness): in the same run thebucketsuite's claim converged fine, so object storage is not the cause here.Evidence
E2E run 30017309131 (attempt 1),
E2E Testsjob, suitechainsaw/harbor(failed after 642s; 42 passed, 1 failed).Harbor-core cannot reach Redis:
All three Sentinel replicas fail their readiness probe for roughly 8m, so the set never forms:
The suite then times out on the harbor HelmRelease:
Object storage is healthy in the very same run, which rules out #3143:
Why this is not #3143
#3143 was the BucketClaim staying
bucketReady: falsebecause the objectstorage-controller image predated #3034. That is resolved onmain: the pin is now thev1.6.0build that carries the requeue patch, andhack/overlay-main-images.shkeeps e2e on current-main digests. The failure here is harbor's Redis Sentinel set not converging, a different subsystem (spotahome redis-operator, not objectstorage).Notes
This fits the loaded-runner readiness flake class described in #3368 (about half of red e2e is infrastructure). A rerun sometimes passes. Worth checking whether the Sentinel readiness budget under harbor is too tight on a busy runner, or whether the set has a genuine convergence race worth fixing rather than waiting out.