Repository navigation
e2e: kubernetes-previous teardown did not drain because a Completed CDI importer pod kept the worker disk PVC Terminating #4374
Description
chainsaw/kubernetes-previous went red in run 35750017261 (attempt 1, 2026-09-22) on its finally step, not on an assertion: every check of the suite passed at 17:47:46 UTC, then cozy_cleanup test-previous-version reported that the tenant did not drain within 300s and that the LINSTOR pools did not return to their pre-suite level. 55 of 56 tests passed in that run.
What was left behind, from the cozyreport artifact of that attempt. The worker disk PVC kubernetes-test-previous-version-md0-vbsvk-tjqz4-disk-system (20Gi, Block, replicated) sat in Terminating for 51 minutes at capture time with the kubernetes.io/pvc-protection finalizer still on it. kubectl describe names one pod using it: importer-kubernetes-test-previous-version-md0-vbsvk-tjqz4-disk-system, and pods.txt shows that pod in Completed (0/1), 57 minutes old, on srv1. Its DataVolume had reported ImportSucceeded at about 17:44 UTC, so the importer pod outlived a successful import by more than 50 minutes, and CDI normally deletes it right after (nothing in packages/apps/kubernetes or packages/system/cdi sets retainAfterCompletion). The CDI scratch PVC …-disk-system-scratch was still Bound. LINSTOR free space on srv1 was 75.2G against a 119.1G baseline; srv2 and srv3 had returned to baseline.
Two things I could not settle from the artifact. Why the importer pod survived its import, and why the pvc-protection finalizer held for a pod in a terminal phase, since the controller is supposed to ignore terminated pods. A second importer pod in the run's history, or a kubelet on srv1 that never unmapped the block device, would explain both, and the artifact captures neither.
Not tied to the change under test: the PR that ran (#3539) touches the OIDC block of the kubernetes chart, and kubernetes-previous creates its tenant without OIDC, where that chart renders byte-identically to main. The drain guard did its job here, turning what would have been storage carry-over into the next suite into a legible red. Filing this to track the class; a second instance with the same importer signature would make it a defect rather than a flake.
Activity
lexfrei commentedon Sep 22, 2026
The rerun passed. Same run, same head, attempt 2 finished green with all 56 tests, so the teardown drain completed inside its 300s the second time and the failure is not deterministic on that commit.
That leaves the importer pod as the thing to explain rather than the timing. A CDI importer that exits Completed and then holds the worker disk PVC through
pvc-protectionfor 51 minutes is not a slow drain, and a budget raise would only move the symptom.lexfrei commentedon Sep 23, 2026
The importer pod was left behind by CDI, and the pvc-protection finalizer is the expected result of that. Neither needs a kubelet or LINSTOR explanation.
CDI deletes the importer pod exactly once. In
updatePvcFromPodit writes theSucceededphase to the PVC, emitsImportSucceeded, and only then deletes the pod (v1.66.1 import-controller.go#L481-L505). If that delete fails, the PVC is requeued, but every later reconcile stops early because the PVC is already complete (#L189). The branch that deletes the pod when the PVC is being terminated is behind the same check. Main has the same code.After that the pod and the PVC wait for each other. The pod has a controller owner reference to the PVC with
blockOwnerDeletion, so the garbage collector removes it only after the PVC is gone. The PVC protection controller in Kubernetes v1.33.12 counts any scheduled pod that names the claim, whatever its phase (pvc_protection_controller.go#L374-L388). The exception for shut-down pods applies only to generic ephemeral volumes. So "the controller is supposed to ignore terminated pods" in the issue text is not true for a regular PVC, and a Completed importer keeps the finalizer on forever.Two corrections to the description. The StorageClass of both PVCs is
local, notreplicated. And the importer pod was created twice under the same name: the first one was interrupted, the second one finished the import and is the one left behind.I still don't know why the one delete failed. The API server was slow around 17:44 UTC (an
etcdserver: request timed outevent), which fits a failed delete call, but that is timing only. Thecdi-deploymentlog was not in the report. #4414 adds it, together with the YAML of CDI pods in every phase, so the next occurrence should show the error.Frequency: I read all 171 E2E (in-tree) job logs, success and failure, that started between 2026-09-16 08:16 UTC and 2026-09-23 08:16 UTC. The kubernetes suites ran in 104 of them, 208 suite runs in total. The teardown drain failed once, in run 35750017261 attempt 1, the run this issue is about.
The CDI side goes upstream, I'm preparing an issue for kubevirt/containerized-data-importer. I'm not adding a workaround to
cozy_cleanup: the same hang hits a user who deletes a VM disk, and the drain check is what made it visible here.lexfrei commentedon Sep 23, 2026
Filed upstream as kubevirt/containerized-data-importer#4292: CDI deletes the importer pod once after a successful import and never retries, so a failed first delete leaves the pod and the PVC waiting on each other.
fix(e2e): collect the CDI controller log and CDI pods in cozyreport (#…