Skip to content

PVC cleanup hooks report success to Helm on every failure path #3566

Closed

Description

The post-delete PVC cleanup hooks in packages/apps/qdrant/templates/hooks/cleanup-pvc.yaml and packages/apps/harbor/templates/hooks/cleanup.yaml cannot fail. Every delete they perform is guarded by || echo "WARNING: ..." >&2, neither script sets -e, and both end on an unconditional echo "... complete.". The container therefore exits 0 on every path, including the ones where nothing was deleted.

Helm reads that exit code and marks the hook succeeded, so helm.sh/hook-delete-policy: hook-succeeded garbage-collects the Job and its pod. The warning went to the pod's stderr, and the pod is now gone. A cleanup that genuinely failed is indistinguishable from one that had nothing to do, both to a human reading helm/flux output and to anything automated on top of them.

Harbor swallows four separate failure paths this way:

  • kubectl delete pvc for the kept jobservice and trivy PVCs
  • kubectl delete certificate for the release's -ingress-tls Certificate
  • kubectl delete challenge for each in-flight ACME challenge
  • kubectl delete ingress for each cm-acme-http-solver-* Ingress

The two for loops add a fifth shape: they iterate over $(kubectl get ...), so a failure of the get itself yields an empty list and the loop body never runs — the same silence, one step earlier.

qdrant has the single PVC path, with the same structure.

Worth stating what this is not: it is not a claim that the deletes currently fail. It is that if one does, the resources stay behind while the uninstall reports success, and the only account of what happened is destroyed by the hook policy moments later. The orphaned-resource issues these hooks were written for are the failure mode that would recur, silently.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/storageIssues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor)
    triage/not-reproducibleIndicates an issue can not be reproduced as described
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
  2. lexfrei commented on Sep 23, 2026

    @lexfrei
    ContributorAuthor

    Fixed by #3593: both cleanup hooks now count failures and exit 1, so Helm no longer reports a failed cleanup as success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/storageIssues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor)kind/bugCategorizes issue or PR as related to a bugtriage/not-reproducibleIndicates an issue can not be reproduced as described

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions