Skip to content

fix(images): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry (harbor, mariadb) #3088

Closed

Description

@scooby87

Describe the bug

The post-delete cleanup Jobs in several app charts hardcode the cleanup image as a
moving Docker Hub tag, bypassing both digest pinning and the Cozystack cluster
images registry:

  • packages/apps/harbor/templates/hooks/cleanup.yaml:30 — image: docker.io/clastix/kubectl:v1.32
  • packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml:23 — image: docker.io/clastix/kubectl:v1.32

This was raised during review of #3077 (Harbor delete-time cleanup). It is not a
regression introduced by that PR
— the same pattern already exists in the MariaDB
cleanup hook — but #3077 duplicates it onto a second chart, so it is worth fixing
both at once before the pattern fans out further.

Two distinct problems with this line

  1. Moving tag, not pinned by digest. v1.32 is mutable: the upstream
    clastix/kubectl repository can re-push the tag to a different image at any
    time. This breaks build reproducibility (same git SHA pulls a different binary
    over time) and is a supply-chain risk (a compromised upstream tag silently
    ships into the cluster).

  2. Not routed through cozy-lib.images-registry. Cozystack exposes a
    cluster-wide registry helper for air-gapped / mirrored installs:

    packages/library/cozy-lib/templates/_cozyconfig.tpl:26
    {{- define "cozy-lib.images-registry" -}}
    {{- (index .Values._cluster "images-registry") | default "" }}
    {{- end }}
    

    Because the image hardcodes docker.io, an air-gapped cluster (no Docker Hub
    egress, everything mirrored internally) cannot pull it — and it fails at the
    worst possible moment: during app uninstall, when the post-delete hook
    runs and the operator has the least ability to intervene. A stuck/failed
    cleanup Job blocks teardown of the release.

Expected behaviour

The cleanup image should be:

  • digest-pinned, e.g. clastix/kubectl:v1.32@sha256:<digest> (tag kept for
    readability, but resolution happens by immutable digest);

  • routed through the cluster images registry, e.g.

    image: {{ include "cozy-lib.images-registry" . }}/clastix/kubectl:v1.32@sha256:<digest>

    so mirrored / air-gapped installs resolve the image from the internal registry.

Actual behaviour

Both cleanup Jobs reference docker.io/clastix/kubectl:v1.32 directly. On a
standard cluster this works; on an air-gapped/mirrored cluster the post-delete
hook Pod fails to pull the image and the cleanup never runs (orphaned PVCs / ACME
artifacts remain, and depending on hook semantics the uninstall may stall).

Scope / acceptance criteria

  • packages/apps/harbor/templates/hooks/cleanup.yaml image is digest-pinned and routed through cozy-lib.images-registry
  • packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml image is digest-pinned and routed through cozy-lib.images-registry
  • Helm unit tests updated/added to assert the rendered image goes through the registry helper
  • Confirm the chosen pattern is correct when images-registry is empty (default "") — the image must still resolve on a standard install (no leading /)

Notes

  • cozy-lib.images-registry is currently only defined, not consumed by any
    chart template (grep finds only its definition). This issue is therefore also
    the opportunity to establish the canonical "how to reference a cluster image"
    pattern, including the empty-default edge case above.
  • A dedicated digest needs to be resolved for clastix/kubectl:v1.32 and recorded
    in the chart(s).

Additional context

Raised in PR #3077 review by IvanHunters ([MINOR] finding): "Cleanup image
docker.io/clastix/kubectl:v1.32 is a moving tag, not pinned by digest, and is
not routed through cozy-lib.images-registry ... Recommend digest-pinning and
routing through cozy-lib.images-registry in both before this is fanned out
further."

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/buildIssues or PRs related to image build infrastructure, multi-arch support
    on Jun 26, 2026
  2. added
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete
    on Jul 1, 2026
  3. added 4 commits that reference this issue on Jul 2, 2026
    57bd941
    123be9e
    a76c592
    e4c6802
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/buildIssues or PRs related to image build infrastructure, multi-arch supportkind/bugCategorizes issue or PR as related to a bugpriority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to completetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions