Skip to content

Helm install race: cluster-autoscaler / kccm / kcsi block on admin-kubeconfig Secret that CAPI has not produced yet #2412

Closed
#2413
Closed
Helm install race: cluster-autoscaler / kccm / kcsi block on admin-kubeconfig Secret that CAPI has not produced yet#2412
#2413
Labels
kind/bugCategorizes issue or PR as related to a bug

Description

@lexfrei
Aleksei Sviridkin (lexfrei)
opened on Apr 16, 2026

Environment

  • Cozystack: v1.2.2 (verified unchanged on v1.3.0-rc.1 and master)
  • Chart: packages/apps/kubernetes
  • Kubernetes: v1.35.0+k3s3

Symptom

helm install for any apps.cozystack.io/Kubernetes application fails with context deadline exceeded after the 5-minute chart wait. The underlying error is a mount-time failure on three Deployments inside the chart:

FailedMount: MountVolume.SetUp failed for volume "kubeconfig":
  secret "<release>-admin-kubeconfig" not found

The affected workloads are:

  • <release>-cluster-autoscaler
  • <release>-kccm (cloud-controller-manager)
  • <release>-kcsi-controller (CSI controller)

HelmRelease / HelmController output

creating 35 resource(s)
beginning wait for 35 resources with timeout of 5m0s
Deployment is not ready: <ns>/<release>-cluster-autoscaler. 0 out of 1 expected pods are ready
(148 duplicate lines omitted)
Error received when checking status of resource <release>-cluster-autoscaler.
Error: 'client rate limiter Wait returned an error: context deadline exceeded'
wait for resources failed after 5m0s: context deadline exceeded

Root cause

The chart renders 35 resources in a single Helm release and relies on the default Helm wait to serialize their readiness. Three workloads in that set mount <release>-admin-kubeconfig as a non-optional Secret volume:

  • packages/apps/kubernetes/templates/cluster-autoscaler/deployment.yaml → secretName: {{ .Release.Name }}-admin-kubeconfig
  • Same pattern for kccm/manager.yaml and csi/controller.yaml

That Secret is not produced by the chart — it is produced asynchronously by the CAPI Kubeadm control-plane provider after the Cluster CR (also created in this same release) has finished provisioning the workload-cluster control plane. On clusters where control-plane provisioning exceeds the 5-minute --wait budget (typical, especially on first boot / slow node images), the Deployment pods stay ContainerCreating for the whole window, Helm reports them not-ready, and the install times out.

CAPI operators who followed along:

Cluster kubernetes-demo-k8s   PHASE: Provisioning   AGE: 3m15s

The Secret appears only once the control plane reaches Ready, but by then the Helm operation has already failed and the HelmRelease is marked install-failed.

Reproduction

  1. Cozystack v1.2.2 (also reproduced with the chart from v1.3.0-rc.1).
  2. Create a Tenant (any name).
  3. Create an apps.cozystack.io/Kubernetes Application inside that tenant with default values.
  4. Wait ~5 minutes. The HelmRelease transitions to install-failed.

Suggested fixes (pick one)

Ordered from most surgical to most invasive:

  1. Mark the admin-kubeconfig secret volume optional on the three Deployments. The workloads would start, crash-loop briefly while the Secret is absent, and recover once CAPI produces it. Single-attribute change per Deployment (optional: true on the secret volume source). Avoids the helm-wait timeout entirely but trades it for a short CrashLoopBackOff window.

  2. Move cluster-autoscaler / kccm / kcsi-controller into a separate HelmRelease that dependsOn the workload-cluster control-plane Secret being present. This is the FluxCD-idiomatic shape and mirrors how the chart already splits kubernetes-demo-k8s-cilium, kubernetes-demo-k8s-coredns, etc. as child HelmReleases that dependsOn the parent (visible in the HelmRelease list during reproduction).

  3. Add an init-container per Deployment that polls for the Secret with a short sleep loop. Inelegant but fully contained.

  4. Raise the Helm wait timeout in the cozystack-operator past the 95th-percentile CAPI provisioning time (15m). This only papers over the race — a slow node can still miss the window — but at least tightens the loop.

Why this matters

First-time-user experience: creating a Kubernetes cluster from the dashboard produces an install-failed HelmRelease, which the UI surfaces as Failed. The cluster eventually does come up (CAPI keeps going even after the HelmRelease gives up), but the retry path is non-obvious — operators typically flux suspend && resume the HelmRelease or delete and recreate the Application.

Related

No matching existing issues found in the repo for admin-kubeconfig, FailedMount, cluster-autoscaler timeout. The last commit touching templates/cluster-autoscaler/ is f6d4541 (March 2025, resource limits). The chart structure has been in this shape since the v0.0.1 prep in February 2024.

Activity

  1. dosubot
    added
    kind/bugCategorizes issue or PR as related to a bug
    on Apr 16, 2026

  2. lexfrei commented on Apr 16, 2026

    @lexfrei
    Aleksei Sviridkin (lexfrei)
    on Apr 16, 2026
    ContributorAuthor

    Closing — my reproduction was confounded. I observed Cluster stuck in Provisioning and <release>-admin-kubeconfig missing after 5 minutes, and jumped to "chart race between CAPI provisioning and helm wait".

    Digging one layer deeper on the same cluster, the Kamaji KamajiControlPlane status reports:

    Message: cannot create or update TenantControlPlane: admission webhook
  "vtenantcontrolplane.kb.io" denied the request:
  tenant-root DataStore does not exist

    kubectl get datastore returns nothing on this platform install, so the control plane was never going to come up in any timeout — the admin-kubeconfig Secret would never be produced.

    The race I described may well still exist on a correctly-configured cozystack install where control-plane provisioning genuinely races the 5m Helm wait (especially for the first tenant where Kamaji has to pull images). But I have not yet reproduced it on such a cluster. If it reappears with a clean DataStore setup, I will re-open with a proper reproduction.

    Apologies for the noise.

  3. Aleksei Sviridkin (lexfrei)
    closed this as completedon Apr 16, 2026

  4. lexfrei commented on Apr 16, 2026

    @lexfrei
    Aleksei Sviridkin (lexfrei)
    on Apr 16, 2026
    ContributorAuthor

    Reopening. On reflection the DataStore gap on my platform install and the chart-level race are independent problems, and the chart race stands on static analysis alone — I do not need a live reproduction to show it.

    Static-analysis case

    The timing contract

    templates/fluxcd.yaml and every other child HelmRelease in packages/apps/kubernetes/templates/helmreleases/ already acknowledge that the parent's admin-kubeconfig Secret is asynchronous: each child sets

    kubeConfig:
  secretRef:
    name: {{ .Release.Name }}-admin-kubeconfig
    key: super-admin.svc
dependsOn:
  - name: {{ .Release.Name }}
timeout: 10m

    Flux HelmController polls the referenced Secret and only starts the child install once it exists. This is correct.

    The inconsistency: three workloads (cluster-autoscaler, kccm, csi-controller) mount the same Secret directly as a Deployment volume in the parent HelmRelease:

    # templates/cluster-autoscaler/deployment.yaml
volumes:
  - secret:
      secretName: {{ .Release.Name }}-admin-kubeconfig
    name: kubeconfig

    No optional: true, no dependsOn, no init-container wait. Verified against current master:

    • packages/apps/kubernetes/templates/cluster-autoscaler/deployment.yaml
    • packages/apps/kubernetes/templates/kccm/manager.yaml
    • packages/apps/kubernetes/templates/csi/deploy.yaml (the only optional: true in that file is on a configMapKeyRef, not on the Secret volume)

    The wait budget

    The parent HelmRelease installed by cozystack-operator does not set install.timeout, so the default applies (5m). The parent chart contains the Cluster/KamajiControlPlane/Machine* CRs whose reconciliation (image pulls, etcd bootstrap, apiserver Ready, CAPI control-plane secret generation) routinely exceeds 5m on a cold node, and is effectively unbounded under load.

    Therefore

    Whenever CAPI control-plane provisioning exceeds the 5m helm-wait budget, those three Deployments stay ContainerCreating with FailedMount: secret "...-admin-kubeconfig" not found, the parent HelmRelease fails with context deadline exceeded, and install.remediation triggers a Helm uninstall remediation — which happens to also clean up the Cluster CR (CAPI was not done), and the whole process restarts. This is visible in any cluster where provisioning-time statistically brushes against the wait timeout, not only on a broken one.

    Platform-level confound was separate

    My dev9 install has no DataStore — Kamaji's vtenantcontrolplane admission webhook rejects every TenantControlPlane and the admin-kubeconfig Secret would never be produced no matter how large the helm wait is. That is a different problem (platform install bug). It made my repro misleading, but it does not refute the chart-level race.

    Fix options still applicable

    In priority order:

    1. Mark the kubeconfig Secret volume optional: true on the three Deployments. Pods would then start and CrashLoopBackOff until the Secret appears, which helmwait tolerates.
    2. Move cluster-autoscaler/kccm/csi-controller to a child HelmRelease that dependsOn the parent and references the Secret via spec.kubeConfig.secretRef — the same pattern already used for cilium, coredns, csi (sub-release), etc.
    3. As a workaround, set install.timeout: 15m on the parent HelmRelease rendered by cozystack-operator so the wait covers realistic control-plane bootstrap time.

    (1) is smallest-diff, (2) is most consistent with the rest of the chart, (3) is band-aid.

    Happy to ship a PR for option (1) or (2) if a maintainer confirms the preferred direction.

  5. Aleksei Sviridkin (lexfrei)
    reopened this on Apr 16, 2026
  6. Aleksei Sviridkin (lexfrei)
    added a commit that references this issue on Apr 16, 2026

    fix(api): scope 15m helm wait budget to Kubernetes Application kind

    73b80bf

  7. tym83 commented on Apr 23, 2026

    @tym83
    Timur Tukaev (tym83)
    on Apr 23, 2026
    Collaborator

    Reproduced on another Cozystack install (v1.2+, kubernetes chart be8bda8d49ff, tenant chart 6856691e01be, K8s v1.33.2). Same FailedMount + Helm 5m timeout loop on cluster-autoscaler/kccm/kcsi.

    Adding one observation that narrows the root cause from "CAPI hasn't produced kubeconfig yet" to "DataStore CR for the tenant is never produced" in our case:

    ```console
    $ kubectl get datastore -A
    NAME DRIVER READY AGE
    tenant-client123 etcd true 94d
    tenant-freedom etcd true 76d
    tenant-internal etcd true 346d
    tenant-root etcd true 147d

    expected here: one for our new tenant namespace — absent

    $ kubectl get tenantcontrolplane -A | grep kubernetes-c1

    briefly appears with Provisioning status, vanishes when HelmRelease retry triggers uninstall

    ```

    Other `Kubernetes` managed services in the same cluster (`tenant-client123/kubernetes-alice`, `tenant-root/kubernetes-*`) are Ready — their DataStore CRs exist and are `READY=true`. Cluster-wide Kamaji setup is healthy; the failure is on the new-tenant path.

    Topology

    Parent tenant nested under `tenant-root` (itself a top-level): `tenant-root` → `tenant-whmcs` → `tenant-whmcs-oojyzknp` (new). `spec.etcd: true` on the child Tenant, `spec.inherit.ingress: true`. Child's etcd StatefulSet (3 replicas) comes up Ready, but the DataStore template in the `tenant` chart doesn't appear to render — or if it renders, the resulting CR is not admitted.

    Hypothesis complement to the OP

    OP suggested options 1-4 (mark kubeconfig volume optional / split HRs / init-container / bump timeout) all treat the symptom. If DataStore CR is truly missing (not just late), a fifth path surfaces: ensure the `tenant` chart's DataStore template is unconditional when `spec.etcd: true`, and that the `kubernetes-` HelmRelease `dependsOn` it. That removes the race entirely rather than tightening tolerances.

    Happy to share full `kubectl describe` on `tenant-whmcs-oojyzknp` Tenant CR / deep-nested tenant spec / events timeline if it helps reproduce the DataStore-missing variant.

  8. Aleksei Sviridkin (lexfrei)
    added a commit that references this issue on Apr 27, 2026

    fix(kubernetes): close admin-kubeconfig race on tenant cluster bootst…

    bdf23e6
  9. Aleksei Sviridkin (lexfrei)
    closed this as completedin #2413on Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions