etcd v1alpha2 adoption migration hard-fails 1.5→1.6 upgrade on clusters with an existing etcd #3243

Activity

1. myasnikovdaniil

   added

   kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bug

   on Jul 8, 2026
2. androndo commented on Jul 9, 2026

   

   Andrey Kolkov (androndo)

   on Jul 9, 2026

   Contributor

   More actions

   Ran a full 1.5.2 → 1.6.0-rc.1 etcd-adoption on a 3-node cluster and drove migration 50 all the way to a healthy adopted v1alpha2 cluster. Some deeper detail on mode 2 (cert re-issue race) plus the additional blockers that surface after it, with fixes that got the adoption to readyMembers=N / Available=True:

   Mode 2 (cert wait) — deeper root cause. It isn't only "a few seconds too early." ensure_wildcard_sans does kubectl get … 2>/dev/null | tr | grep, so on any transient GET failure (API discovery refresh, apiserver blip, throttling) the suppressed error yields an empty string that is indistinguishable from "SAN absent" → false miss, and the 120s wait never recovers. The identical check passes in isolation but the in-migration wait fails ~100% under a flaky API path. Also: the legacy operator is still running during the wait and re-applies the Certificate (stripping the just-added wildcard); the migration only scales it to 0 after the cert step.
   Fix that worked: treat an empty read as a transient error and retry (a real Certificate/Secret never has empty dnsNames/alt-names), and check the Certificate spec.dnsNames (source of truth) in addition to the issued-Secret alt-names annotation.

   Then, in order, three more blockers before the adoption completes:

   1. etcd-migrate has no kubeconfig in-cluster — aborts with error building kubeconfig: stat /root/.kube/config: no such file. It only reads a kubeconfig file and doesn't fall back to the in-cluster SA. Neither the hook Job nor migrations/50 provide one. (filed separately with a fix.)

   2. etcd-operator OOMKilled → CrashLoopBackOff — the v0.5.2 controller Deployment ships limits.memory: 128Mi; the operator needs ~250Mi (VPA's own recommendation). With updateMode: Auto the VPA is expected to rewrite the limit, but a Deployment/pod recreated out of band (e.g. via the etcd-operator upgrade fails on 1.5.x→1.6.0-rc.1: Deployment spec.selector is immutable #3242 workaround) comes up at the chart default 128Mi and OOMs (exit 137) before finishing a reconcile. Deleting the pod so the VPA webhook re-applies (→ 500Mi) fixed it. Consider a higher static floor so the operator isn't dependent on VPA timing.

   3. Adopted pods lose per-pod DNS → operator MemberList fails — after adoption the pods keep subdomain: etcd-headless (and the headless-service-name=etcd-headless annotation), but the legacy etcd-headless Service gets pruned during the transition (the v1alpha2 etcd chart's headless Service is named etcd), so etcd-N.etcd-headless.<ns>.svc stops resolving (no such host). The operator's MemberList then times out and never populates status.readyMembers, so the cluster never goes Ready even though the members are healthy. Re-creating a headless etcd-headless Service (selector on the etcd pods, publishNotReadyAddresses: true) restored DNS and the cluster went readyMembers=3, Available=True. The adoption should keep etcd-headless alive until the members have rolled onto the operator's native Service.

   Net: with the cert-wait fix + a kubeconfig for etcd-migrate + the operator memory floor + keeping etcd-headless alive, the adoption completes cleanly and in-place (pods never restarted, data intact). Happy to send a PR for the migrations/50 bits (cert-wait robustness + kubeconfig).
3. Andrey Kolkov (androndo)

   added 2 commits that reference this issue on Jul 9, 2026

   fix(etcd): keep legacy etcd-headless Service alive during v1alpha2 ad…

   ...

   2932ecc

   fix(etcd): fix etcd-operator 1.5->1.6 upgrade (immutable selector + O…

   ...

   33fa5e5
4. myasnikovdaniil

   added 3 commits that reference this issue on Jul 15, 2026

   fix(etcd): keep legacy etcd-headless Service alive during v1alpha2 ad…

   ...

   d51d691

   fix(etcd): fix etcd-operator 1.5->1.6 upgrade (immutable selector + O…

   ...

   9daaa9f

   fix(platform): make etcd v1alpha2 adoption (migration 50) robust in-c…

   ...

   c2539ca
5. myasnikovdaniil commented on Jul 21, 2026

   

   myasnikovdaniil

   on Jul 21, 2026

   ContributorAuthor

   More actions

   Completed in #3270
6. myasnikovdaniil commented on Jul 21, 2026

   

   myasnikovdaniil

   on Jul 21, 2026

   ContributorAuthor

   More actions

   Fixed by #3270 (f6a03e93f, merged 2026-07-16), which consolidates the etcd v1alpha2 transition fixes for in-cluster 1.5 → 1.6 upgrades. Both failure modes in the description and all three follow-up blockers from Andrey Kolkov (@androndo)'s comment are addressed on main:

   Reported	Fix
   1. Backup-target gate — no way to set ETCD_ADOPT_SKIP_BACKUP without hand-editing the hook Job	migrations.etcdAdoptSkipBackup is now a first-class platform value (core/platform/values.yaml → templates/migration-hook.yaml → the migration's env), covered by tests/migration_hook_skip_backup_test.yaml (bool / string / null / invalid). The refusal message now prints the exact YAML to set.
   1b. Backup-target resolution	backupstrategy-controller derives the default S3 endpoint (and per-driver scheme / TLS) from the provisioned bucket Secret instead of requiring it hand-set, so the etcd strategy resolves by default. Migration 50 also distinguishes a BucketClaim-provisioned bucket (resolved from the projected Secret) from one described by a Strategy CR, and says which it used.
   2. Cert re-issue race	Root-caused as Andrey Kolkov (@androndo) diagnosed: the SAN check now reads both the issued Secret's cert-manager.io/alt-names annotation and the Certificate's spec.dnsNames, and treats an empty read from both as a transient error to retry rather than a false "SAN absent".
   Follow-up 1 — etcd-migrate has no kubeconfig in-cluster	The migration synthesizes an in-cluster kubeconfig from the mounted ServiceAccount and passes it via --kubeconfig. Uses the in-cluster DNS name rather than $KUBERNETES_SERVICE_HOST:PORT, so it is IP-family-agnostic (IPv6-safe).
   Follow-up 2 — operator OOMKilled at limits.memory: 128Mi	Cold-start floor raised to 256Mi, with VPA minAllowed matched to it, so the operator never depends on VPA admission timing to avoid a crash loop.
   Follow-up 3 — adopted pods lose per-pod DNS, MemberList fails	extra/etcd now ships a chart-managed transitional headless etcd-headless Service (publishNotReadyAddresses: true, selector mirroring the operator's native Service), the DNS counterpart of the legacy *.etcd-headless.<ns>.svc SAN already kept for the adoption window. Removable together with that SAN once members roll onto the native subdomain.

   #3270 also carries the fix for the immutable controller-Deployment selector (#3242), which blocks the same upgrade path.

   Availability

   The fix is on main and will ship in v1.6.0-rc.3. Note that v1.6.0-rc.2 was tagged 2026-07-10, six days before the merge, so neither v1.6.0-rc.1 nor v1.6.0-rc.2 contains it — if you are testing the upgrade path against either of those, you will still hit the original hard-fail. Test against rc.3 or a main build.

   Known residuals (not blocking, filing separately if they resurface)

   • The cert re-issue wait is still a fixed 30 × 4s; it was not made configurable, since the empty-read retry addresses the actual cause rather than the symptom.
   • The SAN patch runs before the legacy operator is scaled to 0, and the wait loop patches once and then polls only the Secret. If the legacy operator re-applies the Certificate mid-wait, there is no re-patch. Not observed in the live 1.5.2 → 1.6.0-rc.1 adoption, which completed to readyMembers=3 / Available=True.

   Closing — please reopen if a 1.5 → 1.6 upgrade on a build that includes #3270 still fails.
7. myasnikovdaniil

   closed this as completedon Jul 21, 2026