refactor(kubernetes): replace bootstrap workarounds with proper dependency management #2497

Context

PR #2413 closed the infinite remediation loop during tenant Kubernetes cold start by introducing:

• wait-for-kubeconfig init container with 10m deadline
• Extended HelmRelease install timeout (15m via annotation)
• Status beacon ConfigMap for etcd-less state

This is a working workaround for the Kamaji cold-start race condition, but not a proper long-term solution.

Problems with current approach

1. Timeout doesn't distinguish failure modes: 15m timeout is the same for legitimate cold start and real problems (etcd disabled, storage shortage, Kamaji throttling). This adds 10+ minutes to detecting actual configuration errors.

2. Status beacon via template conditionals: Growing complexity in templates, no validation at API level. The cozystack.io/status-beacon annotation is a forward-looking marker with no current consumer.

3. No pre-flight checks: Configuration errors are only discovered after timeout expiration, poor user experience.

Proper solution options

Option A: Separate HelmRelease topology with dependsOn

• Split kubernetes chart into bootstrap + workload HelmReleases
• Bootstrap HelmRelease waits for Kamaji provisioning
• Workload HelmReleases have dependsOn: [bootstrap]
• Pros: Correct dependency model, fail-fast behavior
• Cons: Major refactoring, new reconciliation topology, 3-4 PRs of work

Option B: Validating webhook

• Admission webhook validates prerequisites (etcd DataStore, storage, etc.)
• Fail-fast on invalid configuration
• Status beacon replaced with proper API field
• Pros: Validation at API boundary, clean user experience
• Cons: New component to deploy, manage certificates, operate

Option C: Kamaji integration improvements

• Kamaji emits Events when admin-kubeconfig is created
• Chart watches Events instead of polling with timeout
• Pros: Event-driven, no timeout workaround needed
• Cons: Depends on upstream Kamaji changes

Acceptance criteria

• Legitimate cold start doesn't require extended timeout
• Configuration errors (missing etcd, storage issues) fail immediately, not after 15m
• Status beacon replaced with validated API mechanism or removed
• Solution documented and covered by tests
• No regression in cold-start success rate

Related

• Original issue: Helm install race: cluster-autoscaler / kccm / kcsi block on admin-kubeconfig Secret that CAPI has not produced yet #2412
• Workaround PR: fix(kubernetes): close admin-kubeconfig race on tenant cluster bootstrap #2413