Proposal: improve error diagnostics surface across controller, UI, and chart authoring #2733

Summary

When an ApplicationDefinition-driven deployment gets stuck at any layer
below the HelmRelease (Pod scheduling, PVC provisioning, image pull, init
container crash, OOM, dependency-not-ready), the failure mode is invisible
from the dashboard and only diagnosable by an operator who already knows
the full Cozystack → Flux → Helm → Pod chain and which kubectl describe
to invoke.

This proposal lays out a multi-layered change to make those failures
legible at every operator touchpoint: dashboard, kubectl, and CLI
tooling. It is intentionally scoped as an umbrella — each numbered item
can be split into a focused PR.

Background

Real example, observed during external-apps catalog development
(aenix-org/cozyllm):

A VllmInference CR is created on a 3-node smoke cluster. The chart
requests 16 GiB memory; one node is tainted with drbd.linbit.com/lost-quorum
and the other two cannot satisfy the memory request. The dashboard shows:

Status:    Progressing
Age:       28m
Message:   Running 'install' action with timeout of 5m0s

The install retries every 5 minutes indefinitely. Only by running
kubectl describe pod -l app.kubernetes.io/instance=<release> does the
operator see the actual reason:

FailedScheduling: 0/3 nodes are available:
  1 node(s) had untolerated taint {drbd.linbit.com/lost-quorum: }
  2 Insufficient memory

This hides legitimate signals from operators who would otherwise
immediately know what to fix. It also disproportionately hurts new
operators who don't know which layer to inspect.

Where errors live today

Failures originate at one of several layers, each with its own status
field and event stream:

ApplicationDefinition CR  (status.conditions)            ← currently shallow
  ↓ rendered by cozystack-controller
HelmRelease            (status.conditions, status.history)
  ↓ reconciled by helm-controller
HelmChart / HelmRepository / GitRepository
  ↓ fetch by source-controller
Helm release
  ↓ helm install
Deployment / StatefulSet / Job
  ↓ manages
ReplicaSet → Pod                                          (events: FailedScheduling, FailedMount, BackOff)
  ↓ schedules onto
Node                                                      (taints, quota, allocatable)
  ↓ runs
Container                                                 (ImagePullBackOff, CrashLoop, OOMKilled, ExitCode)
  ↓ launches
Init container → application                              (logs, init-job exit codes)

The dashboard reads the top of this stack; kubectl get shows only the
CR status field. Operators have to know which kubectl describe to run
at each layer, which presumes Kubernetes internals expertise that
Cozystack's PaaS abstraction is supposed to hide.

Proposal

Six changes, ordered by ROI:

1. Controller: propagate condensed Reason into CR.status.conditions

The cozystack-controller already watches the HelmRelease it renders.
Extend it to also watch the workload resources that release creates
(Pod, PVC, Job) and to derive a single condensed Reason + actionable
message for the CR's status.conditions[type=Ready]:

status:
  conditions:
    - type: Ready
      status: "False"
      reason: SchedulingFailed
      message: "Pod cannot schedule: insufficient memory on 2 of 3 nodes (requested 16Gi, max available 12Gi). Reduce spec.resources.memory or expand cluster capacity."

Proposed Reason enum (extensible):

• SchedulingFailed (no nodes with capacity/taints/affinity)
• StorageNotAvailable (PVC unbound, no provisioner)
• ImagePullFailed
• ContainerCrashing (CrashLoopBackOff, OOMKilled)
• InitContainerFailed
• DependencyNotReady (sibling CR not yet reconciled)
• ChartRenderFailed
• ValidationFailed (admission webhook denial)
• Timeout (catch-all)

This unlocks every downstream consumer (dashboard, kubectl get -o wide,
kubectl describe, future automation) for free.

2. UI: surface events + helm logs in application detail view

Already filed in cozystack/cozystack#3104.

Dependency: item 1 — once CR.status is meaningful, the dashboard can
read it directly without chasing HelmRelease → Pod every render.

3. Chart-author conventions

Document and tool a set of conventions that chart authors can adopt to
improve first-line diagnostics:

• Sane defaults: resource requests sized for small clusters
  (e.g. 1 CPU / 4 GiB) by default; "production" sizing is opt-in via
  spec field. Reduces "my smoke cluster can't deploy this" surprise.
• Preflight init container: a bash init that validates
  prerequisites (GPU present when requested, required secret
  populated, dependency service reachable) and exits non-zero with a
  clear message. Pod surfaces CrashLoopBackOff with actionable logs
  instead of Pending forever.
• NOTES.txt with diagnostic commands: every chart ships a
  per-release help text listing the three or four kubectl commands
  to run if the release doesn't go Ready. Already useful via helm status; could become a dashboard hover-help.
• Schema descriptions with hints: openAPISchema field
  descriptions explicitly call out resource minimums ("requires GPU
  node with at least 24 GiB VRAM") so the dashboard form is
  self-documenting.

None of these require core changes — publish as recommendations in
cozystack.io/docs/applications/external.

4. Admission webhook: pre-submit validation

Reject CRs at creation time when the cluster cannot satisfy them.
Cozystack already enforces tenant quotas at reconciliation time; this
would move the same check to admission so the operator gets the answer
in one second instead of after a 5-minute Helm timeout retry loop:

$ kubectl apply -f vllm.yaml
Error from server: admission webhook "vllminferences.cozystack.io" denied the request:
  spec.resources.memory (16Gi) exceeds tenant quota (8Gi available).
  Reduce the request or contact admin to raise the quota.

Common checks the webhook could perform:

• Tenant quota vs requested resources
• StorageClass referenced in spec exists in cluster
• Chart-declared minimum requirements (via an optional
  ApplicationDefinition.spec.requirements field) vs cluster capacity

5. Diagnostic CLI tooling

A standalone command — either a kubectl plugin or a cozyctl diagnose
binary — that walks the chain CR → HelmRelease → Resources → Pod → Node and prints a structured report:

$ cozyctl diagnose vllminferences vllmtest -n tenant-client123

VllmInference/vllmtest: NotReady (SchedulingFailed)
├─ HelmRelease/vllm-inference-vllmtest: Failed (5/5 install attempts)
│  Helm: PersistentVolumeClaim is not bound
└─ Pod/vllm-inference-vllmtest-74b6c465fd-hdbb4: Pending (15m)
   Events:
     FailedScheduling: 0/3 nodes available
       - 1 untolerated taint {drbd.linbit.com/lost-quorum:}
       - 2 Insufficient memory (need 16Gi, max 12Gi available)

Likely cause: cluster cannot satisfy memory request.
Suggested actions:
  - Reduce spec.resources.memory below 12Gi
  - Recover quorum on node3 (see linstor docs)
  - Add a worker node with ≥16Gi available

The cozystack:debug skill in cozystack/ccp
already performs much of this analysis via LLM. Repackaging the same
logic as a deterministic CLI (or exposing the skill via
cozyctl debug --ai) gives operators the option without requiring
Claude Code.

6. Error catalog (documentation)

A cozystack.io/docs/troubleshooting/reasons page mapping every
Reason from item 1 to:

• What it means in plain language
• The most common root causes
• Step-by-step recovery

Low effort, links from CR status messages and dashboard for users who
want more detail.

Phasing

This is an umbrella proposal — each item can ship independently:

Phase 1 (chart-side, no core changes):

• Item 3 (chart conventions): publish in external-apps docs, retrofit
  cozyllm and the example repo. Smaller defaults + preflight + NOTES.
• Item 6 (error catalog): start the docs page even before item 1 lands.

Phase 2 (core platform UX):

• Item 1 (controller propagation): unblocks every consumer.
• Item 2 (UI surfacing): consume what item 1 produces. Track in
  cozystack-ui#14.

Phase 3 (advanced):

• Item 4 (admission webhook): scope and validation policy need design.
• Item 5 (CLI tooling): can wrap items 1 + 2 once they exist.

Open questions

• Reason enum stability: do we commit to the proposed list, or use
  a free-form string with documented conventions?
• Backward compatibility: item 1 adds fields to existing CRs — fine
  for apiextensions/v1 CRDs, but worth confirming no third-party
  controllers read the current empty status.
• Admission webhook scope: per-tenant (lives next to Tenant) or
  cluster-wide (next to ApplicationDefinitionController)? Affects
  failure-mode and ownership.
• Tooling distribution: bundle CLI with cozy-installer, ship as a
  separate Helm chart, or distribute as a Go binary via Releases?

Context

Observed during development of
aenix-org/cozyllm, an external-
apps catalog adding eight AI/ML applications to the Cozystack
marketplace. Several apps have hard resource requirements that surface
late in the install lifecycle; the silent Progressing state misleads
operators into waiting hours before discovering the install can never
succeed.

Happy to iterate on scope, split into smaller issues, or drive any of
the phases as PRs after discussion.