bug(flux-shard-operator): sharded helm-controller crashloops behind an HTTP proxy, freezing all sharded HelmReleases #3544

What happened

On a cluster installed behind a corporate HTTP proxy, the operator-managed
helm-controller-shard<i> Deployments created by flux-shard-operator are in
permanent CrashLoopBackOff (thousands of restarts). As a result every
HelmRelease carrying sharding.fluxcd.io/key is never reconciled — on the
affected cluster that was 150 of ~240 HelmReleases, frozen for 8 days. The
failure is silent: the shard operator reports the Deployments as created and
Ready-desired; only the pods crashloop.

The helm-controller inside the flux-aio "mega" pod (same image, same env)
keeps working, which masks the problem.

Symptom

Each shard pod:

• logs exactly one line — loading feature gate {"ExternalArtifact":true} —
  then nothing;
• lives ~30s, exits code 2 (reason: Error);
• is killed by the liveness probe before it ever serves /healthz
  (probe failureThreshold 3 × periodSeconds 10 = 30s, no startupProbe).

It never reaches "starting manager"/"starting workers".

Root cause (A/B verified)

The hang is caused by the presence of the HTTP(S)_PROXY env on the cloned
helm-controller, not by NO_PROXY contents and not by the ExternalArtifact
feature gate.

Controlled A/B with throwaway helm-controller pods using a non-existent
--watch-label-selector (so they reconcile nothing, zero blast radius),
differing only in env:

So with the proxy env present, helm-controller makes a blocking external
HTTPS call at startup that stalls through the proxy (the proxy accepts the
connection but never completes to the external host in an isolated network);
the manager never reaches the health server; the liveness probe then kills it.
With no startupProbe it can never recover → eternal crashloop.

The exact external host was not pinned down; a likely candidate is sigstore
TUF initialization (TUF_ROOT=/tmp/.sigstore is set).

Why the flux-aio helm-controller survives

It started once and has never restarted (0 restarts over 8 days) — its
single startup either succeeded or the host was reachable at that moment, and
its /tmp cache persists for the pod's lifetime. The shards restart every ~30s
with a fresh emptyDir /tmp and re-hit the stall each time. This implies a
latent risk: if the flux-aio pod is ever restarted, its helm-controller may
hang the same way.

Where the proxy comes from

• NO_PROXY=.svc is hardcoded on every flux controller in
  internal/fluxinstall/manifests/fluxcd.yaml.
• HTTP_PROXY/HTTPS_PROXY are the operator's own install-level proxy config
  (present on cozystack-operator; needed so the platform can pull through the
  proxy).
• flux-shard-operator clones the helm-controller container from flux-aio and,
  per its README, "sanitises" it (drops localhost cross-container wiring) but
  keeps the proxy env — producing a standalone controller that cannot tolerate
  the proxy at startup.

Two defects

1. flux-shard-operator propagates HTTP_PROXY/HTTPS_PROXY into the cloned
   helm-controller. A sharded helm-controller does not need external egress
   (source-controller does the fetching); the proxy env should be dropped from
   the clone, or the startup egress must be made non-blocking.
2. The cloned Deployment has no startupProbe and an aggressive 30s liveness
   window, so any slow/blocked startup is fatal and unrecoverable.

Suggested fixes

• Strip HTTP_PROXY/HTTPS_PROXY (and the redundant NO_PROXY) from the
  cloned helm-controller container in flux-shard-operator.
• Add a startupProbe to the cloned Deployment (and consider one on the
  flux-aio helm-controller too).
• Make the startup external call fail-fast / time-bounded so a proxy that
  cannot reach the host does not hang the controller forever.

Environment

• cozystack v1.6.0-rc.4, flux helm-controller v1.5.0, flux-shard-operator.
• Cluster installed behind a corporate HTTP proxy (http://<proxy>:3128).