[harbor] Expose replicas and zone-aware placement for the request-path components (nginx, core, registry) #4824

Problem / motivation

We want to run the Harbor app as a highly available, zone-resilient registry. Today that isn't possible.

The only replica knobs the app exposes are database.replicas and redis.replicas. Every docker pull goes through nginx → core (/service/token) → registry, and those components always run as a single pod. Even with 3 PostgreSQL and 3 Redis instances, losing the node that hosts nginx, core or registry stops all pulls and pushes until the pod is rescheduled and ready. For tenants this means new pods and replacement nodes can't pull images, and CI pipelines fail.

The replicated data tier isn't zone-aware either. packages/system/harbor/templates/database.yaml (CNPG Cluster) and redis.yaml (RedisFailover) set no affinity or topology spread. 3 instances can all land in the same zone, and CNPG's default anti-affinity only works per node.

Where it comes from:

• packages/apps/harbor/templates/harbor.yaml only passes down database.replicas and redis.replicas (L147, L153 on main). The WorkloadMonitors for core, registry and portal hard-code replicas: 1 / minReplicas: 1 (L248, L266, L282).
• The vendored upstream chart (packages/system/harbor/charts/harbor, goharbor 1.19.1) already supports replicas, podDisruptionBudget, affinity and topologySpreadConstraints for nginx, portal, core, jobservice, registry, trivy and exporter. Only the wrapper is missing them.
• Registry storage is pinned to S3 (imageChartStorage.type: s3, COSI BucketClaim), so registry replicas share no local state. Core keeps its state in PostgreSQL and Redis.
• jobservice.jobLoggers defaults to [file] on a ReadWriteOnce jobLog PVC. This is the only thing preventing jobservice from scaling out. database and stdout are supported alternatives.

Proposed solution

1. Expose replicas for nginx, core and registry, and optionally portal. Default stays 1, so existing installs don't change. Example:

   nginx:    { replicas: 2 }
   core:     { replicas: 2, resourcesPreset: t1.medium }
   registry: { replicas: 2, resourcesPreset: t1.medium }

2. When replicas > 1 for a component, render:

   • podDisruptionBudget: { enabled: true, maxUnavailable: 1 }
   • topologySpreadConstraints on topology.kubernetes.io/zone (whenUnsatisfiable: ScheduleAnyway), plus preferred hostname anti-affinity.

   These are values of the vendored chart, so no template changes are needed in it.

3. Derive the WorkloadMonitor replicas / minReplicas from the new values instead of hard-coding 1.

4. Add zone spread to the CNPG Cluster (affinity.topologyKey: topology.kubernetes.io/zone) and to the RedisFailover redis and sentinel pods, so that replicas: 3 survives the loss of a zone.

5. Optional, as a follow-up: jobservice.replicas, switching jobLoggers to [database] when replicas > 1.

Expected behavior: with replicas: 2 on nginx, core and registry in a multi-zone cluster, draining or losing one node or zone keeps docker login / pull / push working.

Scope check

• This affects a single component / package (if it spans many, consider a design proposal)
• I can describe the expected behavior concretely (if not, consider a Discussion first)

The changes are limited to packages/apps/harbor and packages/system/harbor (wrapper values and templates only; the vendored chart is unchanged).

Alternatives considered

• Patching the rendered Deployments or the harbor-*-system HelmRelease: reverted on every reconcile, same situation as [harbor] No way to disable registry upload purging — every surface that could is Helm-owned and reverts #3972.
• Only adding replicas to the data tier: already possible, but it doesn't improve availability while nginx, core and registry stay single pods.
• Per-app schedulingClass (feat(apps): per-app schedulingClass field for selective workload placement #2622): complementary. It controls where pods go, not how many, so it doesn't remove the single point of failure.
• Tenants running their own Harbor in a tenant Kubernetes cluster: works, but defeats the purpose of the managed app.

Additional context

• [harbor] Add managed Harbor container registry #2058: original Harbor app (replicas only designed for PostgreSQL/Redis)
• feat(apps): per-app schedulingClass field for selective workload placement #2622: per-app schedulingClass
• e2e: harbor suite flakes when rfs-harbor-test-redis Sentinel HA never reaches readiness #3445 / fix(harbor): guarantee CPU for redis sentinels #3446: Redis Sentinel convergence
• [harbor] No way to disable registry upload purging — every surface that could is Helm-owned and reverts #3972: registry settings not reachable from harbor.spec
• Observed on a v1.6.0 install: core, registry, portal, nginx, jobservice and trivy each run exactly one pod.