Skip to content

Commit 587f6a7

Browse files
mattia-eleuterimattia-eleuteri
committed
feat(kamaji): default to 2 replicas, drop telemetry webhook
The kamaji-controller-manager runs ~9 sub-controllers per TenantControlPlane (bootstraptoken, coredns, konnectivity, kubeadm*, kubelet, kubeproxy, etc.) inside a single Go process, AND serves the admission webhooks for tenantcontrolplanes/datastores via the kamaji-webhook-service ClusterIP. With the upstream default replicaCount=1, every webhook RT shares CPU with the reconciler goroutines. On a multi-tenant cluster (~20 TCPs), this pushes apiserver-observed p99 for mtenantcontrolplane.kb.io and vtenantcontrolplane.kb.io from the kamaji-side ~10ms to 85-95ms. Three changes, each safe on small clusters and high-impact on large ones: * replicaCount=2 — HA-safe baseline. --leader-elect is already enabled upstream, so only one replica reconciles tenants; the follower's CPU is fully available for webhook handlers, cutting p99 ~70% on busy clusters. On a single-node cluster both pods schedule normally. * telemetry.disabled=true — drops both the --disable-telemetry handler and the telemetry.kamaji.clastix.io admission webhook entry from the ValidatingWebhookConfiguration. The latter requires patches/1.diff to wrap the telemetry block in a {{ if not .Values.telemetry.disabled }} conditional inside the vendored controller-gen file. Without the webhook entry, the apiserver no longer issues a TCP round-trip to a no-op handler on every TenantControlPlane CREATE/UPDATE/DELETE. * podAntiAffinity (preferred) — spread the two replicas across nodes when possible. preferredDuringScheduling so single-node clusters still schedule (both pods colocated, which is acceptable). Measured on a 22-TCP production cluster (Hikube, before/after): mtenantcontrolplane p99 85 → 25 ms (-71%), vtenantcontrolplane p99 92 → 25 ms (-73%), telemetry webhook impact 291 → 0. Total Kamaji admission impact -83% (830 → 140). The patches/1.diff approach (re-applied by `make update`) mirrors the pattern already used in fluxcd-operator and ingress-nginx and survives the next clastix/kamaji chart bump. Signed-off-by: mattia-eleuteri <[email protected]>
1 parent d0c12c6 commit 587f6a7

4 files changed

Lines changed: 50 additions & 0 deletions

File tree

‎packages/system/kamaji/Makefile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@ update:
1010
curl -sSL https://github.com/clastix/kamaji/archive/refs/tags/$${tag}.tar.gz | \
1111
tar -xzvf - --strip 1 kamaji-$${tag}/charts && \
1212
sed -i "/ARG VERSION/ s|=.*|=$${tag}|g" images/kamaji/Dockerfile
13+
# Re-apply Cozystack-side patches over the freshly extracted chart so
14+
# `make update` stays reproducible. Mirrors the pattern used by
15+
# packages/system/fluxcd-operator and packages/system/ingress-nginx.
16+
patch --no-backup-if-mismatch -p1 < patches/1.diff
1317

1418
image:
1519
docker buildx build images/kamaji \

‎packages/system/kamaji/charts/kamaji/controller-gen/validating-webhook.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
{{- if not .Values.telemetry.disabled }}
12
- admissionReviewVersions:
23
- v1
34
clientConfig:
@@ -19,6 +20,7 @@
1920
resources:
2021
- tenantcontrolplanes
2122
sideEffects: None
23+
{{- end }}
2224
- admissionReviewVersions:
2325
- v1
2426
clientConfig:

‎packages/system/kamaji/patches/1.diff‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
diff --git a/packages/system/kamaji/charts/kamaji/controller-gen/validating-webhook.yaml b/packages/system/kamaji/charts/kamaji/controller-gen/validating-webhook.yaml
2+
index 0000000..0000001 100644
3+
--- a/charts/kamaji/controller-gen/validating-webhook.yaml
4+
+++ b/charts/kamaji/controller-gen/validating-webhook.yaml
5+
@@ -1,3 +1,4 @@
6+
+{{- if not .Values.telemetry.disabled }}
7+
- admissionReviewVersions:
8+
- v1
9+
clientConfig:
10+
@@ -19,6 +20,7 @@
11+
resources:
12+
- tenantcontrolplanes
13+
sideEffects: None
14+
+{{- end }}
15+
- admissionReviewVersions:
16+
- v1
17+
clientConfig:

‎packages/system/kamaji/values.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,33 @@
11
kamaji:
22
etcd:
33
deploy: false
4+
# HA-safe default: two replicas serve admission webhooks behind the
5+
# kamaji-webhook-service Service. --leader-elect is already enabled
6+
# upstream, so only one replica reconciles tenant control planes — the
7+
# follower's CPU is fully available for webhook handlers, eliminating the
8+
# CPU contention that pushes p99 admission latency from ~10ms (kamaji
9+
# handler) up to ~90ms (apiserver-observed) on single-replica deployments.
10+
replicaCount: 2
11+
# Drop the telemetry handler and its admission webhook. With
12+
# telemetry.disabled=true, the controller is started with
13+
# --disable-telemetry AND the corresponding webhook entry is omitted
14+
# from the ValidatingWebhookConfiguration (see patches/1.diff), so the
15+
# apiserver no longer pays a round-trip to a no-op handler on every
16+
# TenantControlPlane CREATE/UPDATE/DELETE.
17+
telemetry:
18+
disabled: true
19+
# Soft anti-affinity: spread the kamaji controller replicas across nodes
20+
# when possible. preferredDuringScheduling means single-node clusters
21+
# still schedule both pods (just colocated, which is acceptable).
22+
affinity:
23+
podAntiAffinity:
24+
preferredDuringSchedulingIgnoredDuringExecution:
25+
- weight: 100
26+
podAffinityTerm:
27+
labelSelector:
28+
matchLabels:
29+
app.kubernetes.io/name: kamaji
30+
topologyKey: kubernetes.io/hostname
431
image:
532
pullPolicy: IfNotPresent
633
tag: v1.4.0-rc.2@sha256:e78ec08dd84eb2ee13b84b304da09ca866b6714af6dcf8b80635cae8c15f1628

0 commit comments

Comments
 (0)