Skip to content

[kafka] Migrate from ZooKeeper to KRaft mode - #2352

Merged
scooby87 merged 18 commits into
cozystack:mainfrom
matthieu-robin:KRaft
Sep 22, 2026
Merged

scooby87 merged 18 commits into
cozystack:mainfrom
matthieu-robin:KRaft

Conversation

@matthieu-robin

@matthieu-robin Matthieu ROBIN (matthieu-robin) commented Apr 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Replace ZooKeeper-based Kafka deployments with KRaft (Kafka Raft) metadata mode
  • Existing ZooKeeper instances migrate automatically and in place via a Helm pre-install,pre-upgrade hook Job that drives Strimzi's ZK-to-KRaft migration state machine
  • New instances deploy directly in KRaft mode with broker and controller KafkaNodePools
  • The node pools are owned by the migration hook, not by regular chart manifests (see "Why the pools are hook-managed")
  • Kafka version is user-configurable (v3.9 default, v3.8 available)
  • Strimzi operator is unchanged in this PR (stays at 0.45.1-rc1); 0.45.x is the last line that supports ZooKeeper→KRaft migration
  • ZooKeeper removed from the dashboard UI (ApplicationDefinition)

Migration strategy

Both fresh installs and ZooKeeper→KRaft migrations are driven by a single Helm hook Job (templates/migration-hook.yaml, pre-install,pre-upgrade) that owns the KafkaNodePools. On every install and upgrade the Job:

  1. Ensures both node pools exist (idempotent): a broker pool named exactly kafka for a migrating cluster — so Strimzi adopts the existing <cluster>-kafka-N brokers and their PVCs in place — or b-<hash> for a fresh cluster, plus a controller pool c-<hash>. The controller pool pins strimzi.io/next-node-ids above the broker range so the broker pool keeps node IDs [0..N-1] and Strimzi cannot hand the existing broker IDs to the fresh controllers. The shared kafka name is claimed atomically (via kubectl create) so only one migration per namespace can win it.
  2. Inspects the Kafka CR: no CR → fresh install (pools ensured; the main apply then creates the KRaft Kafka CR, nothing to migrate); already strimzi.io/kraft=enabled or status.kafkaMetadataState: KRaft → done; any other read error → fail closed. Otherwise (a ZooKeeper cluster) it migrates.
  3. Annotates strimzi.io/node-pools=enabled then strimzi.io/kraft=migration, polls status.kafkaMetadataState through ZooKeeper → KRaftMigration → KRaftDualWriting → KRaftPostMigration → PreKRaft → KRaft, and only once a safe state is reached flips strimzi.io/kraft=enabled. If it times out in an intermediate state it aborts without finalising, so Helm fails the upgrade and the next reconcile retries from where it left off.

Data is migrated in place: brokers keep their node IDs and reuse their existing PVCs, nothing is copied. Migration is one-way — Strimzi has no supported KRaft→ZooKeeper rollback once complete, so lowering the version field or rolling the app back to a pre-KRaft chart is unsupported (take a backup first).

Why the pools are hook-managed

The hook must create the pools and walk the migration state machine before the main release apply. A resource a hook creates cannot also be a regular chart manifest across the ZK→KRaft boundary: Helm's main apply finds the pool live in the cluster but absent from the previous (ZooKeeper) release revision and aborts with original object KafkaNodePool ... not found, leaving the release failed even though the cluster migrated fine (reproduced under server-side apply, the helm-controller default). Managing the pools only from the hook avoids that, so templates/kafkanodepools.yaml and the deployed-version gate ConfigMap (templates/version.yaml) are removed and the pools are applied idempotently by the Job on every install and upgrade.

Bug fixes included

  • ZooKeeper metricsConfig referenced wrong ConfigMap key (kafka-metrics-config.yml instead of zookeeper-metrics-config.yml)
  • ZooKeeper storageClass used kafka.storageClass instead of zookeeper.storageClass
  • WorkloadMonitor used .Values.replicas instead of .Values.kafka.replicas
  • validateNameLength now caps a Kafka app name at the KRaft controller pod-hostname budget (<release>-c-<hash>-<id> must fit 63 chars), instead of admitting names that later fail the chart render forever

What's next

A follow-up PR will upgrade Strimzi to 0.51.0 and Kafka to 4.2 once all instances are confirmed migrated to KRaft. That PR is already prepared on branch KRaft-upgrade-strimzi-051.

Test plan

  • helm template renders correctly for new KRaft instances (no ZooKeeper refs)
  • helm template with custom values (1 replica, external, topics, v3.8)
  • Version validation rejects unsupported versions via JSON schema
  • helm-unittest: 86/86 across 9 suites
  • hack/kafka-migration-hook.bats: 12/12 — the real hook script executed against a stub kubectl (resolution branches, fail-closed reads, atomic collision guard, finalize guard)
  • Operator template renders with the unchanged Strimzi 0.45.1-rc1
  • Live migration on a disposable cluster under server-side apply: fresh install → healthy KRaft, release deployed; in-place migration of a 20000-message ZooKeeper cluster → release deployed, broker PVCs adopted in place (same UIDs), and the KRaft data byte-for-byte identical to the pre-migration ZooKeeper data (matching sha256 over every partition:offset:key:value)

Release note

[kafka] Enable KRaft metadata mode with automatic, in-place ZooKeeper-to-KRaft migration. Kafka version is now configurable (v3.9, v3.8). The Strimzi operator is unchanged (0.45.1-rc1).

Summary by CodeRabbit

  • New Features

    • Added chart-level Kafka version selector (default v3.9) and KRaft-aware node pool rendering with controller storage sizing.
  • Updates

    • Default Kafka appVersion updated to 3.9.0; operator and image tag defaults aligned to 0.45.0.
    • Templates now omit ZooKeeper resources and metrics when KRaft is in use.
  • Tests

    • New Helm unit tests for Kafka CR, node pools, metrics, topics, dashboard mappings, and workload monitors.
  • Documentation

    • Values, schema, and README updated to document new version and controller storage options.

@dosubot dosubot Bot added size/XXL This PR changes 1000+ lines, ignoring generated files kind/feature Categorizes issue or PR as related to a new feature labels Apr 7, 2026
@coderabbitai

coderabbitai Bot commented Apr 7, 2026 •

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Add chart-driven Kafka versioning (default v3.9), KRaft-aware templates that omit ZooKeeper when appropriate, introduce KafkaNodePool templates, update Strimzi operator chart/CRDs and image tags, add Helm unit tests, and adjust e2e test to assume KRaft defaults.

Changes

Cohort / File(s) Summary
Kafka version & values
packages/apps/kafka/Chart.yaml, packages/apps/kafka/files/versions.yaml, packages/apps/kafka/values.yaml, packages/apps/kafka/values.schema.json
Add top-level version (default v3.9), versions map (v3.9 -> 3.9.0, v3.8 -> 3.8.1), and update Chart appVersion to 3.9.0.
Core templates
packages/apps/kafka/templates/_versions.tpl, packages/apps/kafka/templates/kafka.yaml, packages/apps/kafka/templates/kafkanodepools.yaml
New versionMap template; kafka.yaml resolves version, looks up existing Kafka to compute KRaft vs ZooKeeper and conditionally omits spec.zookeeper; new kafkanodepools template renders KafkaNodePool resources (combined or separate controller/broker).
Monitoring & RBAC templates
packages/apps/kafka/templates/dashboard-resourcemap.yaml, packages/apps/kafka/templates/metrics-configmap.yaml, packages/apps/kafka/templates/workloadmonitor.yaml
Templates use lookup to set $isKRaft and adjust RBAC resourceNames, metrics ConfigMap entries, WorkloadMonitor kinds/selectors/replicas; Zookeeper-related resources omitted when KRaft.
Helm unit tests
packages/apps/kafka/tests/*_test.yaml (multiple files)
Add tests for kafka.yaml, kafkanodepools, dashboard-resourcemap, workloadmonitor, metrics-configmap, topics; cover version resolution, KRaft conditions, node-pool rendering, metrics, storage, and replicas.
E2E test script
hack/e2e-apps/kafka.bats
Inline Kafka manifest set to spec.version: "v3.9" and removed spec.zookeeper; remove Zookeeper readiness checks and explicit Zookeeper PVC deletion in teardown.
API & cozyRD schema
api/apps/v1alpha1/kafka/types.go, packages/system/kafka-rd/cozyrds/kafka.yaml
Add spec.version field and Version type (enum v3.9,v3.8, default v3.9); add kafka.controllerStorageSize schema/field and adjust zookeeper description and dashboard keys order.
Strimzi operator chart & values
packages/system/kafka-operator/Makefile, packages/system/kafka-operator/charts/strimzi-kafka-operator/*
Downgrade/operator chart metadata and default image tags from 0.45.1-rc1 → 0.45.0, adjust kafkaBridge tag; update image-tag mappings to include -kafka-3.9.0.
Strimzi CRD schema cleanup
packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/*
Remove OpenAPI schema properties: resources.claims[].request, securityContext.seLinuxChangePolicy, and securityContext.supplementalGroupsPolicy across multiple CRDs.
Docs & Makefile
packages/apps/kafka/README.md, packages/apps/kafka/Makefile
Document new version and kafka.controllerStorageSize parameters and add test Makefile target for helm unittest ..

Sequence Diagram(s)

sequenceDiagram
    participant Helm as Helm renderer
    participant K8s as Kubernetes API
    participant Strimzi as Strimzi operator
    participant Storage as PV/PVC

    Helm->>K8s: lookup Kafka CR and KafkaNodePool (lookup)
    alt Kafka absent or status.kafkaMetadataState == "KRaft"
        Helm->>K8s: render Kafka CR without `spec.zookeeper` (annotate strimzi.io/kraft)
        Helm->>K8s: render KafkaNodePool(s) (combined or broker/controller)
    else
        Helm->>K8s: render Kafka CR with `spec.zookeeper` and zookeeper metrics
        Helm->>K8s: render KafkaNodePool(s) as appropriate
    end
    K8s->>Strimzi: deliver Kafka/KafkaNodePool resources
    Strimzi->>K8s: reconcile -> create/update Pods, Services, Endpoints, PVCs
    K8s->>Storage: bind PVCs (persistent-claim volumes)
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: migrating Kafka deployments from ZooKeeper to KRaft mode.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dosubot dosubot Bot added size/XL This PR changes 500-999 lines, ignoring generated files and removed size/XXL This PR changes 1000+ lines, ignoring generated files labels Apr 7, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new VM import system, including the vm-import application, a vm-adoption-controller for integrating imported VMs into Cozystack, and updates to the Kafka operator and chart. The reviewer highlighted that the health check server port in the controller is not exposed in the deployment, and suggested reconsidering the strict error handling for missing VM templates and the name length validation for VMInstances.

I am having trouble creating individual review comments. Click here to see my feedback.

packages/system/vm-adoption-controller/images/controller/main.go (110-127)

high

The health check server is running on port 8081, but this port is not defined in the Deployment spec. Ensure the port is exposed in the Deployment to allow probes to function correctly.

packages/system/vm-adoption-controller/images/controller/main.go (406-408)

medium

The controller returns an error if spec.template is missing. Consider if this is too strict for all VM types, or if a more graceful handling (e.g., logging and skipping) is appropriate for partially defined VMs.

packages/system/vm-adoption-controller/images/controller/main.go (522-524)

medium

The controller fails if the generated VMInstance name exceeds 63 characters. Consider truncating the name or adding a hash suffix to ensure the name remains valid while avoiding collisions.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🧹 Nitpick comments (17)
packages/system/forklift-operator/Makefile (1)

15-15: Consider simplifying the namespace document stripping logic.

The Python one-liner is complex and hard to maintain. A more readable approach using yq (if available) or a simpler Python script would improve maintainability.

♻️ Suggested alternative using grep/awk
-	python3 -c "import sys; content=open('templates/forklift-operator.yaml').read(); docs=content.split('---'); result=[d for d in docs if 'kind: Namespace' not in d.strip().split(chr(10))[0:5] or not any(l.strip()=='kind: Namespace' for l in d.strip().split(chr(10)))]; open('templates/forklift-operator.yaml','w').write('---'.join([d for d in result if d.strip() and not any(l.strip()=='kind: Namespace' for l in d.strip().split(chr(10)))]))"
+	python3 -c "\
+import yaml; \
+docs = list(yaml.safe_load_all(open('templates/forklift-operator.yaml'))); \
+filtered = [d for d in docs if d and d.get('kind') != 'Namespace']; \
+open('templates/forklift-operator.yaml', 'w').write('---\n'.join(yaml.dump(d) for d in filtered))" || \
+	awk 'BEGIN{RS="---"; ORS="---"} !/kind: Namespace/' templates/forklift-operator.yaml > tmp && mv tmp templates/forklift-operator.yaml
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/forklift-operator/Makefile` at line 15, The current complex
python3 -c one-liner that edits 'templates/forklift-operator.yaml' should be
replaced with a simpler, maintainable step: either call a short standalone
script (e.g., scripts/remove_namespace_docs.py) or use yq to filter out
documents with "kind: Namespace", and update the Makefile to invoke that
script/command instead of the long inline python command; locate the existing
python3 -c "import sys; content=..." one-liner in the Makefile and swap it for a
readable command that clearly removes YAML documents where kind: Namespace from
templates/forklift-operator.yaml.
packages/apps/vm-import/docs/scripts/adopt-vm.sh (2)

69-69: Use bash parameter expansion instead of sed.

Per shellcheck SC2001, bash parameter expansion is more efficient and avoids spawning a subprocess.

♻️ Suggested fix
-            DISK_REF=$(echo "$disk" | sed "s/^vm-disk-//")
+            DISK_REF="${disk#vm-disk-}"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/vm-import/docs/scripts/adopt-vm.sh` at line 69, Replace the
sed-based strip with bash parameter expansion: in the assignment for DISK_REF
(currently using echo "$disk" | sed "s/^vm-disk-//"), use parameter expansion on
the variable disk (remove the "vm-disk-" prefix via ${disk#vm-disk-}) so you
avoid spawning a subprocess and satisfy SC2001; ensure this is used wherever
DISK_REF is computed from disk.

102-102: Use bash parameter expansion for consistency.

Same pattern as line 69 - prefer bash built-ins over external commands.

♻️ Suggested fix
-            SUBNET_NAME=$(echo "$net" | sed 's/^.*\///')
+            SUBNET_NAME="${net##*/}"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/vm-import/docs/scripts/adopt-vm.sh` at line 102, Replace the
external sed call used to set SUBNET_NAME from the net variable with bash
parameter expansion to extract the substring after the last '/' so it matches
the pattern used earlier (same approach as on line 69); update the assignment
that sets SUBNET_NAME to use parameter expansion on net to avoid spawning sed
and keep consistent shell-builtins usage.
packages/apps/kafka/templates/kafkanodepools.yaml (1)

70-84: Controller pool is missing resources specification.

The broker pool (line 50) and combined pool (line 23) include the resources specification via cozy-lib.resources.defaultingSanitize, but the controller pool does not. This may be intentional if controllers have minimal resource needs, but it creates an inconsistency.

💡 Suggested fix to add resources
 spec:
   replicas: {{ $controllerReplicas }}
   roles:
     - controller
+  resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 4 }}
   storage:
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kafka/templates/kafkanodepools.yaml` around lines 70 - 84, The
controller node pool spec is missing the resources field; add the same resources
stanza used in the broker and combined pools by invoking the cozy-lib helper
(cozy-lib.resources.defaultingSanitize) inside the controller pool spec (the
block under "spec:" for the controller role) so controllers include a resources
section consistent with the other pools; ensure you place the resources entry
alongside replicas/roles/storage in the controller pool definition and use the
same template invocation and parameters as the broker/combined pool.
packages/system/forklift/templates/forklift-cr.yaml (1)

3-5: Consider using templated namespace.

The namespace is hardcoded to cozy-forklift. If the Helm release is deployed to a different namespace, this CR will still target cozy-forklift, which may cause deployment issues or require cross-namespace permissions.

If this is intentional (e.g., the ForkliftController must always be in cozy-forklift), consider adding a comment explaining the constraint.

💡 Suggested change to use release namespace
 metadata:
   name: forklift-controller
-  namespace: cozy-forklift
+  namespace: {{ .Release.Namespace }}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/forklift/templates/forklift-cr.yaml` around lines 3 - 5, The
namespace in the custom resource is hardcoded to "cozy-forklift" which prevents
the CR from following the Helm release namespace; update metadata.namespace to
use the Helm release namespace (e.g., the release namespace template variable)
so the CR is deployed into the same namespace as the release, or if the
controller must always live in "cozy-forklift" add a clear comment explaining
that constraint; target the metadata.namespace field and validate behavior via
the ForkliftController CR (metadata.name: forklift-controller) after change.
packages/system/vm-adoption-controller/images/controller/Dockerfile (1)

13-13: Hardcoded GOARCH=amd64 limits multi-architecture builds.

Consider using build arguments to support multi-arch images (e.g., arm64), which is increasingly common for Kubernetes clusters.

♻️ Suggested improvement for multi-arch support
-RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o controller main.go
+ARG TARGETOS=linux
+ARG TARGETARCH=amd64
+RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -a -o controller main.go
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/vm-adoption-controller/images/controller/Dockerfile` at line
13, The Dockerfile currently hardcodes GOARCH=amd64 in the build step (RUN
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o controller main.go); change
this to use a build ARG (e.g., ARG TARGETARCH) and substitute GOARCH with that
ARG (or default to amd64) so the build supports multi-arch (arm64, amd64) when
invoked with buildx/--platform; update the Dockerfile to declare the ARG and use
it in the RUN that builds main.go to produce the controller binary.
packages/system/vm-instance-rd/cozyrds/vm-instance.yaml (1)

11-11: Consider adding spec.fullnameOverride to keysOrder for stable UI ordering.

Since fullnameOverride is now in the OpenAPI schema, explicitly ordering it can keep form layout deterministic.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/vm-instance-rd/cozyrds/vm-instance.yaml` at line 11, The
OpenAPI schema now includes fullnameOverride but the form ordering (keysOrder)
omits spec.fullnameOverride; add "spec.fullnameOverride" into the keysOrder
array where other spec fields (e.g., "spec.instanceProfile", "spec.resources",
"spec.runStrategy") are listed so the UI renders fullnameOverride in a stable,
deterministic position; touch the keysOrder entry in the vm-instance.yaml schema
and insert spec.fullnameOverride next to the other spec.* keys.
hack/e2e-apps/kafka.bats (1)

3-40: Prefer framework-level teardown over inline cleanup in this E2E test.

Please move the in-test kubectl delete cleanup to shared teardown so cleanup is consistently handled across failures and reruns.

Based on learnings: In E2E bats tests like hack/e2e-apps/harbor.bats, replace inline cleanup in test bodies with framework/global teardown cleanup.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@hack/e2e-apps/kafka.bats` around lines 3 - 40, The test "Create Kafka"
contains an inline cleanup line (kubectl -n tenant-test delete
kafka.apps.cozystack.io $name) which should be removed from the test body and
instead registered in the framework/global teardown used by other E2E tests
(e.g., the pattern used in harbor.bats); update the test to omit the delete, and
add the Kafka resource cleanup (deleting kafka $name in namespace tenant-test)
to the shared teardown hook so cleanup runs even on failures and reruns.
packages/apps/kafka/tests/metrics-configmap_test.yaml (1)

29-35: Add a migration-path test case for ZooKeeper metrics presence.

Current tests cover new-instance KRaft defaults well; consider adding one case that simulates existing ZooKeeper metadata state and asserts data["zookeeper-metrics-config.yml"] is rendered during migration stages.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kafka/tests/metrics-configmap_test.yaml` around lines 29 - 35,
Add a new test case alongside the existing "it: does not include zookeeper
metrics for new instance" that simulates an existing ZooKeeper metadata state
and asserts the zookeeper metrics config is rendered; specifically, create an
"it" block describing migration with pre-existing ZK metadata and include an
asserts entry using "exists: path: data[\"zookeeper-metrics-config.yml\"]" so
the test expects that key to be present during migration rendering (mirror the
structure used by the existing test but flip the notExists to exists to validate
the migration path).
packages/apps/vm-import/docs/examples/advanced-import.yaml (1)

18-18: Thumbprint format appears truncated in example.

The SHA-1 thumbprint should be 40 hexadecimal characters (20 bytes). The example shows "11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33" which is only 19 bytes. Consider using a full 20-byte example for accuracy:

-  thumbprint: "11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33"
+  thumbprint: "11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33:44"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/vm-import/docs/examples/advanced-import.yaml` at line 18, The
thumbprint value in the advanced-import.yaml example is missing one byte (SHA-1
should be 20 bytes / 40 hex chars); update the "thumbprint" example to a full
20-byte SHA-1 formatted as 40 hexadecimal characters (optionally colon-separated
per the existing style) so the example is accurate—locate the thumbprint field
in the examples/advanced-import.yaml and replace the truncated string with a
complete 20-byte hex thumbprint.
packages/apps/vm-import/docs/examples/monitoring-import.md (1)

168-169: Consider using runStrategy instead of deprecated running field.

Modern KubeVirt versions deprecate spec.running in favor of spec.runStrategy. The README.md in this same package already uses runStrategy correctly (lines 147-149).

📝 Suggested fix for consistency
 # Start VMs
-kubectl patch vm nginx-frontend-01 -n production --type merge -p '{"spec":{"running":true}}'
+kubectl patch vm nginx-frontend-01 -n production --type merge -p '{"spec":{"runStrategy":"Always"}}'
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/vm-import/docs/examples/monitoring-import.md` around lines 168
- 169, The patch command shown uses the deprecated VM field spec.running; update
the example to use spec.runStrategy instead by changing the patched payload to
set spec.runStrategy (e.g., "RunStrategyAlways" or the appropriate strategy)
rather than spec.running, ensuring consistency with the README example that
already uses runStrategy and preserving the same resource (vm nginx-frontend-01)
and namespace (production).
packages/system/vm-adoption-controller/images/controller/main.go (4)

119-127: Add timeout context for graceful server shutdown.

Using context.Background() for shutdown means it could hang indefinitely. Consider using a timeout context.

Proposed fix
 		go func() {
 			<-ctx.Done()
-			server.Shutdown(context.Background())
+			shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+			defer cancel()
+			server.Shutdown(shutdownCtx)
 		}()
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/vm-adoption-controller/images/controller/main.go` around
lines 119 - 127, The shutdown goroutine currently calls
server.Shutdown(context.Background()) which can hang; change it to create a
timeout context (e.g. ctxShutdown, cancel :=
context.WithTimeout(context.Background(), 5*time.Second)), call
server.Shutdown(ctxShutdown) and defer cancel() (or call cancel after Shutdown)
and log any Shutdown error; update the goroutine that references server and ctx
so it uses the new ctxShutdown and cancel to ensure the HTTP server shuts down
within the timeout.

111-118: Unchecked error from w.Write.

The return value from w.Write is ignored. While unlikely to fail for small writes, this is flagged by linters.

Proposed fix
 		mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
 			w.WriteHeader(http.StatusOK)
-			w.Write([]byte("ok"))
+			_, _ = w.Write([]byte("ok"))
 		})
 		mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
 			w.WriteHeader(http.StatusOK)
-			w.Write([]byte("ok"))
+			_, _ = w.Write([]byte("ok"))
 		})
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/vm-adoption-controller/images/controller/main.go` around
lines 111 - 118, The health/readiness handlers registered via mux.HandleFunc
(the "/healthz" and "/readyz" functions) ignore the error returned by w.Write;
update both handlers to capture the return value (err := w.Write(...)) and
handle it (e.g., log the error with log.Printf or klog.Errorf) if non-nil so
linters stop flagging the unchecked write error; do this in the anonymous
handler closures that call w.Write and keep the existing
w.WriteHeader(http.StatusOK) behavior.

145-163: Consider using controller-runtime for watch-based reconciliation.

The current polling approach with a ticker works but has higher API server load compared to watch-based reconciliation. For production workloads, controller-runtime with informers provides:

  • Event-driven reconciliation (lower latency)
  • Built-in leader election
  • Automatic rate limiting and retries
  • Standard metrics and logging

This is a larger architectural consideration for future iterations.

Based on learnings: "Follow controller-runtime patterns and kubebuilder style for Go code."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/vm-adoption-controller/images/controller/main.go` around
lines 145 - 163, The Run method on AdoptionController currently polls with a
time.NewTicker (watchInterval) and calls c.reconcile(ctx), which causes higher
API load; refactor to use controller-runtime patterns: create a
SetupWithManager(ctrl.Manager) for AdoptionController, register
watches/informers for VirtualMachine resources and any related types, implement
Reconcile as the controller-runtime reconcile.Reconciler (move logic from
reconcile into Reconcile(ctx, req)), and use mgr.Start(ctx) (with manager
configured for leader election, rate-limiting and metrics) instead of the
ticker-driven AdoptionController.Run; ensure you remove or deprecate the polling
Run and wire the controller via ctrl.NewControllerManagedBy(mgr).For(...) so
events drive reconciliation.

691-696: Silently skipping missing DataVolumes could mask issues.

If a DataVolume is expected but missing, logging a warning and continuing may hide problems. Consider whether this should be an error that triggers rollback, or at minimum log at a higher level.

 		dv, err := c.dynamicClient.Resource(dvGVR).Namespace(namespace).Get(ctx, dvName, metav1.GetOptions{})
 		if err != nil {
-			klog.Warningf("DataVolume %s/%s not found, skipping Helm labeling: %v", namespace, dvName, err)
+			klog.Errorf("DataVolume %s/%s not found, skipping Helm labeling: %v", namespace, dvName, err)
 			continue
 		}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/vm-adoption-controller/images/controller/main.go` around
lines 691 - 696, The loop that fetches DataVolumes (for _, dvName := range
dvNames) currently logs a warning and continues when
c.dynamicClient.Resource(dvGVR).Namespace(namespace).Get(...) returns an error,
which can hide missing expected resources; change this to fail-fast or at least
escalate the handling: either return/propagate an error from the enclosing
function when a DataVolume is missing (stop the Helm labeling and trigger
rollback) or collect missing dvNames and after the loop return a consolidated
error; if you must continue, change klog.Warningf to klog.Errorf and include
namespace, dvName and the error to make the condition visible. Ensure you update
callers to handle the returned error if you choose to propagate.
hack/e2e-apps/vm-import.bats (2)

46-47: Replace fixed sleep with retry-based waiting.

The sleep 5 before kubectl wait is a fragile pattern. Consider using a retry loop or removing the sleep since kubectl wait already handles polling.

-  sleep 5
-  kubectl -n tenant-test wait hr vm-import-${name} --timeout=100s --for=condition=ready
+  # kubectl wait handles polling internally; increase timeout if needed
+  kubectl -n tenant-test wait hr vm-import-${name} --timeout=120s --for=condition=ready

Also applies to: 126-127, 179-180

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@hack/e2e-apps/vm-import.bats` around lines 46 - 47, Replace the fragile fixed
"sleep 5" before the kubectl wait with a retry-based check: instead of sleeping,
loop until the resource exists and is in a state where "kubectl -n tenant-test
wait hr vm-import-${name} --timeout=100s --for=condition=ready" will succeed (or
run kubectl wait in a retry loop with short backoff and a max attempts); update
the three occurrences (the "sleep 5" before the kubectl wait at the shown line
and the two other similar pairs around lines 126-127 and 179-180) to use this
retry logic so the script polls reliably rather than relying on a fixed delay.

86-91: Consider moving cleanup to framework-level teardown.

Inline cleanup at the end of tests won't execute if the test fails mid-execution, potentially leaving resources behind. Consider using Bats teardown or teardown_file functions to ensure cleanup runs regardless of test outcome.

# Example teardown approach
teardown() {
  # Clean up any VMImport resources created by tests
  kubectl -n tenant-test delete vmimport.apps.cozystack.io --all --ignore-not-found
  kubectl -n tenant-test delete secret -l app=vm-import-test --ignore-not-found
}

Based on learnings: "In E2E bats tests, replace inline cleanup with framework-level teardown/cleanup. Do not rely on traps or per-test cleanup."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@hack/e2e-apps/vm-import.bats` around lines 86 - 91, Move the inline cleanup
commands at the end of the test into a framework-level teardown (teardown or
teardown_file) so they always run even if the test fails; remove the inline
kubectl -n tenant-test delete vmimport.apps.cozystack.io ${name} and kubectl -n
tenant-test delete secret vmware-credentials-${name} lines from the test body
and instead implement teardown() that deletes VMImport resources and secrets
reliably (use kubectl -n tenant-test delete vmimport.apps.cozystack.io --all
--ignore-not-found or delete by label, and kubectl -n tenant-test delete secret
-l app=vm-import-test --ignore-not-found) and keep the wait/timeout logic in
teardown if needed to ensure resources are cleaned before exiting.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/apps/kafka/templates/kafka.yaml`:
- Around line 4-8: Remove the hardcoded "ZooKeeper" default when computing
$metadataState (use empty string via dig), then change the $isKRaft gate to true
when $metadataState is any of "KRaftPostMigration", "PreKRaft", or "KRaft"
(instead of only "KRaft"); additionally, when status is unset (empty
$metadataState) fallback to checking the resource annotation "strimzi.io/kraft"
for an "enabled" value to decide KRaft mode; apply the identical changes to the
duplicate logic in workloadmonitor.yaml so both templates follow Strimzi 0.45.1
migration semantics (use symbols $metadataState, $isExisting, $isKRaft, dig, and
$existingKafka to locate the code).

In `@packages/apps/vm-import/docs/scripts/adopt-vm.sh`:
- Line 181: The pipeline using DISKS_YAML with grep -c can produce a non-zero
exit status (grep returns 1 when no matches) which will abort the script under
set -e; update the substitution that prints "Disks: ..." so the grep failure
can't cause script exit (for example, append a harmless "|| true" to the grep
invocation or otherwise ensure the command substitution always returns a zero
exit status) so the output still shows the count (DISKS_YAML / grep -c "name:")
but the script won't be terminated by grep's non-zero exit code.

In `@packages/apps/vm-import/values.schema.json`:
- Around line 40-48: The schema currently leaves sourceUrl and sourceSecretName
defaulting to empty strings in the generated values.schema.json; update the
cozyvalues-gen input (the generator configuration/template that produces
values.schema.json) to mark "sourceUrl" and "sourceSecretName" as required and
remove or unset their "" defaults (so generated schema omits default or uses
no-empty validation), and apply the same change for the other occurrences noted
(the keys referenced around lines 99-102 in the generated file); regenerate via
make generate to verify the generated values.schema.json no longer exposes blank
defaults for these VMware connection settings.

In `@packages/apps/vm-instance/README.md`:
- Line 50: The docs use the literal string "vm-disk-<name>" which may be parsed
as an HTML tag; update the generator/source config used by cozyvalues-gen so the
placeholder is escaped or rendered as code (e.g., "vm-disk-&lt;name&gt;" or wrap
in backticks) in the generated README (change the template that emits the
disks[i].dvName description), then regenerate the docs so the README reflects
the escaped/code-formatted placeholder instead of a raw `<name>`.

In `@packages/system/vm-adoption-controller/images/controller/main.go`:
- Around line 540-544: The current pre-check using
c.dynamicClient.Resource(vmInstanceGVR).Namespace(vm.Namespace).Get(...) can
race with creation; update the VMInstance creation path (the call to
c.dynamicClient.Resource(vmInstanceGVR).Namespace(vm.Namespace).Create(...)) to
treat apierrors.IsAlreadyExists(err) as a success: when Create returns
AlreadyExists, log that the VMInstance was created concurrently and call
c.labelVMAsAdopted(ctx, vm.Namespace, vm.Name, vmInstanceName, helmReleaseName)
instead of treating it as a failure; also add the import for
"k8s.io/apimachinery/pkg/api/errors" (alias apierrors) so you can call
apierrors.IsAlreadyExists(err) in that create-error handling branch.

In `@packages/system/vm-adoption-controller/README.md`:
- Around line 39-40: Update the README example to match the chart default for
the watch interval: change the documented watchInterval value from 30 to 15 (the
key is watchInterval in the VM adoption controller config snippet) so the README
and chart defaults are consistent and avoid operator confusion.

In `@packages/system/vm-adoption-controller/templates/clusterrole.yaml`:
- Around line 17-20: The ClusterRole grants the "delete" verb on the
cluster-scoped resource vminstances under apiGroups "apps.cozystack.io"; remove
"delete" from the verbs array for the vminstances rule in
templates/clusterrole.yaml (the ClusterRole rule that lists apiGroups:
["apps.cozystack.io"] and resources: ["vminstances"]) so the verbs become
["get","list","watch","create","update","patch"] to enforce least-privilege for
the VM adoption controller.

---

Nitpick comments:
In `@hack/e2e-apps/kafka.bats`:
- Around line 3-40: The test "Create Kafka" contains an inline cleanup line
(kubectl -n tenant-test delete kafka.apps.cozystack.io $name) which should be
removed from the test body and instead registered in the framework/global
teardown used by other E2E tests (e.g., the pattern used in harbor.bats); update
the test to omit the delete, and add the Kafka resource cleanup (deleting kafka
$name in namespace tenant-test) to the shared teardown hook so cleanup runs even
on failures and reruns.

In `@hack/e2e-apps/vm-import.bats`:
- Around line 46-47: Replace the fragile fixed "sleep 5" before the kubectl wait
with a retry-based check: instead of sleeping, loop until the resource exists
and is in a state where "kubectl -n tenant-test wait hr vm-import-${name}
--timeout=100s --for=condition=ready" will succeed (or run kubectl wait in a
retry loop with short backoff and a max attempts); update the three occurrences
(the "sleep 5" before the kubectl wait at the shown line and the two other
similar pairs around lines 126-127 and 179-180) to use this retry logic so the
script polls reliably rather than relying on a fixed delay.
- Around line 86-91: Move the inline cleanup commands at the end of the test
into a framework-level teardown (teardown or teardown_file) so they always run
even if the test fails; remove the inline kubectl -n tenant-test delete
vmimport.apps.cozystack.io ${name} and kubectl -n tenant-test delete secret
vmware-credentials-${name} lines from the test body and instead implement
teardown() that deletes VMImport resources and secrets reliably (use kubectl -n
tenant-test delete vmimport.apps.cozystack.io --all --ignore-not-found or delete
by label, and kubectl -n tenant-test delete secret -l app=vm-import-test
--ignore-not-found) and keep the wait/timeout logic in teardown if needed to
ensure resources are cleaned before exiting.

In `@packages/apps/kafka/templates/kafkanodepools.yaml`:
- Around line 70-84: The controller node pool spec is missing the resources
field; add the same resources stanza used in the broker and combined pools by
invoking the cozy-lib helper (cozy-lib.resources.defaultingSanitize) inside the
controller pool spec (the block under "spec:" for the controller role) so
controllers include a resources section consistent with the other pools; ensure
you place the resources entry alongside replicas/roles/storage in the controller
pool definition and use the same template invocation and parameters as the
broker/combined pool.

In `@packages/apps/kafka/tests/metrics-configmap_test.yaml`:
- Around line 29-35: Add a new test case alongside the existing "it: does not
include zookeeper metrics for new instance" that simulates an existing ZooKeeper
metadata state and asserts the zookeeper metrics config is rendered;
specifically, create an "it" block describing migration with pre-existing ZK
metadata and include an asserts entry using "exists: path:
data[\"zookeeper-metrics-config.yml\"]" so the test expects that key to be
present during migration rendering (mirror the structure used by the existing
test but flip the notExists to exists to validate the migration path).

In `@packages/apps/vm-import/docs/examples/advanced-import.yaml`:
- Line 18: The thumbprint value in the advanced-import.yaml example is missing
one byte (SHA-1 should be 20 bytes / 40 hex chars); update the "thumbprint"
example to a full 20-byte SHA-1 formatted as 40 hexadecimal characters
(optionally colon-separated per the existing style) so the example is
accurate—locate the thumbprint field in the examples/advanced-import.yaml and
replace the truncated string with a complete 20-byte hex thumbprint.

In `@packages/apps/vm-import/docs/examples/monitoring-import.md`:
- Around line 168-169: The patch command shown uses the deprecated VM field
spec.running; update the example to use spec.runStrategy instead by changing the
patched payload to set spec.runStrategy (e.g., "RunStrategyAlways" or the
appropriate strategy) rather than spec.running, ensuring consistency with the
README example that already uses runStrategy and preserving the same resource
(vm nginx-frontend-01) and namespace (production).

In `@packages/apps/vm-import/docs/scripts/adopt-vm.sh`:
- Line 69: Replace the sed-based strip with bash parameter expansion: in the
assignment for DISK_REF (currently using echo "$disk" | sed "s/^vm-disk-//"),
use parameter expansion on the variable disk (remove the "vm-disk-" prefix via
${disk#vm-disk-}) so you avoid spawning a subprocess and satisfy SC2001; ensure
this is used wherever DISK_REF is computed from disk.
- Line 102: Replace the external sed call used to set SUBNET_NAME from the net
variable with bash parameter expansion to extract the substring after the last
'/' so it matches the pattern used earlier (same approach as on line 69); update
the assignment that sets SUBNET_NAME to use parameter expansion on net to avoid
spawning sed and keep consistent shell-builtins usage.

In `@packages/system/forklift-operator/Makefile`:
- Line 15: The current complex python3 -c one-liner that edits
'templates/forklift-operator.yaml' should be replaced with a simpler,
maintainable step: either call a short standalone script (e.g.,
scripts/remove_namespace_docs.py) or use yq to filter out documents with "kind:
Namespace", and update the Makefile to invoke that script/command instead of the
long inline python command; locate the existing python3 -c "import sys;
content=..." one-liner in the Makefile and swap it for a readable command that
clearly removes YAML documents where kind: Namespace from
templates/forklift-operator.yaml.

In `@packages/system/forklift/templates/forklift-cr.yaml`:
- Around line 3-5: The namespace in the custom resource is hardcoded to
"cozy-forklift" which prevents the CR from following the Helm release namespace;
update metadata.namespace to use the Helm release namespace (e.g., the release
namespace template variable) so the CR is deployed into the same namespace as
the release, or if the controller must always live in "cozy-forklift" add a
clear comment explaining that constraint; target the metadata.namespace field
and validate behavior via the ForkliftController CR (metadata.name:
forklift-controller) after change.

In `@packages/system/vm-adoption-controller/images/controller/Dockerfile`:
- Line 13: The Dockerfile currently hardcodes GOARCH=amd64 in the build step
(RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o controller main.go);
change this to use a build ARG (e.g., ARG TARGETARCH) and substitute GOARCH with
that ARG (or default to amd64) so the build supports multi-arch (arm64, amd64)
when invoked with buildx/--platform; update the Dockerfile to declare the ARG
and use it in the RUN that builds main.go to produce the controller binary.

In `@packages/system/vm-adoption-controller/images/controller/main.go`:
- Around line 119-127: The shutdown goroutine currently calls
server.Shutdown(context.Background()) which can hang; change it to create a
timeout context (e.g. ctxShutdown, cancel :=
context.WithTimeout(context.Background(), 5*time.Second)), call
server.Shutdown(ctxShutdown) and defer cancel() (or call cancel after Shutdown)
and log any Shutdown error; update the goroutine that references server and ctx
so it uses the new ctxShutdown and cancel to ensure the HTTP server shuts down
within the timeout.
- Around line 111-118: The health/readiness handlers registered via
mux.HandleFunc (the "/healthz" and "/readyz" functions) ignore the error
returned by w.Write; update both handlers to capture the return value (err :=
w.Write(...)) and handle it (e.g., log the error with log.Printf or klog.Errorf)
if non-nil so linters stop flagging the unchecked write error; do this in the
anonymous handler closures that call w.Write and keep the existing
w.WriteHeader(http.StatusOK) behavior.
- Around line 145-163: The Run method on AdoptionController currently polls with
a time.NewTicker (watchInterval) and calls c.reconcile(ctx), which causes higher
API load; refactor to use controller-runtime patterns: create a
SetupWithManager(ctrl.Manager) for AdoptionController, register
watches/informers for VirtualMachine resources and any related types, implement
Reconcile as the controller-runtime reconcile.Reconciler (move logic from
reconcile into Reconcile(ctx, req)), and use mgr.Start(ctx) (with manager
configured for leader election, rate-limiting and metrics) instead of the
ticker-driven AdoptionController.Run; ensure you remove or deprecate the polling
Run and wire the controller via ctrl.NewControllerManagedBy(mgr).For(...) so
events drive reconciliation.
- Around line 691-696: The loop that fetches DataVolumes (for _, dvName := range
dvNames) currently logs a warning and continues when
c.dynamicClient.Resource(dvGVR).Namespace(namespace).Get(...) returns an error,
which can hide missing expected resources; change this to fail-fast or at least
escalate the handling: either return/propagate an error from the enclosing
function when a DataVolume is missing (stop the Helm labeling and trigger
rollback) or collect missing dvNames and after the loop return a consolidated
error; if you must continue, change klog.Warningf to klog.Errorf and include
namespace, dvName and the error to make the condition visible. Ensure you update
callers to handle the returned error if you choose to propagate.

In `@packages/system/vm-instance-rd/cozyrds/vm-instance.yaml`:
- Line 11: The OpenAPI schema now includes fullnameOverride but the form
ordering (keysOrder) omits spec.fullnameOverride; add "spec.fullnameOverride"
into the keysOrder array where other spec fields (e.g., "spec.instanceProfile",
"spec.resources", "spec.runStrategy") are listed so the UI renders
fullnameOverride in a stable, deterministic position; touch the keysOrder entry
in the vm-instance.yaml schema and insert spec.fullnameOverride next to the
other spec.* keys.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b11569c5-21a4-48a3-8a38-6fd0fb5fc3ac

📥 Commits

Reviewing files that changed from the base of the PR and between 38624b7 and 52280af.

⛔ Files ignored due to path filters (2)
  • packages/apps/vm-import/logos/vm-import.svg is excluded by !**/*.svg
  • packages/system/vm-adoption-controller/images/controller/go.sum is excluded by !**/*.sum
📒 Files selected for processing (84)
  • hack/e2e-apps/kafka.bats
  • hack/e2e-apps/vm-import.bats
  • packages/apps/kafka/Chart.yaml
  • packages/apps/kafka/files/versions.yaml
  • packages/apps/kafka/templates/_versions.tpl
  • packages/apps/kafka/templates/dashboard-resourcemap.yaml
  • packages/apps/kafka/templates/kafka.yaml
  • packages/apps/kafka/templates/kafkanodepools.yaml
  • packages/apps/kafka/templates/metrics-configmap.yaml
  • packages/apps/kafka/templates/workloadmonitor.yaml
  • packages/apps/kafka/tests/dashboard-resourcemap_test.yaml
  • packages/apps/kafka/tests/kafka_test.yaml
  • packages/apps/kafka/tests/kafkanodepools_test.yaml
  • packages/apps/kafka/tests/metrics-configmap_test.yaml
  • packages/apps/kafka/tests/topics_test.yaml
  • packages/apps/kafka/tests/workloadmonitor_test.yaml
  • packages/apps/kafka/values.schema.json
  • packages/apps/kafka/values.yaml
  • packages/apps/vm-import/.helmignore
  • packages/apps/vm-import/ADOPTION_DESIGN.md
  • packages/apps/vm-import/Chart.yaml
  • packages/apps/vm-import/Makefile
  • packages/apps/vm-import/README.md
  • packages/apps/vm-import/charts/cozy-lib
  • packages/apps/vm-import/docs/adoption.md
  • packages/apps/vm-import/docs/examples/advanced-import.yaml
  • packages/apps/vm-import/docs/examples/monitoring-import.md
  • packages/apps/vm-import/docs/examples/simple-import.yaml
  • packages/apps/vm-import/docs/scripts/adopt-vm.sh
  • packages/apps/vm-import/templates/dashboard-resourcemap.yaml
  • packages/apps/vm-import/templates/migration.yaml
  • packages/apps/vm-import/templates/networkmap.yaml
  • packages/apps/vm-import/templates/plan.yaml
  • packages/apps/vm-import/templates/provider.yaml
  • packages/apps/vm-import/templates/storagemap.yaml
  • packages/apps/vm-import/templates/workloadmonitor.yaml
  • packages/apps/vm-import/values.schema.json
  • packages/apps/vm-import/values.yaml
  • packages/apps/vm-instance/Makefile
  • packages/apps/vm-instance/README.md
  • packages/apps/vm-instance/templates/vm.yaml
  • packages/apps/vm-instance/values.schema.json
  • packages/apps/vm-instance/values.yaml
  • packages/core/platform/sources/forklift.yaml
  • packages/core/platform/sources/vm-adoption-controller.yaml
  • packages/core/platform/sources/vm-import-application.yaml
  • packages/core/platform/templates/bundles/iaas.yaml
  • packages/system/forklift-operator/Chart.yaml
  • packages/system/forklift-operator/Makefile
  • packages/system/forklift-operator/templates/forklift-operator.yaml
  • packages/system/forklift/Chart.yaml
  • packages/system/forklift/Makefile
  • packages/system/forklift/templates/forklift-cr.yaml
  • packages/system/forklift/values.yaml
  • packages/system/kafka-operator/Makefile
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/Chart.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/README.md
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/040-Crd-kafka.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/041-Crd-kafkaconnect.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/045-Crd-kafkamirrormaker.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/046-Crd-kafkabridge.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/048-Crd-kafkamirrormaker2.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/04A-Crd-kafkanodepool.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/templates/_kafka_image_map.tpl
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/values.yaml
  • packages/system/kafka-rd/cozyrds/kafka.yaml
  • packages/system/vm-adoption-controller/Chart.yaml
  • packages/system/vm-adoption-controller/Makefile
  • packages/system/vm-adoption-controller/README.md
  • packages/system/vm-adoption-controller/images/controller/.gitignore
  • packages/system/vm-adoption-controller/images/controller/Dockerfile
  • packages/system/vm-adoption-controller/images/controller/go.mod
  • packages/system/vm-adoption-controller/images/controller/main.go
  • packages/system/vm-adoption-controller/templates/clusterrole.yaml
  • packages/system/vm-adoption-controller/templates/clusterrolebinding.yaml
  • packages/system/vm-adoption-controller/templates/deployment.yaml
  • packages/system/vm-adoption-controller/templates/serviceaccount.yaml
  • packages/system/vm-adoption-controller/values.yaml
  • packages/system/vm-import-rd/Chart.yaml
  • packages/system/vm-import-rd/Makefile
  • packages/system/vm-import-rd/cozyrds/vm-import.yaml
  • packages/system/vm-import-rd/templates/cozyrd.yaml
  • packages/system/vm-import-rd/values.yaml
  • packages/system/vm-instance-rd/cozyrds/vm-instance.yaml
💤 Files with no reviewable changes (5)
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/048-Crd-kafkamirrormaker2.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/045-Crd-kafkamirrormaker.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/046-Crd-kafkabridge.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/04A-Crd-kafkanodepool.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/040-Crd-kafka.yaml

Comment thread packages/apps/kafka/templates/kafka.yaml Outdated
echo -e "${GREEN}=== Adoption Summary ===${NC}"
echo "VM: $VM_NAME (namespace: $NAMESPACE)"
echo "Current state: $([ "$RUNNING" = "true" ] && echo "Running" || echo "Stopped")"
echo "Disks: $(echo "$DISKS_YAML" | grep -c "name:" || echo "0")"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

grep -c returns exit code 1 when no matches, which could exit the script.

With set -e enabled, grep -c "name:" will terminate the script if DISKS_YAML contains no matches. The || echo "0" fallback won't help because it applies to the entire pipeline substitution context.

🛡️ Suggested fix
-echo "Disks: $(echo "$DISKS_YAML" | grep -c "name:" || echo "0")"
+DISK_COUNT=$(echo "$DISKS_YAML" | grep -c "name:" || true)
+echo "Disks: ${DISK_COUNT:-0}"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
echo "Disks: $(echo "$DISKS_YAML" | grep -c "name:" || echo "0")"
DISK_COUNT=$(echo "$DISKS_YAML" | grep -c "name:" || true)
echo "Disks: ${DISK_COUNT:-0}"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/vm-import/docs/scripts/adopt-vm.sh` at line 181, The pipeline
using DISKS_YAML with grep -c can produce a non-zero exit status (grep returns 1
when no matches) which will abort the script under set -e; update the
substitution that prints "Disks: ..." so the grep failure can't cause script
exit (for example, append a harmless "|| true" to the grep invocation or
otherwise ensure the command substitution always returns a zero exit status) so
the output still shows the count (DISKS_YAML / grep -c "name:") but the script
won't be terminated by grep's non-zero exit code.

Comment on lines +40 to +48
"sourceSecretName": {
"description": "Name of the Kubernetes Secret containing VMware credentials (`user`, `password`, `thumbprint` keys).",
"type": "string",
"default": ""
},
"sourceUrl": {
"description": "URL of the VMware vCenter server (e.g. `https://vcenter.example.com/sdk`).",
"type": "string",
"default": ""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Blank VMware connection settings are still part of the default value surface.

sourceUrl and sourceSecretName are both documented as required, but their defaults are still "". That means the chart still presents an obviously unusable source endpoint/credential reference as part of its default values instead of catching it earlier. Please tighten this in the generator input rather than editing this generated file directly.

Based on learnings, values.schema.json is generated by cozyvalues-gen and manual edits are overwritten on make generate.

Also applies to: 99-102

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/vm-import/values.schema.json` around lines 40 - 48, The schema
currently leaves sourceUrl and sourceSecretName defaulting to empty strings in
the generated values.schema.json; update the cozyvalues-gen input (the generator
configuration/template that produces values.schema.json) to mark "sourceUrl" and
"sourceSecretName" as required and remove or unset their "" defaults (so
generated schema omits default or uses no-empty validation), and apply the same
change for the other occurrences noted (the keys referenced around lines 99-102
in the generated file); regenerate via make generate to verify the generated
values.schema.json no longer exposes blank defaults for these VMware connection
settings.

Comment thread packages/apps/vm-instance/README.md Outdated
| `networks[i].name` | Network attachment name. | `string` | `""` |
| `subnets` | Deprecated: use networks instead. | `[]object` | `[]` |
| `subnets[i].name` | Network attachment name. | `string` | `""` |
| `disks[i].dvName` | Explicit DataVolume name to reference. If set, used instead of the default vm-disk-<name> convention. | `string` | `""` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Escape the <name> placeholder in docs text.

Line 50 uses vm-disk-<name> as plain text; <name> can be treated as an HTML tag in Markdown and render incorrectly. Please encode or code-format it, then regenerate docs from the source config.

✏️ Proposed fix
-| `disks[i].dvName`   | Explicit DataVolume name to reference. If set, used instead of the default vm-disk-<name> convention.                             | `string`   | `""`        |
+| `disks[i].dvName`   | Explicit DataVolume name to reference. If set, used instead of the default `vm-disk-&lt;name&gt;` convention.                       | `string`   | `""`        |

Based on learnings: In readme files within packages that are auto-generated by cozyvalues-gen, manual changes are overwritten; update source/generator config and regenerate.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
| `disks[i].dvName` | Explicit DataVolume name to reference. If set, used instead of the default vm-disk-<name> convention. | `string` | `""` |
| `disks[i].dvName` | Explicit DataVolume name to reference. If set, used instead of the default `vm-disk-&lt;name&gt;` convention. | `string` | `""` |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/vm-instance/README.md` at line 50, The docs use the literal
string "vm-disk-<name>" which may be parsed as an HTML tag; update the
generator/source config used by cozyvalues-gen so the placeholder is escaped or
rendered as code (e.g., "vm-disk-&lt;name&gt;" or wrap in backticks) in the
generated README (change the template that emits the disks[i].dvName
description), then regenerate the docs so the README reflects the
escaped/code-formatted placeholder instead of a raw `<name>`.

Comment on lines +540 to +544
_, err = c.dynamicClient.Resource(vmInstanceGVR).Namespace(vm.Namespace).Get(ctx, vmInstanceName, metav1.GetOptions{})
if err == nil {
klog.Infof("VMInstance %s/%s already exists, ensuring VM is labeled", vm.Namespace, vmInstanceName)
return c.labelVMAsAdopted(ctx, vm.Namespace, vm.Name, vmInstanceName, helmReleaseName)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Potential race condition between VMInstance existence check and creation.

Between checking if VMInstance exists (line 540) and creating it (line 598), another controller replica or process could create the same VMInstance, leading to a conflict error.

Consider handling AlreadyExists error on creation as a success case rather than relying solely on the pre-check:

_, err = c.dynamicClient.Resource(vmInstanceGVR).Namespace(vm.Namespace).Create(ctx, vmInstance, metav1.CreateOptions{})
if err != nil {
    if apierrors.IsAlreadyExists(err) {
        klog.Infof("VMInstance %s/%s was created concurrently, ensuring VM is labeled", vm.Namespace, vmInstanceName)
        return c.labelVMAsAdopted(ctx, vm.Namespace, vm.Name, vmInstanceName, helmReleaseName)
    }
    // ... existing rollback logic
}

This requires importing k8s.io/apimachinery/pkg/api/errors.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/vm-adoption-controller/images/controller/main.go` around
lines 540 - 544, The current pre-check using
c.dynamicClient.Resource(vmInstanceGVR).Namespace(vm.Namespace).Get(...) can
race with creation; update the VMInstance creation path (the call to
c.dynamicClient.Resource(vmInstanceGVR).Namespace(vm.Namespace).Create(...)) to
treat apierrors.IsAlreadyExists(err) as a success: when Create returns
AlreadyExists, log that the VMInstance was created concurrently and call
c.labelVMAsAdopted(ctx, vm.Namespace, vm.Name, vmInstanceName, helmReleaseName)
instead of treating it as a failure; also add the import for
"k8s.io/apimachinery/pkg/api/errors" (alias apierrors) so you can call
apierrors.IsAlreadyExists(err) in that create-error handling branch.

Comment on lines +39 to +40
watchInterval: 30 # Check for new VMs every 30 seconds
watchNamespace: "" # Optional: watch specific namespace only (empty = all)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Update the documented default watch interval to match chart defaults.

README says 30s, but the chart default is 15s. This creates operator-facing confusion during debugging/tuning.

📝 Suggested doc fix
-  watchInterval: 30          # Check for new VMs every 30 seconds
+  watchInterval: 15          # Check for new VMs every 15 seconds
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
watchInterval: 30 # Check for new VMs every 30 seconds
watchNamespace: "" # Optional: watch specific namespace only (empty = all)
watchInterval: 15 # Check for new VMs every 15 seconds
watchNamespace: "" # Optional: watch specific namespace only (empty = all)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/vm-adoption-controller/README.md` around lines 39 - 40,
Update the README example to match the chart default for the watch interval:
change the documented watchInterval value from 30 to 15 (the key is
watchInterval in the VM adoption controller config snippet) so the README and
chart defaults are consistent and avoid operator confusion.

Comment on lines +17 to +20
- apiGroups: ["apps.cozystack.io"]
resources: ["vminstances"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Drop delete from vminstances unless a concrete delete path exists.

This is broader than necessary for adoption and weakens least-privilege RBAC for a cluster-scoped controller.

🔐 Suggested RBAC tightening
 - apiGroups: ["apps.cozystack.io"]
   resources: ["vminstances"]
-  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  verbs: ["get", "list", "watch", "create", "update", "patch"]
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- apiGroups: ["apps.cozystack.io"]
resources: ["vminstances"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps.cozystack.io"]
resources: ["vminstances"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/system/vm-adoption-controller/templates/clusterrole.yaml` around
lines 17 - 20, The ClusterRole grants the "delete" verb on the cluster-scoped
resource vminstances under apiGroups "apps.cozystack.io"; remove "delete" from
the verbs array for the vminstances rule in templates/clusterrole.yaml (the
ClusterRole rule that lists apiGroups: ["apps.cozystack.io"] and resources:
["vminstances"]) so the verbs become
["get","list","watch","create","update","patch"] to enforce least-privilege for
the VM adoption controller.

@dosubot dosubot Bot added size/XXL This PR changes 1000+ lines, ignoring generated files and removed size/XL This PR changes 500-999 lines, ignoring generated files labels Apr 7, 2026
@dosubot dosubot Bot added size/XL This PR changes 500-999 lines, ignoring generated files and removed size/XXL This PR changes 1000+ lines, ignoring generated files labels Apr 7, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
packages/apps/kafka/templates/kafka.yaml (1)

1-8: ⚠️ Potential issue | 🔴 Critical

Split the KRaft cutover flag from ZooKeeper cleanup.

strimzi.io/kraft must flip to enabled once Strimzi reaches KRaftPostMigration, while spec.zookeeper is only removed after ZooKeeper-related resources are gone. Reusing one $isKRaft flag for both behaviors means this code stalls the migration at migration, and simply broadening the condition would prune ZooKeeper too early. Also, defaulting an existing CR with missing status back to "ZooKeeper" can regress an already-enabled rollout, so the cutover decision needs a fallback to the current resource state when status.kafkaMetadataState is still unset. Please compute separate booleans here and reuse the same helper in the sibling templates. (strimzi.io)

Also applies to: 16-22, 69-88

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kafka/templates/kafka.yaml` around lines 1 - 8, The template
currently uses a single $isKRaft flag causing premature ZooKeeper removal and
regressions when status is unset; instead compute two booleans (e.g.
$shouldEnableKRaftFlag and $shouldRemoveZookeeperSpec) using the existing
lookup/dig results: use dig "status" "kafkaMetadataState" $existingKafka but
treat an empty status as “unknown” and fall back to the current resource
spec/state (inspect $existingKafka.spec or the existing label/annotation that
indicates KRaft) to decide the cutover flag, while only setting
$shouldRemoveZookeeperSpec when the status explicitly indicates ZooKeeper
resources are gone (e.g. "KRaftPostMigration" or equivalent); replace uses of
$isKRaft with these two booleans and extract this logic into a helper so sibling
templates can call the same decision logic.
🧹 Nitpick comments (1)
api/apps/v1alpha1/kafka/types.go (1)

97-98: Consider adding a drift guard between Go enum values and Helm version map.

To avoid future mismatch between type Version enum values and packages/apps/kafka/files/versions.yaml keys, add a small CI/unit check that compares both sets.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@api/apps/v1alpha1/kafka/types.go` around lines 97 - 98, Add a CI/unit test
that ensures the Go enum type Version in api/apps/v1alpha1/kafka/types.go stays
in sync with the YAML map of supported versions by parsing the YAML file that
lists version keys and comparing it to the set of defined Version constants;
implement a test named TestVersionsEnumMatchesYAML that programmatically
collects all constants of type Version (via reflection or by importing the
package and enumerating known consts) and loads the YAML version keys, then fail
the test if any key is missing from the enum or vice versa, with clear error
messages indicating which values are out of sync.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@packages/apps/kafka/templates/kafka.yaml`:
- Around line 1-8: The template currently uses a single $isKRaft flag causing
premature ZooKeeper removal and regressions when status is unset; instead
compute two booleans (e.g. $shouldEnableKRaftFlag and
$shouldRemoveZookeeperSpec) using the existing lookup/dig results: use dig
"status" "kafkaMetadataState" $existingKafka but treat an empty status as
“unknown” and fall back to the current resource spec/state (inspect
$existingKafka.spec or the existing label/annotation that indicates KRaft) to
decide the cutover flag, while only setting $shouldRemoveZookeeperSpec when the
status explicitly indicates ZooKeeper resources are gone (e.g.
"KRaftPostMigration" or equivalent); replace uses of $isKRaft with these two
booleans and extract this logic into a helper so sibling templates can call the
same decision logic.

---

Nitpick comments:
In `@api/apps/v1alpha1/kafka/types.go`:
- Around line 97-98: Add a CI/unit test that ensures the Go enum type Version in
api/apps/v1alpha1/kafka/types.go stays in sync with the YAML map of supported
versions by parsing the YAML file that lists version keys and comparing it to
the set of defined Version constants; implement a test named
TestVersionsEnumMatchesYAML that programmatically collects all constants of type
Version (via reflection or by importing the package and enumerating known
consts) and loads the YAML version keys, then fail the test if any key is
missing from the enum or vice versa, with clear error messages indicating which
values are out of sync.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2f677575-eb0e-4a74-a986-e95a63e9253e

📥 Commits

Reviewing files that changed from the base of the PR and between 19029df and 3ff482c.

📒 Files selected for processing (31)
  • api/apps/v1alpha1/kafka/types.go
  • hack/e2e-apps/kafka.bats
  • packages/apps/kafka/Chart.yaml
  • packages/apps/kafka/README.md
  • packages/apps/kafka/files/versions.yaml
  • packages/apps/kafka/templates/_versions.tpl
  • packages/apps/kafka/templates/dashboard-resourcemap.yaml
  • packages/apps/kafka/templates/kafka.yaml
  • packages/apps/kafka/templates/kafkanodepools.yaml
  • packages/apps/kafka/templates/metrics-configmap.yaml
  • packages/apps/kafka/templates/workloadmonitor.yaml
  • packages/apps/kafka/tests/dashboard-resourcemap_test.yaml
  • packages/apps/kafka/tests/kafka_test.yaml
  • packages/apps/kafka/tests/kafkanodepools_test.yaml
  • packages/apps/kafka/tests/metrics-configmap_test.yaml
  • packages/apps/kafka/tests/topics_test.yaml
  • packages/apps/kafka/tests/workloadmonitor_test.yaml
  • packages/apps/kafka/values.schema.json
  • packages/apps/kafka/values.yaml
  • packages/system/kafka-operator/Makefile
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/Chart.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/README.md
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/040-Crd-kafka.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/041-Crd-kafkaconnect.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/045-Crd-kafkamirrormaker.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/046-Crd-kafkabridge.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/048-Crd-kafkamirrormaker2.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/04A-Crd-kafkanodepool.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/templates/_kafka_image_map.tpl
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/values.yaml
  • packages/system/kafka-rd/cozyrds/kafka.yaml
💤 Files with no reviewable changes (5)
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/048-Crd-kafkamirrormaker2.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/045-Crd-kafkamirrormaker.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/04A-Crd-kafkanodepool.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/046-Crd-kafkabridge.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/040-Crd-kafka.yaml
✅ Files skipped from review due to trivial changes (14)
  • packages/system/kafka-operator/Makefile
  • packages/apps/kafka/files/versions.yaml
  • packages/apps/kafka/Chart.yaml
  • packages/apps/kafka/values.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/Chart.yaml
  • packages/apps/kafka/tests/metrics-configmap_test.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/README.md
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/values.yaml
  • packages/apps/kafka/tests/topics_test.yaml
  • packages/apps/kafka/values.schema.json
  • packages/apps/kafka/tests/dashboard-resourcemap_test.yaml
  • packages/apps/kafka/tests/workloadmonitor_test.yaml
  • packages/apps/kafka/tests/kafkanodepools_test.yaml
  • packages/apps/kafka/tests/kafka_test.yaml
🚧 Files skipped from review as they are similar to previous changes (5)
  • packages/apps/kafka/README.md
  • hack/e2e-apps/kafka.bats
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/templates/_kafka_image_map.tpl
  • packages/system/kafka-rd/cozyrds/kafka.yaml
  • packages/apps/kafka/templates/_versions.tpl

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

♻️ Duplicate comments (1)
packages/apps/kafka/templates/kafka.yaml (1)

1-8: ⚠️ Potential issue | 🔴 Critical

The KRaft cutover gate is still too narrow.

Still unresolved: Line 6 falls back to "ZooKeeper", and Line 8 only treats "KRaft" as the KRaft side of the migration. In Strimzi’s migration state machine that can keep strimzi.io/kraft: migration and the ZooKeeper blocks rendered past the cutover point, which risks stalling the final transition. Please treat KRaftPostMigration, PreKRaft, and KRaft as KRaft, and fall back to the current annotation when status is empty; the same duplicated gate exists in metrics-configmap.yaml, workloadmonitor.yaml, and dashboard-resourcemap.yaml.

Strimzi 0.45.x ZooKeeper to KRaft migration: when should the Kafka CR annotation `strimzi.io/kraft` switch from `migration` to `enabled`, and which `status.kafkaMetadataState` values should be treated as the KRaft side of the cutover?
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kafka/templates/kafka.yaml` around lines 1 - 8, The current
KRaft gate in kafka.yaml treats only "KRaft" as KRaft and falls back to
"ZooKeeper" when status is empty; update the logic that sets $metadataState and
computes $isKRaft so that: when $metadataState is empty you read the CR
annotation strimzi.io/kraft as the fallback, and treat any of "KRaft",
"KRaftPostMigration", or "PreKRaft" as KRaft; locate the lookup/dig usage that
assigns $metadataState and the $isKRaft definition and change the comparison to
include those three values (and use the annotation when status is empty); apply
the same fix to metrics-configmap.yaml, workloadmonitor.yaml, and
dashboard-resourcemap.yaml where the identical gate is duplicated.
🧹 Nitpick comments (1)
packages/apps/kafka/tests/kafkanodepools_test.yaml (1)

6-123: Cover the existing-broker branch before merge.

Every case here renders the empty-lookup path, so the split-pool migration branch in packages/apps/kafka/templates/kafkanodepools.yaml:38-85 still has no assertions around document count, controller replica quorum, or controller storage. This PR’s upgrade path depends on that branch; please add direct coverage, or extract the layout decision behind something the suite can drive.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kafka/tests/kafkanodepools_test.yaml` around lines 6 - 123, The
test suite only exercises the empty-lookup path and misses the existing-broker
(split-pool migration) branch in
packages/apps/kafka/templates/kafkanodepools.yaml (lines ~38-85); add explicit
test coverage in packages/apps/kafka/tests/kafkanodepools_test.yaml that drives
the template into the existing-broker branch (e.g., set a value like
kafka.existingBroker: true or kafka.splitPool: true) and assert the resulting
documents include the split-pool outputs: verify document count, presence/Kind
of the controller KafkaNodePool, controller replica quorum (spec.replicas for
controller), and controller storage settings (spec.storage.* for the controller
pool); alternatively, refactor the template decision into a tunable input (e.g.,
values.kafkanodepools.layout or values.kafka.splitPool) and add tests that set
that input to exercise and assert the split-pool branch.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@api/apps/v1alpha1/kafka/types.go`:
- Around line 24-26: The Version field on the Kafka CR is unconstrained so
invalid values slip into the CR; add enum validation for supported chart
versions by annotating the Version string (the Version field on the Kafka types
struct in types.go) with a kubebuilder validation enum (e.g. allowed values
"v3.8" and "v3.9") in the values-gen/api source so the generated CRD rejects
unsupported values, then regenerate the CRD/values (run the
values-gen/controller-gen step used by this repo) to apply the new enum
validation.

In `@packages/apps/kafka/templates/kafkanodepools.yaml`:
- Around line 70-84: Controller-only node pool is missing spec.resources,
causing controller pods to inherit no requests/limits; add the same resources
rendering used in the other pools into the controller-only pool block (the spec
under the controller-only pool that begins with "replicas: {{
$controllerReplicas }}" in kafkanodepools.yaml). Copy the existing
spec.resources Helm templating (the {{- with .Values.kafka.resources }}...{{-
end }} block used in the other node pools around lines 23/50) into this
controller pool so controllers get the same requests/limits and
limits/quota-safe behavior.

In
`@packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/041-Crd-kafkaconnect.yaml`:
- Line 2469: You edited the KafkaConnect CRD's description in the vendored chart
(the "KafkaConnect" CRD inside the strimzi-kafka-operator chart, e.g.
041-Crd-kafkaconnect.yaml) — revert that direct change and instead update the
chart via the chart vendoring workflow: pull or update the upstream
strimzi-kafka-operator chart (using your repo's chart vendoring tool or Helm
chart update process), apply the description fix in the upstream chart source,
run the vendoring command to regenerate the vendored charts, and commit the
regenerated vendor files so the change is tracked through the proper chart
update mechanism rather than editing the vendored CRD's description field
directly.

---

Duplicate comments:
In `@packages/apps/kafka/templates/kafka.yaml`:
- Around line 1-8: The current KRaft gate in kafka.yaml treats only "KRaft" as
KRaft and falls back to "ZooKeeper" when status is empty; update the logic that
sets $metadataState and computes $isKRaft so that: when $metadataState is empty
you read the CR annotation strimzi.io/kraft as the fallback, and treat any of
"KRaft", "KRaftPostMigration", or "PreKRaft" as KRaft; locate the lookup/dig
usage that assigns $metadataState and the $isKRaft definition and change the
comparison to include those three values (and use the annotation when status is
empty); apply the same fix to metrics-configmap.yaml, workloadmonitor.yaml, and
dashboard-resourcemap.yaml where the identical gate is duplicated.

---

Nitpick comments:
In `@packages/apps/kafka/tests/kafkanodepools_test.yaml`:
- Around line 6-123: The test suite only exercises the empty-lookup path and
misses the existing-broker (split-pool migration) branch in
packages/apps/kafka/templates/kafkanodepools.yaml (lines ~38-85); add explicit
test coverage in packages/apps/kafka/tests/kafkanodepools_test.yaml that drives
the template into the existing-broker branch (e.g., set a value like
kafka.existingBroker: true or kafka.splitPool: true) and assert the resulting
documents include the split-pool outputs: verify document count, presence/Kind
of the controller KafkaNodePool, controller replica quorum (spec.replicas for
controller), and controller storage settings (spec.storage.* for the controller
pool); alternatively, refactor the template decision into a tunable input (e.g.,
values.kafkanodepools.layout or values.kafka.splitPool) and add tests that set
that input to exercise and assert the split-pool branch.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d5e76cd9-a63e-438e-b58c-15b584b156d1

📥 Commits

Reviewing files that changed from the base of the PR and between 3ff482c and 2a6fbec.

📒 Files selected for processing (31)
  • api/apps/v1alpha1/kafka/types.go
  • hack/e2e-apps/kafka.bats
  • packages/apps/kafka/Chart.yaml
  • packages/apps/kafka/README.md
  • packages/apps/kafka/files/versions.yaml
  • packages/apps/kafka/templates/_versions.tpl
  • packages/apps/kafka/templates/dashboard-resourcemap.yaml
  • packages/apps/kafka/templates/kafka.yaml
  • packages/apps/kafka/templates/kafkanodepools.yaml
  • packages/apps/kafka/templates/metrics-configmap.yaml
  • packages/apps/kafka/templates/workloadmonitor.yaml
  • packages/apps/kafka/tests/dashboard-resourcemap_test.yaml
  • packages/apps/kafka/tests/kafka_test.yaml
  • packages/apps/kafka/tests/kafkanodepools_test.yaml
  • packages/apps/kafka/tests/metrics-configmap_test.yaml
  • packages/apps/kafka/tests/topics_test.yaml
  • packages/apps/kafka/tests/workloadmonitor_test.yaml
  • packages/apps/kafka/values.schema.json
  • packages/apps/kafka/values.yaml
  • packages/system/kafka-operator/Makefile
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/Chart.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/README.md
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/040-Crd-kafka.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/041-Crd-kafkaconnect.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/045-Crd-kafkamirrormaker.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/046-Crd-kafkabridge.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/048-Crd-kafkamirrormaker2.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/04A-Crd-kafkanodepool.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/templates/_kafka_image_map.tpl
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/values.yaml
  • packages/system/kafka-rd/cozyrds/kafka.yaml
💤 Files with no reviewable changes (5)
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/046-Crd-kafkabridge.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/048-Crd-kafkamirrormaker2.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/045-Crd-kafkamirrormaker.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/04A-Crd-kafkanodepool.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/040-Crd-kafka.yaml
✅ Files skipped from review due to trivial changes (12)
  • packages/apps/kafka/Chart.yaml
  • packages/apps/kafka/values.yaml
  • packages/apps/kafka/values.schema.json
  • packages/system/kafka-operator/Makefile
  • packages/apps/kafka/files/versions.yaml
  • packages/apps/kafka/templates/_versions.tpl
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/Chart.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/values.yaml
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/README.md
  • packages/apps/kafka/tests/topics_test.yaml
  • packages/apps/kafka/tests/dashboard-resourcemap_test.yaml
  • packages/apps/kafka/tests/workloadmonitor_test.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
  • packages/apps/kafka/README.md
  • packages/apps/kafka/tests/metrics-configmap_test.yaml
  • hack/e2e-apps/kafka.bats
  • packages/system/kafka-operator/charts/strimzi-kafka-operator/templates/_kafka_image_map.tpl
  • packages/system/kafka-rd/cozyrds/kafka.yaml
  • packages/apps/kafka/tests/kafka_test.yaml

Comment thread api/apps/v1alpha1/kafka/types.go Outdated
Comment thread packages/apps/kafka/templates/kafkanodepools.yaml Outdated
items:
type: string
description: "Configures additional options which will be passed to the Kaniko executor when building the new Connect image. Allowed options are: --customPlatform, --custom-platform, --insecure, --insecure-pull, --insecure-registry, --log-format, --log-timestamp, --registry-mirror, --reproducible, --single-snapshot, --skip-tls-verify, --skip-tls-verify-pull, --skip-tls-verify-registry, --verbosity, --snapshotMode, --use-new-run, --registry-certificate, --registry-client-cert, --ignore-path. These options will be used only on Kubernetes where the Kaniko executor is used. They will be ignored on OpenShift. The options are described in the link:https://github.com/GoogleContainerTools/kaniko[Kaniko GitHub repository^]. Changing this field does not trigger new build of the Kafka Connect image."
description: "Configures additional options which will be passed to the Kaniko executor when building the new Connect image. Allowed options are: --customPlatform, --custom-platform, --insecure, --insecure-pull, --insecure-registry, --log-format, --log-timestamp, --registry-mirror, --reproducible, --single-snapshot, --skip-tls-verify, --skip-tls-verify-pull, --skip-tls-verify-registry, --verbosity, --snapshotMode, --use-new-run, --registry-certificate, --registry-client-cert. These options will be used only on Kubernetes where the Kaniko executor is used. They will be ignored on OpenShift. The options are described in the link:https://github.com/GoogleContainerTools/kaniko[Kaniko GitHub repository^]. Changing this field does not trigger new build of the Kafka Connect image."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Avoid direct edits in vendored chart CRDs.

This change is in a vendored chart path (.../charts/strimzi-kafka-operator/...). Please apply it through the chart vendoring/update workflow instead of patching the vendored CRD directly, to prevent upstream drift.

As per coding guidelines: "Do NOT directly edit upstream Helm charts in vendored charts directory; use proper chart vendoring mechanisms".

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@packages/system/kafka-operator/charts/strimzi-kafka-operator/crds/041-Crd-kafkaconnect.yaml`
at line 2469, You edited the KafkaConnect CRD's description in the vendored
chart (the "KafkaConnect" CRD inside the strimzi-kafka-operator chart, e.g.
041-Crd-kafkaconnect.yaml) — revert that direct change and instead update the
chart via the chart vendoring workflow: pull or update the upstream
strimzi-kafka-operator chart (using your repo's chart vendoring tool or Helm
chart update process), apply the description fix in the upstream chart source,
run the vendoring command to regenerate the vendored charts, and commit the
regenerated vendor files so the change is tracked through the proper chart
update mechanism rather than editing the vendored CRD's description field
directly.

@myasnikovdaniil

myasnikovdaniil commented Apr 8, 2026 •

Copy link
Copy Markdown
Contributor

Hi! Matthieu ROBIN (@matthieu-robin) Thank you for the contribution. I found some issues that need to be fixed, here is the summary

Critical Issues

1. Combined KafkaNodePool cannot be used during migration

The PR creates a combined pool (broker+controller) for new instances and separate pools for existing (migrating) instances. However, Strimzi docs explicitly state that migration from ZooKeeper to KRaft in "combined" mode is not supported. You must use dedicated/separate controller and broker pools during migration.

The PR's detection logic is:

{{- $existingBrokerPool := lookup "kafka.strimzi.io/v1beta2" "KafkaNodePool" .Release.Namespace (printf "%s-broker" .Release.Name) }}
{{- $useCombinedPool := empty $existingBrokerPool }}

Since existing ZK-based clusters have no KafkaNodePool resources at all (pools are a KRaft/node-pool-era concept), $existingBrokerPool is always empty for them. This means $useCombinedPool = true, and existing clusters also get the combined pool — which Strimzi does not support for migration. Migration will fail.

Source: https://strimzi.io/docs/operators/0.45.1/deploying#proc-deploy-migrate-kraft-str

2. Deadlock in migration state logic

The template computes:

{{- $isKRaft := or (not $isExisting) (eq $metadataState "KRaft") }}

This evaluates to true only when:

  • The cluster doesn't exist yet (new deploy), OR
  • The cluster has fully completed migration (kafkaMetadataState == "KRaft")

The $isKRaft variable controls whether ZooKeeper resources are rendered (zookeeper spec, zookeeper metrics, zookeeper workloadmonitor). During migration, the states progress:

ZooKeeper → KRaftMigration → KRaftDualWriting → KRaftPostMigration → PreKRaft → KRaft

The problem: To finalize migration and reach KRaft state, the strimzi.io/kraft annotation must be changed from migration to enabled. But the template sets this annotation based on $isKRaft:

{{- if $isKRaft }}
strimzi.io/kraft: enabled
{{- else }}
strimzi.io/kraft: migration
{{- end }}

So the annotation stays migration for ALL intermediate states (KRaftMigration, KRaftDualWriting, KRaftPostMigration, PreKRaft) because none of them satisfy eq $metadataState "KRaft". The operator will reach KRaftPostMigration and stop there — it cannot finalize to KRaft because the annotation never flips to enabled.

Even if someone manually triggers another Helm upgrade, the lookup will see KRaftPostMigration (not KRaft), so $isKRaft stays false, and the annotation stays migration. The migration can never complete automatically.

3. Migration runs automatically without user control

The current design triggers migration automatically when the chart is upgraded (via Flux). The flow would be:

  1. Cozystack updates, Flux reconciles the HelmRelease
  2. lookup finds existing Kafka CR in ZooKeeper state
  3. Template sets strimzi.io/kraft: migration
  4. Strimzi operator begins migration automatically

Problems:

  • Users have no visibility into or control over when migration starts
  • No one monitors the migration progress
  • Even if the deadlock in issue 2 were fixed, subsequent state transitions require another Helm upgrade, but nothing triggers one — lookup values are only evaluated during install/upgrade, not continuously
  • There's no rollback mechanism exposed to users (Strimzi supports strimzi.io/kraft: rollback before KRaftPostMigration)
  • This is a significant infrastructure change that should require explicit user action

Architectural Issues

4. Annotation-driven migration requires manual state transitions

Per Strimzi docs, the operator is semi-automated. It does NOT auto-advance through all states. The strimzi.io/kraft annotation must be changed by the user at key transition points:

Annotation value Meaning
disabled ZooKeeper-based cluster
migration Initiates migration, keeps rollback open
enabled Finalizes migration to KRaft
rollback Reverts to ZooKeeper (only before KRaftPostMigration)

The template only supports migration and enabled, missing disabled and rollback.

5. Repeated lookup logic across templates

The same state detection block is copy-pasted across 4 templates:

  • kafka.yaml
  • metrics-configmap.yaml
  • workloadmonitor.yaml
  • dashboard-resourcemap.yaml

Should be extracted to a shared named template.

6. ZooKeeper still in values/schema but invisible to new users

values.yaml, values.schema.json, and kafka-rd/cozyrds/kafka.yaml still expose the full zookeeper: configuration section. New KRaft instances don't use these values. This is confusing.

Minor Issues

7. No enum constraint on version field

values.schema.json defines version as free-form string. Should have "enum": ["v3.9", "v3.8"] for dashboard dropdown and early validation.

8. Controller pool storage hardcoded to 5Gi

kafkanodepools.yaml hardcodes size: 5Gi for the separate controller pool with no user-facing configuration.

9. deleteClaim: true on controller pool

Controller nodes hold KRaft metadata. deleteClaim: true means scaling down destroys this data. The old ZK PVCs used deleteClaim: false. Controller PVCs should probably also use false.

10. keys output in error message is unordered

_versions.tpl: (keys $versionMap) returns keys in random order. Use sortAlpha:

{{- printf `...allowed versions are %s` (keys $versionMap | sortAlpha) | fail }}

Proposed Direction

Instead of automatic migration, the kafka chart should support migration as an explicit, user-driven process:

  1. New instances: Deploy directly in KRaft mode with combined pool (current behavior is correct for this case)
  2. Existing instances: Continue running in ZooKeeper mode after chart upgrade — no automatic migration
  3. User-initiated migration: User explicitly opts in by setting a value (e.g., migration.enabled: true or mode: kraft) in their Kafka app configuration in Cozystack
  4. Migration finalization: User explicitly finalizes by changing a value after verifying migration health
  5. Rollback support: User can request rollback before finalization
  6. Fresh redeploy: Users can also choose to delete and recreate their Kafka in KRaft mode from scratch

This approach:

  • Gives users full control over a risky infrastructure change
  • Makes migration observable and reversible
  • Avoids the lookup-based deadlock
  • Can be documented as a step-by-step upgrade guide
  • Aligns with Strimzi's own semi-manual migration design

I think proposal needs to be discussed with codeowners for clarification.

@dosubot dosubot Bot added size/XXL This PR changes 1000+ lines, ignoring generated files and removed size/XL This PR changes 500-999 lines, ignoring generated files labels Apr 8, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (1)
packages/apps/kafka/templates/kafkanodepools.yaml (1)

70-74: ⚠️ Potential issue | 🟠 Major

Controller pool still misses resource requests/limits.

The controller-only KafkaNodePool defines replicas and roles but no spec.resources, unlike the other pools. This can cause quota/LimitRange failures and unstable scheduling for the control quorum.

Minimal fix
 spec:
   replicas: {{ $controllerReplicas }}
   roles:
     - controller
+  resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 4 }}
   storage:
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kafka/templates/kafkanodepools.yaml` around lines 70 - 74, The
controller-only KafkaNodePool is missing a spec.resources block; add a
spec.resources stanza to the controller KafkaNodePool (the same block placement
as in the other node pools) containing requests and limits for CPU and memory
(and any other cluster-required resource fields) so the controller replicas have
guaranteed scheduling and respect quota/LimitRange; copy the resources values
used by the other pools or set controller-appropriate values and ensure units
and keys match (spec.resources.requests and spec.resources.limits).
🧹 Nitpick comments (1)
packages/apps/kafka/Makefile (1)

4-6: LGTM! The test target integrates correctly with the CI discovery script.

The target name matches what hack/helm-unit-tests.sh expects, and since package.mk doesn't define a test target, there's no override conflict.

Minor optional suggestion: consider declaring test as .PHONY to ensure it always runs even if a file or directory named test is ever created:

.PHONY: test
test:
	helm unittest .
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/apps/kafka/Makefile` around lines 4 - 6, Add a .PHONY declaration
for the Makefile target named test so the target always runs even if a file or
directory called test exists; modify the Makefile to include a line declaring
.PHONY: test followed by the existing test target (the target named "test" that
calls "helm unittest .") so CI discovery behavior is unchanged but the target
cannot be shadowed by a filesystem entry.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/apps/kafka/templates/kafkanodepools.yaml`:
- Around line 83-84: The KRaft controller metadata volume is configured with
deleteClaim: true which risks deleting controller metadata; locate the KRaft
controller volume block (the entry with kraftMetadata: shared) and change
deleteClaim from true to false so the PVC is retained across node pool/cluster
lifecycle operations; verify the final YAML leaves deleteClaim unset or
explicitly set to false to preserve recoverability.
- Around line 2-3: The current logic uses only $existingBrokerPool/
$useCombinedPool to decide combined vs separate pools which misroutes clusters
undergoing ZK→KRaft migration; update the selection to also inspect the Kafka
resource's migration annotation or status (e.g., check .spec or
.metadata.annotations on the Kafka CR for Strimzi migration flags) and only
choose the combined pool when there is no existing broker pool AND the Kafka
resource indicates "not migrating"; change the condition that builds
$useCombinedPool accordingly (referencing $existingBrokerPool, $useCombinedPool,
and the Kafka CR lookup), and in the separate controller pool block (the block
that sets roles: [controller]) add the same resources block present in the
combined/broker pool and remove or set deleteClaim to false to avoid metadata
loss during controller lifecycle operations.

---

Duplicate comments:
In `@packages/apps/kafka/templates/kafkanodepools.yaml`:
- Around line 70-74: The controller-only KafkaNodePool is missing a
spec.resources block; add a spec.resources stanza to the controller
KafkaNodePool (the same block placement as in the other node pools) containing
requests and limits for CPU and memory (and any other cluster-required resource
fields) so the controller replicas have guaranteed scheduling and respect
quota/LimitRange; copy the resources values used by the other pools or set
controller-appropriate values and ensure units and keys match
(spec.resources.requests and spec.resources.limits).

---

Nitpick comments:
In `@packages/apps/kafka/Makefile`:
- Around line 4-6: Add a .PHONY declaration for the Makefile target named test
so the target always runs even if a file or directory called test exists; modify
the Makefile to include a line declaring .PHONY: test followed by the existing
test target (the target named "test" that calls "helm unittest .") so CI
discovery behavior is unchanged but the target cannot be shadowed by a
filesystem entry.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 13c7fd4c-3278-4731-97c3-fc1b227a759e

📥 Commits

Reviewing files that changed from the base of the PR and between 2a6fbec and e5b0be2.

📒 Files selected for processing (9)
  • api/apps/v1alpha1/kafka/types.go
  • packages/apps/kafka/Makefile
  • packages/apps/kafka/README.md
  • packages/apps/kafka/templates/kafkanodepools.yaml
  • packages/apps/kafka/tests/kafkanodepools_test.yaml
  • packages/apps/kafka/tests/workloadmonitor_test.yaml
  • packages/apps/kafka/values.schema.json
  • packages/apps/kafka/values.yaml
  • packages/system/kafka-rd/cozyrds/kafka.yaml
🚧 Files skipped from review as they are similar to previous changes (5)
  • packages/apps/kafka/README.md
  • api/apps/v1alpha1/kafka/types.go
  • packages/apps/kafka/tests/kafkanodepools_test.yaml
  • packages/system/kafka-rd/cozyrds/kafka.yaml
  • packages/apps/kafka/tests/workloadmonitor_test.yaml

Comment thread packages/apps/kafka/templates/kafkanodepools.yaml Outdated
Comment thread packages/apps/kafka/templates/kafkanodepools.yaml Outdated
@scooby87
scooby87 force-pushed the KRaft branch 2 times, most recently from 99184fb to 2cc05e7 Compare September 15, 2026 11:17
Matthieu and others added 15 commits September 17, 2026 11:09
…ation

Replace ZooKeeper-based Kafka with KRaft (Kafka Raft) metadata mode.
Strimzi 0.45 is the last line that still supports ZooKeeper, so this
prepares the transition before ZooKeeper support is removed upstream.

Chart changes:
- The Kafka CR always renders strimzi.io/kraft=enabled with node pools
  and no spec.zookeeper block. Storage, replicas and resources move to
  separate broker + controller KafkaNodePools (a homogeneous separate-pool
  layout works for both fresh installs and migrations; Strimzi 0.45 does
  not support migration into a combined pool).
- Kafka version is now user-configurable via the standard files/versions.yaml
  pattern (v3.9 default, v3.8 available); appVersion bumped to 3.9.0.
- WorkloadMonitors track the broker and controller pools (no zookeeper
  monitor); the dashboard RBAC grants the controller monitor and drops the
  zookeeper one.

Migration of existing ZooKeeper clusters is handled by a Helm pre-upgrade
Job (templates/migration-hook.yaml) gated by a <release>-kafka-deployed-version
ConfigMap, following the established seaweedfs/etcd hook pattern. The Job
creates the broker + controller pools with Helm ownership labels, drives the
Strimzi ZK->KRaft state machine (node-pools=enabled, kraft=migration ->
kraft=enabled once KRaftPostMigration is reached) and is skipped once the
ConfigMap records version >= "1".

Testing:
- Helm unit tests cover the Kafka CR, node pools, migration hook, metrics,
  topics, version map, dashboard RBAC and workload monitors.
- The e2e suite is ported to the current Chainsaw framework
  (hack/e2e-chainsaw/kafka): it asserts the KRaft topology (both node pools,
  kafkaMetadataState=KRaft), the absence of ZooKeeper, and bootstrap
  endpoints.

Rebased onto current main: TLS listener support, entity-operator resource
guards, x-cozystack-options schema annotations and the WorkloadMonitor
replica-path fix are all preserved and reconciled with KRaft.

Signed-off-by: Matthieu <[email protected]>
Signed-off-by: Alexey Artamonov <[email protected]>
…aft finalize

Follow-up to the KRaft migration addressing the branch review.

- kafka-rd cozyrds: restore the two public CA certs in secrets.include
  (clients-ca-cert + cluster-ca-cert). The rebase had collapsed them to
  the bare clients-ca, which is Strimzi's clients-CA *private key* secret
  — a private-key exposure to the tenant and a TLS regression (clients
  need cluster-ca-cert to trust the broker listener). Covered by the
  existing hack/check-kafka-rd-secrets.bats.

- migration-hook: guard the KRaft finalize step. The wait loop only exited
  via break on success and fell through to unconditionally apply
  strimzi.io/kraft=enabled on timeout — unsafe and irreversible if the
  cluster is still in ZooKeeper/KRaftMigration. Add a migration_ready flag
  and exit 1 when the safe state (KRaftPostMigration+) is not reached,
  leaving the deployed-version ConfigMap unstamped so Helm retries. New
  matchRegex assertions in migration-hook_test.yaml cover the guard.

- migration-hook: raise activeDeadlineSeconds 1800 -> 3600 so a
  legitimately-progressing migration is not SIGKILLed before the in-script
  waits and error paths can run.

- Extract the duplicated controller-quorum logic into a shared
  kafka.controllerReplicas helper (used by kafkanodepools, workloadmonitor
  and migration-hook) so the three stay in lockstep.

- Reflow the README migration section to one line per paragraph per the
  repo prose rule.

Signed-off-by: Alexey Artamonov <[email protected]>
…ok safety, tests

Addresses the branch review on the ZK->KRaft migration path.

CRITICAL — broker node pool adoption. Strimzi derives broker/PVC names as
<cluster>-<pool>-<id>; a pre-node-pool ZooKeeper cluster has brokers
<cluster>-kafka-N, so the adopting broker pool must be named exactly "kafka"
or the operator provisions fresh empty brokers and orphans the tenant's data.
A new kafka.brokerPoolName helper resolves the broker pool to "kafka" for an
existing (migrating) cluster and keeps the release-scoped "<release>-broker"
for fresh installs, so multiple Kafka instances still coexist in one namespace.
The migration hook creates the pool as "kafka" and fails closed if a "kafka"
pool already belongs to another cluster (node pool names are namespace-unique,
so in-place migration of co-namespaced ZK clusters is unsupported). Wired
through kafkanodepools.yaml, workloadmonitor.yaml and the hook.

MAJOR — migration robustness.
- Raise the HelmRelease timeout: the migration Job can run tens of minutes but
  the release timeout defaults to 10m, so helm-controller aborted and thrashed
  the pre-upgrade hook. Add release.cozystack.io/helm-install-timeout=45m on the
  kafka ApplicationDefinition, above the Job's activeDeadlineSeconds (2400s).
- Idempotent migration: do not re-issue kraft=migration once the cluster is past
  the ZooKeeper phase (an invalid backward transition Strimzi rejects on retry).
- Fail closed on the decision gates: distinguish a genuine NotFound (skip) from
  any other kubectl error (abort), and never treat an empty kafkaMetadataState on
  an existing CR as "migrated".

MINOR — version downgrade documented as unsupported; integer (not lexical)
compare in the version gate; deleteClaim/PVC-retention and the one-per-namespace
migration limitation documented in the README; docs/storage-immutability.md
updated for KRaft.

Tests — helm-unittest coverage for the broker pool name, the hook guards and the
idempotent/fail-closed logic; a bats check for the helm-install-timeout budget
ordering; and a Chainsaw ZK->KRaft migration suite that seeds a ZooKeeper
cluster, drives the migration, and asserts the original broker PVC survives (not
recreated) and the cluster reaches KRaft. The migration sequence and the
Chainsaw assertions were validated against a live cluster.

Signed-off-by: Alexey Artamonov <[email protected]>
Follow-up to the KRaft-migration review. The broker KafkaNodePool must be named
exactly "kafka" so Strimzi adopts an existing ZooKeeper cluster's
<cluster>-kafka-N brokers and data in place during migration, but KafkaNodePool
object names are unique within a namespace — so two Kafka clusters cannot both
own a "kafka" pool there. This is an upstream constraint and Strimzi's own
guidance is one Kafka cluster per namespace
(strimzi/strimzi-kafka-operator discussions/11120).

Adopt that model instead of the earlier fresh-vs-migrated conditional naming:

- New kafka.assertSingleInstance helper fails the render (with a clear message
  and the upstream reference) if the namespace already holds another Kafka
  cluster. Reconciles/upgrades of the same release are allowed; lookup returns
  nothing during `helm template`, so unit tests never trip it.
- Node pools are now named "kafka" (broker) and "controller" unconditionally,
  which also adopts existing brokers on migration and yields clean
  <cluster>-kafka-N / <cluster>-controller-N names. The fresh-vs-migrated
  brokerPoolName helper is removed.
- WorkloadMonitor pool-name selectors and the migration hook updated to match;
  unit tests updated for the fixed pool names.
- README documents the one-Kafka-per-namespace rule and links the Strimzi
  discussion.

The ZK->KRaft Chainsaw migration suite (seed ZooKeeper cluster, migrate, assert
the original broker PVC survives and no wrongly-named PVC is created, cluster
reaches KRaft) was run green against a live cluster.

Signed-off-by: Alexey Artamonov <[email protected]>
The ZK->KRaft pre-upgrade hook created the adopting broker ("kafka") and
"controller" KafkaNodePools without a resources block. Those hook-created pools
govern the Kafka pods for the entire migration window — Helm applies the
resourced steady-state kafkanodepools.yaml only after the pre-upgrade hook
finishes. In a tenant namespace the LimitRange defaults every container to
128Mi, so the migration's controller (and rolled broker) JVMs OOM, never reach
KRaftPostMigration, and the hook times out — wedging the migration and thrashing
it on every reconcile.

Give both hook pools the same resources the steady-state pools get
(cozy-lib.resources.defaultingSanitize from kafka.resourcesPreset/resources).
This also avoids a redundant second broker roll (resource-less -> resourced)
after the hook completes.

Tests: a helm-unittest assertion that both migration pools render a resources
block; the Chainsaw ZK->KRaft suite now seeds under a 128Mi LimitRange (and
gives the seed cluster explicit resources), reproducing the tenant-quota
condition, and asserts the migration still reaches KRaft with the broker PVC
adopted. Validated green against a live cluster under the LimitRange.

Signed-off-by: Alexey Artamonov <[email protected]>
Addresses the second review round on the ZK->KRaft migration.

Migration hook:
- Fully qualify the Strimzi resources as kafkas.kafka.strimzi.io /
  kafkanodepools.kafka.strimzi.io. The bare plural "kafkas" also exists in this
  chart's apps.cozystack.io group, so a bare name resolves by per-cluster
  discovery priority and can hit the tenant kind — which the hook Role does not
  grant, failing the pre-upgrade hook deterministically.
- Classify NotFound structurally via `kubectl get --ignore-not-found` instead of
  substring-matching stderr, so an unrelated "... not found" error can no longer
  be mistaken for a fresh install and permanently skip a live ZooKeeper cluster.
  This also drops the jq dependency.
- Treat a KRaft-native / already-migrated CR (annotated strimzi.io/kraft=enabled)
  as done, never driving it backwards into kraft=migration.
- Keep only `before-hook-creation` on the Job's hook-delete-policy so a failed
  attempt's pod logs survive for a post-mortem until the next run.
- Route the kubectl image through cozy-lib.image (cluster registry mirror) and
  pin it by digest (clastix/kubectl:v1.32@sha256:...), matching the other
  cozystack hooks, instead of an unpinned docker.io tag.

Controller quorum:
- Fix the controller node-pool replica count at creation instead of deriving it
  from kafka.replicas on every reconcile. Strimzi 0.45 cannot scale a controller
  pool, so a day-2 edit of kafka.replicas (e.g. 1 -> 3) must not flip the quorum;
  the helper now keeps the existing "controller" pool's replica count when it is
  present.

Docs:
- Document that migration to KRaft is one-way: neither a version-field downgrade
  nor a pre-KRaft chart rollback is supported once migrated (Strimzi has no
  KRaft->ZooKeeper rollback after completion).

E2E:
- Run the ZK->KRaft chainsaw migration in its own namespace instead of the
  shared tenant-test. It raw-applies Strimzi resources plus a namespace-wide
  LimitRange and leaves deleteClaim:false PVCs, which polluted the sibling kafka
  suite. Raise the seed readiness timeout for CI headroom. Validated against a
  live cluster: migration adopts the broker PVC in place and reaches KRaft under
  a 128Mi LimitRange, in isolation.

Signed-off-by: Alexey Artamonov <[email protected]>
The strimzi-kafka Grafana dashboard still had a 'ZK request latency' panel
(kafka_server_zookeeperclientmetrics_zookeeperrequestlatencyms), which goes
permanently empty after the ZooKeeper->KRaft migration. Replace it with a
KRaft controller commit-latency panel (kafka_server_raftmetrics_commit_latency_avg,
scraped from the controller pods). Metric names verified against a live KRaft
cluster. The PVC-exclusion filter that references zookeeper stays valid and is
left as is.

Signed-off-by: Alexey Artamonov <[email protected]>
The removed bats file grepped two static literals in two separate
packages (helm-install-timeout in system/kafka-rd, activeDeadlineSeconds
in apps/kafka) and asserted an ordering between them. Substring matching
cannot verify the contract it targets: the real invariant is between the
running migration Job and the release timeout helm-controller actually
enforces on the pre-upgrade hook, not between two YAML literals in the
tree. It also greened only for the minute-format literal and errored on
any legitimate reformat (e.g. "1h" / "2700s").

The timeout > Job-deadline relationship is already documented inline at
both values. Its machine-checkable home is an e2e that upgrades an
existing release through the HelmRelease pre-upgrade hook and confirms a
real migration completes inside the release timeout (the chainsaw suite
drives migration via kubectl, bypassing that path).

Signed-off-by: Andrey Kolkov <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Fresh KRaft clusters no longer need to be alone in a namespace. Their
node pools get short, release-hashed names — "b-<hash>" for brokers,
"c-<hash>" for controllers — instead of the fixed "kafka"/"controller".
Each release hashes differently, so the names are unique within the
namespace (KafkaNodePool object names are namespace-unique) and any
number of Kafka instances coexist. The hash keeps the pool name a fixed
length: Strimzi derives every pod/PVC name as <cluster>-<pool>-<id> and
the cluster name is already the release name, so embedding the release
name in the pool name too would double it into every pod hostname and
blow the 63-character DNS label limit for long release names. A
render-time guard fails with a clear message when even the hashed
controller node name would still exceed 63 characters.

The fixed "kafka" name is still required to adopt an existing ZooKeeper
cluster's <cluster>-kafka-N brokers and data in place during migration,
so kafka.brokerPoolName resolves to "kafka" for a migrating/migrated
cluster and "b-<hash>" for a fresh one. A second ZooKeeper->KRaft
migration into a namespace that already holds a "kafka" pool owned by
another cluster fails closed, both at render time and in the pre-upgrade
hook (whose owner probe uses --ignore-not-found + an explicit exit-code
check, so a transient kubectl error aborts rather than silently waiving
the collision guard). This replaces the previous blanket "one Kafka per
namespace" guard, which over-restricted the fresh case.

The pool names are resolved from a live Helm lookup; that is safe because
Helm propagates any non-NotFound read error and fails the whole render
(pkg/engine/lookup_func.go), so a transient apiserver/RBAC miss retries
rather than silently flipping a live cluster's pool name — an empty
lookup means genuine absence or a never-applied client-side render.

The migration hook and workload monitors follow the same naming. Tests:
helm-unittest fixtures (via kubernetesProvider) cover every resolution
branch and the fail-closed guards; hack/kafka-migration-hook.bats drives
the migration Job's real shell script against a stub kubectl across the
fresh / read-error / already-KRaft / owner-collision / happy-path /
stuck / retry cases. Verified on a live cluster: in-place ZooKeeper->KRaft
migration preserved all data and adopted the original broker PVCs, and
two fresh clusters coexist with distinct pools, pods and PVCs.

Signed-off-by: Alexey Artamonov <[email protected]>
The controller commit-latency panel selected controller pods with
pod=~"$strimzi_cluster_name-controller-.*", but the controller node pool
is named c-<hash> (kafka.controllerPoolName), so Strimzi names controller
pods <cluster>-c-<hash>-<id>. The literal -controller- never matched, so
the panel was permanently empty. Select the pool by its -c- token instead.

Signed-off-by: Andrey Kolkov <[email protected]>
Two live-reproduced blockers in the in-place ZK->KRaft migration are
fixed, plus the concurrency guard behind them.

1. Data loss (node IDs). The migration created the broker and controller
   KafkaNodePools together without pinning node IDs. On Strimzi 0.45 the
   controller pool deterministically took the existing broker IDs
   [0..N-1], so the brokers were recreated empty at [N..] and their data
   PVCs were orphaned (reproduced 2/2 on dev9: 20000 messages -> 0,
   partitions leaderless, migration wedged in KRaftMigration). Naming the
   broker pool "kafka" is necessary but NOT sufficient: Strimzi adopts by
   node id, not pool name. The controller pool now pins
   strimzi.io/next-node-ids above the broker range, so the broker pool
   keeps [0..N-1] and adopts the existing <cluster>-kafka-N brokers and
   their data in place.

2. Release stuck "failed" (Helm lifecycle). The pre-upgrade hook creates
   the pools, and the chart also declared them as regular manifests, so
   Helm's main apply tried to reconcile a resource that exists in the
   cluster but was absent from the previous (ZooKeeper) release revision
   and aborted with "original object KafkaNodePool ... not found",
   leaving the HelmRelease failed even though the cluster migrated fine
   (reproduced 3/3 under server-side apply, the production
   helm-controller path). This is unavoidable while the pools are Helm
   main-manifest resources across the ZK->KRaft boundary, so the pools
   are now managed entirely by the migration Job (a pre-install,
   pre-upgrade hook) on every install and upgrade, and removed from the
   regular manifests. The Job ensures both pools (creating or updating
   them idempotently), then drives the migration state machine.
   templates/kafkanodepools.yaml and the deployed-version gate ConfigMap
   (templates/version.yaml) are removed; the broker pool name is resolved
   per path by kafka.brokerPoolName ("b-<hash>" fresh, "kafka"
   migrating), and the render-time name-length guard moves to the Kafka
   CR template.

3. Concurrency (one migration per namespace). The collision guard read
   the "kafka" pool owner and then applied, a non-atomic read-then-write
   two concurrent per-namespace migrations could both pass. When the pool
   is absent the hook now uses "kubectl create" as the atomic arbiter:
   exactly one concurrent creator wins, the loser re-reads the owner and
   aborts.

Verified live on dev9 under server-side apply: a fresh install comes up
KRaft with pools at [0,1,2]/[3,4,5] and the release is "deployed"; an
in-place migration of a 20000-message ZooKeeper cluster ends "deployed",
adopts the broker PVCs in place (same UIDs), and the KRaft data is
byte-for-byte identical to the pre-migration ZooKeeper data (matching
sha256 over every partition:offset:key:value). helm-unittest 86/86 and
hack/kafka-migration-hook.bats 12/12 (the real hook script against a stub
kubectl) cover the resolution branches, the fail-closed reads, the atomic
collision guard and the finalize guard.

Signed-off-by: Alexey Artamonov <[email protected]>
A Kafka cluster's KRaft controller pods are named
"<release>-c-<hash>-<id>" (8-hex hash, id at least one digit), and that
derived pod hostname must fit the 63-char DNS-1123 label limit. That is
tighter than the 53-char Helm release-name budget validateNameLength
enforced, so a release name of 51-53 characters was admitted but then
failed the chart render (kafka.assertNameLength) forever, an
un-recoverable upgrade dead-end for an existing cluster whose name is
immutable. Cap the Kafka app name at admission, mirroring the existing
Kubernetes worker-pool cap, so the overflow surfaces on the Kafka CR the
operator is editing instead of at render time on a pod Strimzi can never
create. The chart's render guard remains the second line of defense and
also catches larger clusters whose higher node ids need more digits.

Signed-off-by: Alexey Artamonov <[email protected]>
Extends the ZooKeeper->KRaft migration hook so that N Kafka clusters
sharing one namespace all reach KRaft automatically, with no manual
steps. This closes the last gap in the migration: previously only one
cluster per namespace could migrate in place (the broker pool must be
named exactly "kafka" to adopt the existing <cluster>-kafka-N brokers,
and that name is namespace-unique), so a second co-namespaced ZooKeeper
cluster hit the render guard and its HelmRelease never rendered again --
stuck on ZooKeeper with no way forward once a future Strimzi drops the
migration.

The clusters now serialize on the "kafka" name automatically. Whichever
upgrade wins the name (claimed atomically via kubectl create) migrates
in place; a cluster that loses fails its hook fast and its next reconcile
retries. Once a migrated cluster sees other ZooKeeper clusters still
waiting, it frees the name: it stands up its own release-hashed
"b-<hash>" broker pool (node IDs pinned clear of the existing broker and
controller ranges), enables Cruise Control, drains every partition off
the "kafka"-pool brokers with a remove-brokers KafkaRebalance, deletes
the drained pool, and removes Cruise Control again. The next waiting
cluster's retry adopts the freed name; the last one to migrate keeps
"kafka". Everything happens inside the one namespace -- the hook's Role
is namespaced and it never reads or touches another.

Robustness details learned from live runs:
- The drain rebalance sets skipHardGoalCheck: true. A mandatory broker
  evacuation must not be blocked by Cruise Control's capacity hard goals;
  under a small resources preset the capacity model can briefly
  over-estimate CPU from noisy samples and reject the proposal with
  CpuCapacityGoal even when real CPU is near idle.
- The drain wait refreshes the proposal (strimzi.io/rebalance=refresh) on
  any NotReady and keeps polling, rather than treating it as terminal: a
  just-enabled Cruise Control is unreachable for a minute or two, and its
  model settles as metric windows age out. This also makes the drain
  resumable -- a re-run refreshes a stale rebalance instead of reading its
  old failure forever.
- Cruise Control is enabled with explicit resources (it is a JVM and a
  tenant LimitRange would OOM it at 128Mi), transiently -- kafka.yaml
  never sets spec.cruiseControl, so Helm's server-side apply does not
  fight the hook patching it on and off.

Every phase is idempotent so a Job killed mid-drain (the drain scales
with data size and can run for hours) resumes on retry; the Job deadline
and the release's helm-install-timeout are raised accordingly.

Verified on a live cluster across five runs of two co-namespaced
ZooKeeper Kafkas migrating to KRaft: both reach KRaft automatically, the
first drains onto its b-<hash> pool and frees the name, the second adopts
"kafka" in place (broker node IDs and PVCs unchanged), and each cluster's
data is byte-for-byte identical before and after. helm-unittest 87/87 and
hack/kafka-migration-hook.bats 14/14 (the real hook script against a stub
kubectl) cover the freeing decision, the atomic name claim, the drain's
refresh/skip-hard-goal handling, and the "delete the pool only after the
rebalance is Ready" guard.

Signed-off-by: Alexey Artamonov <[email protected]>
The "free the kafka pool" gate counted other ZooKeeper clusters through a
`kubectl ... | awk` pipeline under `set -eu` without `pipefail`, its
kubectl error dropped by `2>/dev/null`. A failed read then counted zero and
kept the shared pool, stranding any co-namespaced cluster still waiting for
the name. Separate the read status from the count, mirroring the drain read
below, and cover it with a bats case.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <[email protected]>
The fresh-create Kafka suite relied on the shared 5m cleanup default, but a
KRaft cluster tears down broker and controller node pools, the entity
operator, and their PVCs — more work than a single ZooKeeper node. Under a
contended CI run (concurrent Postgres/OpenSearch/Redis suites) that teardown
overran 5m and failed the suite after every assertion had already passed, so
cleanup latency, not the test, was deciding the verdict. Give this suite its
own 8m cleanup timeout; the migration suite runs in its own namespace and is
unaffected.

Signed-off-by: Alexey Artamonov <[email protected]>
Signed-off-by: Alexey Artamonov <[email protected]>

# Conflicts:
#	docs/storage-immutability.md
#	packages/apps/kafka/tests/topics_test.yaml
#	packages/system/kafka-rd/cozyrds/kafka.yaml

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing left. Please drop the kafka-zk-to-kraft-migration chainsaw suite in this PR.

It does not run the hook, it reimplements it next to it: pool named controller instead of c-<hash>, no strimzi.io/next-node-ids, one broker so the id collision cannot happen at all. That is why it stayed green on the head where an in-place migration lost every message, which you then reproduced 2/2 by hand on a live cluster. It costs 380s on every kafka PR, it asserts that a PVC object exists and never reads a message back, and the strimzi bump has to delete it anyway. A suite that cannot fail on our code will only get in the way later, someone reads it as migration coverage and trusts it. The bats suite and your live runs are the real coverage here.

The multi-instance case is closed, that was my blocker from the last round. Freed name goes through a real drain and the waiting release retries into it.

Rest is not blocking, leaving as is:

  1. Migration wait is still a flat 240 iterations while the real bounds are the job deadline and helm-install-timeout. Timeout on no progress instead (keep last kafkaMetadataState, reset the counter when it changes), then a slow cluster does not flap failed/retry.
  2. Operator is still pinned to 0.45.1-rc1, so tenant brokers run an image tagged rc1 and 3.9.2 cant be selected. Version map of 0.45.2 is a superset of the current one, so it is a drop-in bump, separate PR.

Render error in _versions.tpl still tells operator to migrate one at a time or use separate namespaces, that is not true anymore.

Gate for the strimzi 0.46+ bump is #4365 now, it should not live as a comment here.

The kafka-zk-to-kraft-migration suite did not exercise the pre-upgrade hook,
it reimplemented a simplified flow beside it: a controller pool named
"controller" instead of the hashed c-<hash>, no strimzi.io/next-node-ids pin,
and a single broker so the controller-vs-broker node-id collision could never
occur. That is exactly the defect the hook exists to prevent, so the suite
stayed green on a revision whose in-place migration lost every message —
reproduced by hand 2/2 on a live cluster. A suite that cannot fail on this
code is worse than no suite: it reads as migration coverage and invites trust
it has not earned, while costing ~380s on every Kafka PR and asserting only
that a PVC object exists, never reading a record back. The migration hook's
bats suite and the live upgrade runs are the real coverage; the Strimzi bump
would delete this suite regardless.

Signed-off-by: Alexey Artamonov <[email protected]>
The brokerPoolName render guard still told operators to "migrate one at a time
or use separate namespaces" when a sibling cluster owned the shared "kafka"
pool. That predates automatic name freeing: the migration hook now drains the
first cluster off the shared pool with a Cruise Control rebalance and releases
the name, and the waiting cluster's HelmRelease retries into it — every
co-namespaced Kafka reaches KRaft, no manual action required. Reword the fail
message and the helper comment to describe the transient wait-and-retry, and
update the unit test's error pattern to match.

Signed-off-by: Alexey Artamonov <[email protected]>
@scooby87

Copy link
Copy Markdown
Contributor

myasnikovdaniil thanks for the re-review — both points are addressed on bad152443:

  1. Dropped the kafka-zk-to-kraft-migration chainsaw suite (0b93328). You're right that it never ran the hook — it reimplemented a simplified flow beside it (a controller pool instead of the hashed c-<hash>, no strimzi.io/next-node-ids, a single broker), so the controller-vs-broker node-id collision could not occur and the suite stayed green on the revision whose in-place migration lost every message. The migration hook's bats suite plus the live upgrade runs are the real coverage.

  2. Corrected the stale render error in _versions.tpl (bad1524). It no longer tells operators to "migrate one at a time or use separate namespaces"; it now describes the automatic name freeing — the first cluster drains off the shared kafka pool with a Cruise Control rebalance and releases the name, and the waiting cluster's HelmRelease retries into it, so every co-namespaced Kafka reaches KRaft. The helper comment and the unit test's error pattern were updated to match.

Non-blocking items left as-is per your note: the wait loop's no-progress timeout, the operator bump to 0.45.2 (separate PR), and the Strimzi 0.46+ upgrade gate, which is now tracked as #4365 rather than a comment here.

Re-requesting your review.

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suite is gone and the render error is correct now, approving. The non-blocking ones stay as notes, #4365 holds the gate for the strimzi bump.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants