Repository navigation
fix(virtual-machine): vminstance disk race + bump VM IP/ready timeouts - #2555
Conversation
Three fixes for the vminstance bats, surfaced once the 3x retry was dropped from `Run E2E tests`: 1. Disk delete-recreate race. The teardown used `kubectl delete vmdisk … || true` and immediately created a new disk; sometimes the new disk reconciled before the old finalizer completed, leaving a duplicate PVC and the new VM stuck. Drop the `|| true`, block until the old disk is gone, then create. 2. VM IP wait timeout 20s -> 120s. Under nested-virt + concurrent load, a fresh VM sometimes takes 60-90s to acquire its IP via DHCP. The 20s bound was tripping deterministically. 3. VM ready timeout 20s -> 60s. Same root cause; the readiness probe needs the IP to be reachable, so the lower bound has to grow alongside the IP wait. Signed-off-by: Myasnikov Daniil <[email protected]> Assisted-By: Claude <[email protected]>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses flakiness in the end-to-end testing suite for virtual machine instances. By improving resource deletion synchronization and adjusting timeouts to account for the performance characteristics of nested virtualization, the changes ensure more reliable test execution. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
📝 WalkthroughWalkthroughStrengthen e2e test determinism in ChangesVM Disk & VM Instance e2e flow
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request improves the reliability of E2E tests for VM instances and disks by replacing static sleep calls with dynamic waits for HelmRelease resources. It also increases timeouts for resource deletion, IP assignment, and VM readiness while removing error suppression during cleanup to prevent race conditions. The feedback suggests increasing the HelmRelease readiness wait timeout from 5 seconds to 60 seconds to ensure consistency and prevent intermittent failures in slow CI environments.
| # Wait for the operator to materialise the HelmRelease before kubectl wait | ||
| # kicks in (kubectl wait errors immediately if the object does not exist yet). | ||
| timeout 60 sh -ec "until kubectl -n tenant-test get hr vm-disk-$name >/dev/null 2>&1; do sleep 2; done" | ||
| kubectl -n tenant-test wait hr vm-disk-$name --timeout=5s --for=condition=ready |
There was a problem hiding this comment.
The 5-second timeout for the HelmRelease readiness check is quite tight for slow CI environments or runners under heavy load. Given that other timeouts in this PR have been increased to 60s or 120s to improve reliability, it would be safer to increase this one as well to prevent intermittent failures.
kubectl -n tenant-test wait hr vm-disk-$name --timeout=60s --for=condition=ready
There was a problem hiding this comment.
Agreed — 5s was meant to catch a hung operator quickly, but under runner load the readiness condition can legitimately lag a few seconds past the existence poll. Bumped to 60s in 25180c5 for parity with the rest of the bumps in this PR.
| # routinely takes 30-60s under runner load; the previous 20s was unrealistic | ||
| # and produced flakes. 120s is a comfortable upper bound for nested virt. | ||
| timeout 120 sh -ec "until kubectl -n tenant-test get vmi vm-instance-$name -o jsonpath='{.status.interfaces[0].ipAddress}' | grep -q '[0-9]'; do sleep 2; done" | ||
| kubectl -n tenant-test wait hr vm-instance-$name --timeout=5s --for=condition=ready |
There was a problem hiding this comment.
Similar to the vm-disk check, a 5-second timeout for the HelmRelease readiness might lead to flakes on slow runners. Increasing this to 60s ensures consistency with the other timeout adjustments and improves the robustness of the E2E test.
kubectl -n tenant-test wait hr vm-instance-$name --timeout=60s --for=condition=ready
There was a problem hiding this comment.
Same fix as the vm-disk thread — bumped to 60s in 25180c5. In practice the HR is ready well before this line (the IP-assignment poll above implies the VM has launched), but there's no upside to keeping this aggressive.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
hack/e2e-apps/vminstance.bats (1)
83-84: ⚡ Quick winMove end-of-test cleanup to a
teardown()function and add--ignore-not-found.bats'
teardown()runs after each test regardless of whether it passed or failed. As thesedeletecalls are in the test body, any failure betweenkubectl apply(line 42) and here (e.g., the 120 s IP-assignmenttimeouton line 79 expiring) leaves the VMInstance and VMDisk in the cluster. The next cleanup only coversvminstances(line 41); thevmdisksurvives until "Create a VM Disk" runs, meaning a standalone re-run of just this test accumulates leaked disks.Additionally, lines 83-84 lack
--ignore-not-found. If either resource is gone by the time cleanup runs, thedeletefails and the test is reported as failed despite having exercised its actual assertions successfully.♻️ Proposed refactor: move cleanup to `teardown()`
+teardown() { + kubectl -n tenant-test delete vminstances.apps.cozystack.io test --ignore-not-found --timeout=3m || true + kubectl -n tenant-test delete vmdisks.apps.cozystack.io test --ignore-not-found --timeout=3m || true +} + `@test` "Create a VM Disk" {- kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --timeout=3m - kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --timeout=3m }The
|| trueinteardown()is intentional: a failed cleanup must not mask a passing test, and repeated teardown calls on already-deleted resources should not block the next test. The pre-run deletes at lines 12-13 and 41 can remain as a defensive "flush stale state before setup" guard, or be folded intosetup()for the same reason.Based on learnings: the codebase convention for E2E bats tests is to replace inline cleanup with framework-level
teardown()/teardown_file()rather than embeddingkubectl deletein the test body.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@hack/e2e-apps/vminstance.bats` around lines 83 - 84, Move the two inline cleanup kubectl delete calls for vminstances.apps.cozystack.io and vmdisks.apps.cozystack.io out of the test body and into a bats teardown() function so they always run after each test; in teardown() call kubectl delete for $name and $diskName with the --ignore-not-found flag and ensure the commands are resilient (e.g., append || true) so cleanup failures or already-removed resources don't cause test failures, and remove the original delete lines from the test body.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@hack/e2e-apps/vminstance.bats`:
- Around line 83-84: Move the two inline cleanup kubectl delete calls for
vminstances.apps.cozystack.io and vmdisks.apps.cozystack.io out of the test body
and into a bats teardown() function so they always run after each test; in
teardown() call kubectl delete for $name and $diskName with the
--ignore-not-found flag and ensure the commands are resilient (e.g., append ||
true) so cleanup failures or already-removed resources don't cause test
failures, and remove the original delete lines from the test body.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 2d448539-abf7-4a16-85f7-fd6612b4fb2e
📒 Files selected for processing (1)
hack/e2e-apps/vminstance.bats
5s was too tight under runner load — both vm-disk-$name and vm-instance-$name HR waits could miss the operator's reconcile window. Aligns with the other timeout bumps in this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
hack/e2e-apps/vminstance.bats (1)
12-13: 🏗️ Heavy liftMove inline cleanup to framework-level teardown for failure-safe cleanup.
Cleanup is still embedded in test bodies, so failures before the final delete path can leave residue. Please migrate these deletes to Bats teardown/global cleanup to guarantee execution regardless of where a test aborts.
Based on learnings: “In E2E bats tests … replace inline cleanup … with framework-level teardown/cleanup … apply this pattern to all bats tests under hack/e2e-apps.”
Also applies to: 41-42, 83-84
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-apps/vminstance.bats` around lines 12 - 13, The two inline kubectl cleanup commands should be moved into a framework-level teardown to guarantee execution on test failure: remove the inline deletes from test bodies and implement a teardown_file (or teardown) function in vminstance.bats that iterates over a shared list/array of resource names (e.g., resource_names or created_vms) and runs kubectl -n tenant-test delete vminstances.apps.cozystack.io <name> and kubectl -n tenant-test delete vmdisks.apps.cozystack.io <name> with --ignore-not-found --timeout=3m; update the test resource-creation code (places where name is assigned) to append the name to that shared list so the teardown can clean them up, and apply the same refactor for the other inline deletes referenced (lines 41-42 and 83-84).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@hack/e2e-apps/vminstance.bats`:
- Around line 12-13: The two inline kubectl cleanup commands should be moved
into a framework-level teardown to guarantee execution on test failure: remove
the inline deletes from test bodies and implement a teardown_file (or teardown)
function in vminstance.bats that iterates over a shared list/array of resource
names (e.g., resource_names or created_vms) and runs kubectl -n tenant-test
delete vminstances.apps.cozystack.io <name> and kubectl -n tenant-test delete
vmdisks.apps.cozystack.io <name> with --ignore-not-found --timeout=3m; update
the test resource-creation code (places where name is assigned) to append the
name to that shared list so the teardown can clean them up, and apply the same
refactor for the other inline deletes referenced (lines 41-42 and 83-84).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 9c70cbdd-31a1-4c2e-8b9a-a9bb0f18f5f5
📒 Files selected for processing (1)
hack/e2e-apps/vminstance.bats
|
Re: the Good catch on the leak path. Confirmed: if I'd rather keep this PR scoped to the disk race + timeout bumps it advertises, and address the |
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
Business context: vminstance.bats fails non-deterministically once the 3× retry on Run E2E tests is dropped (in #2500). Three classes of flake: (1) old vmdisk in finalizer drain races a fresh apply because || true swallowed the 2m delete timeout, (2) nested-virt VM IP DHCP routinely takes 60-90s but the wait was capped at 20s, (3) HR-ready window after IP assignment was 20s but qemu-guest-agent + readiness probe needs longer.
Both commits address those classes; Gemini's 5s HR-ready timeout was bumped to 60s in 25180c5.
Non-blocking follow-ups
-
Final-cleanup timeout couples teardown latency to test result (lines 83-84). The post-success
kubectl delete vminstances/vmdisks --timeout=3m(no--ignore-not-found, no|| true) turns slow but correct finalizer drain into a test failure after the actual assertions have passed. The pre-PR shape was also blocking (kubectl deletedefaults to--wait=truewith a size-based timeout) but had no explicit cap, so a long teardown blocked CI rather than failing it. The 3m cap on lines 83-84 plus the absence of--ignore-not-foundintroduces two new failure modes: (a) finalizer drain >3m on a loaded runner fails the suite, (b) any race that already deleted the resource fails the suite. Symmetrize with the pre-clean shape on lines 12-13/41:kubectl delete ... --ignore-not-found --timeout=3m. Optional but cheap. -
Pre-clean asymmetry on line 41. "Create a VM Instance" pre-deletes only
vminstances, notvmdisks. Inert in CI because the whole.batsfile runs in order and lines 12-13 cover both, but a standalone re-run of just the second test would find a stale vmdisk. Acknowledged in the PR thread; cross-suite teardown is a separate refactor. Optional one-liner if it's cheap:kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --ignore-not-found --timeout=3mnext to line 41. -
CodeRabbit's
teardown()suggestion is a no-op here.hack/cozytest.shis pure shell, not real bats — the@test "..."blocks become regular shell functions running underset -e, and ateardown()definition would not auto-run between tests. Author was right to decline. (Minor follow-up worth tracking separately: a one-line comment in eachhack/e2e-apps/*.batsclarifying the shim status would prevent reviewers from suggesting the same refactor on every PR. Out of scope here.) -
Two commits could squash before merge. 25180c5 (
bump HR readiness wait from 5s to 60s) responds to bot feedback on 4a5946a and is part of the same logical "right-size the timeouts in this test" change. The current pair isn't bad — message 2 stands on its own — but a single commit reads more cleanly ingit log. -
Description nit. "Surfaced once the 3x retry was dropped from
Run E2E tests" reads as if the retry has been dropped onmain; it's still in.github/workflows/pull-requests.yaml:267and is dropped in #2500. Tightening to "will surface once #2500 lands" or "surfaces under #2500" helps future bisecters.
Symmetrize the post-success cleanup on lines 83-84 with the pre-clean shape on lines 12-13/41. Without --ignore-not-found, a race that already removed the resource (e.g., a parallel teardown or finalizer-driven GC) turns a passing test into a failure after assertions have already run. Review feedback PR #2555 (lexfrei follow-up #1). Signed-off-by: Myasnikov Daniil <[email protected]>
The pre-clean on line 41 only covered vminstances, leaving a leak path for standalone re-runs of the second test (e.g., bats --filter 'VM Instance'): a stale vmdisk from a prior run would survive into kubectl apply, racing the same delete-finalizer-drain that lines 12-13 already guard against. Inert in CI (the file runs in order and lines 12-13 already pre-clean both kinds), but cheap to symmetrize. Review feedback PR #2555 (lexfrei follow-up #2). Signed-off-by: Myasnikov Daniil <[email protected]>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
hack/e2e-apps/vminstance.bats (1)
86-87: ⚡ Quick winCleanup leak path: move final deletes to a
teardown()functionIf the IP-poll (line 82), HR-ready (line 83), or VM-ready (line 85) assertion times out, bats exits the test immediately and lines 86–87 never execute, leaving
vminstancesandvmdisksresources live in the cluster. A subsequent standalone or re-run invocation will then hit the pre-clean on lines 43–44 and wait up to 3 min each for those orphaned resources to finalise.Moving cleanup into a
teardown()guarantees it runs regardless of how the test exits.♻️ Proposed refactor: `teardown()` function
+teardown() { + kubectl -n tenant-test delete vminstances.apps.cozystack.io test --ignore-not-found --timeout=3m || true + kubectl -n tenant-test delete vmdisks.apps.cozystack.io test --ignore-not-found --timeout=3m || true +} + `@test` "Create a VM Disk" {- kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m - kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --ignore-not-found --timeout=3m }Note:
teardown()must suppress errors (|| true) because bats marks the test as failed if teardown itself exits non-zero and the resource may already be gone.Based on learnings: "In E2E bats tests like hack/e2e-apps/harbor.bats, replace inline cleanup (e.g., kubectl delete in the test body) with framework-level teardown/cleanup… Apply this pattern to all bats tests under hack/e2e-apps."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-apps/vminstance.bats` around lines 86 - 87, Move the two inline cleanup kubectl delete calls that remove vminstances and vmdisks (the commands using $name and $diskName) into a bats teardown() function so cleanup always runs even if earlier assertions time out; implement teardown() to run kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m and kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --ignore-not-found --timeout=3m, ensure each delete command suppresses errors (append || true) and remove the original inline deletes from the test body so only teardown handles final deletion.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@hack/e2e-apps/vminstance.bats`:
- Around line 86-87: Move the two inline cleanup kubectl delete calls that
remove vminstances and vmdisks (the commands using $name and $diskName) into a
bats teardown() function so cleanup always runs even if earlier assertions time
out; implement teardown() to run kubectl -n tenant-test delete
vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m and kubectl
-n tenant-test delete vmdisks.apps.cozystack.io $diskName --ignore-not-found
--timeout=3m, ensure each delete command suppresses errors (append || true) and
remove the original inline deletes from the test body so only teardown handles
final deletion.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 23fabbe9-121a-4f8b-86d0-f865ad6724d3
📒 Files selected for processing (1)
hack/e2e-apps/vminstance.bats
"Create a VM Instance" referenced the VMDisk created by "Create a VM Disk" via the shared name "test". This implicit ordering meant any defensive pre-clean of vmdisks in test 2 would destroy the disk test 1 just provisioned, and the vm-instance chart's lookup-or-fail on the DataVolume would then surface as "Specified disk not exists in cluster: test". Make each test self-contained: - "Create a VM Disk" cleans up its own VMDisk at the end. - "Create a VM Instance" creates its own VMDisk inline (renamed to "vmi-test" to avoid colliding with any in-flight finalizer drain on test 1's resources) and waits HR -> DV -> PVC before applying the VMInstance. Replaces the pre-delete approach from the prior followup commit, which broke the data dependency. Signed-off-by: Myasnikov Daniil <[email protected]>
71219ab to
209fa08
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/e2e-apps/vminstance.bats`:
- Around line 100-111: The inline cleanup commands that delete
vminstances.apps.cozystack.io and vmdisks.apps.cozystack.io (the kubectl -n
tenant-test delete ... calls near the end of the test) can be skipped if earlier
waits fail, so move these deletions into the test framework's teardown or global
cleanup routine instead of inside the test body; remove the kubectl delete lines
from vminstance.bats and register the resource names (e.g., $name and $diskName
created by the test) with the framework-level teardown function so that teardown
always attempts deletion regardless of test failures, ensuring cleanup runs even
when waits like get hr/get vmi or wait vm fail.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 2aac7d69-a77e-4b34-b71e-c006d8c38dec
📒 Files selected for processing (1)
hack/e2e-apps/vminstance.bats
| # Wait for the operator to materialise the HelmRelease before downstream | ||
| # waits proceed (kubectl wait errors immediately if the HR does not exist). | ||
| timeout 60 sh -ec "until kubectl -n tenant-test get hr vm-instance-$name >/dev/null 2>&1; do sleep 2; done" | ||
| # Nested KubeVirt VM startup (virt-launcher + libvirt + cloud-init DHCP) | ||
| # routinely takes 30-60s under runner load; the previous 20s was unrealistic | ||
| # and produced flakes. 120s is a comfortable upper bound for nested virt. | ||
| timeout 120 sh -ec "until kubectl -n tenant-test get vmi vm-instance-$name -o jsonpath='{.status.interfaces[0].ipAddress}' | grep -q '[0-9]'; do sleep 2; done" | ||
| kubectl -n tenant-test wait hr vm-instance-$name --timeout=60s --for=condition=ready | ||
| # VM ready follows IP assignment closely; 60s gives buffer for the qemu-guest-agent. | ||
| kubectl -n tenant-test wait vm vm-instance-$name --timeout=60s --for=condition=ready | ||
| kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m | ||
| kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --ignore-not-found --timeout=3m |
There was a problem hiding this comment.
Cleanup is still not guaranteed when waits fail mid-test.
If Line 102/106/107/109 fails, Lines 110-111 won’t run, so vmdisk/vminstance can leak and affect later runs. Move resource deletion into teardown()/framework-level cleanup and keep test bodies focused on create/assert steps.
Based on learnings: In E2E bats tests under hack/e2e-apps, cleanup should be framework-level teardown rather than inline per-test cleanup to ensure reliable cleanup on failures.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/e2e-apps/vminstance.bats` around lines 100 - 111, The inline cleanup
commands that delete vminstances.apps.cozystack.io and vmdisks.apps.cozystack.io
(the kubectl -n tenant-test delete ... calls near the end of the test) can be
skipped if earlier waits fail, so move these deletions into the test framework's
teardown or global cleanup routine instead of inside the test body; remove the
kubectl delete lines from vminstance.bats and register the resource names (e.g.,
$name and $diskName created by the test) with the framework-level teardown
function so that teardown always attempts deletion regardless of test failures,
ensuring cleanup runs even when waits like get hr/get vmi or wait vm fail.
There was a problem hiding this comment.
Yes, proper cleanup may be helpful. I created followup issue for this #2583. But anyway it MAY affect just tests runned after failed test, so i think it is not that big scope, considering failure in test already blocks tests, and it will need fix and rerun.
There was a problem hiding this comment.
@myasnikovdaniil, that's a fair point — a mid-test failure already halts the suite and requires manual intervention, so the incremental blast radius of a leaked resource is low in practice. Tracking it in #2583 for a proper teardown() refactor across the bats suite is the right call. Thanks for filing the follow-up!
🧠 Learnings used
Learnt from: lexfrei
Repo: cozystack/cozystack PR: 2058
File: hack/e2e-apps/harbor.bats:3-45
Timestamp: 2026-02-16T18:50:59.859Z
Learning: In E2E bats tests like hack/e2e-apps/harbor.bats, replace inline cleanup (e.g., kubectl delete in the test body) with framework-level teardown/cleanup. Do not rely on traps or per-test cleanup; instead, move cleanup logic to a global or framework-provided teardown so the framework can clean namespaces between runs. Apply this pattern to all bats tests under hack/e2e-apps (and similar E2E suites) to ensure consistent and reliable cleanup without embedding it in individual tests.
Learnt from: myasnikovdaniil
Repo: cozystack/cozystack PR: 2559
File: hack/select-e2e_test.bats:75-80
Timestamp: 2026-05-02T14:46:54.942Z
Learning: In Bash Automated Testing System (Bats) tests, the `run` helper captures command output and strips the trailing newline from `$output`. Therefore, for single-line commands, exact string assertions like `[ "$output" = "redis" ]` should be safe and should not be tripped up by trailing newline/whitespace. No extra `xargs`-style trimming is needed solely to remove a trailing newline for single-line outputs.
Learnt from: myasnikovdaniil
Repo: cozystack/cozystack PR: 2554
File: hack/e2e-install-cozystack.bats:289-291
Timestamp: 2026-05-05T08:35:37.488Z
Learning: In cozystack e2e Bats tests, Tenant CR–backed resources (e.g., `hr/tenant-test` and the corresponding `HelmRelease` objects for other Tenant CRs) are created asynchronously by `cozystack-controller` after the Tenant CR is applied; they are not rendered by a parent Helm chart. Because of this, there’s a real race between `kubectl apply` of the Tenant and the controller materializing the `HelmRelease`/`hr` resources. Before running `kubectl wait` on those HelmReleases in tests, add an existence backstop by polling until the expected resource exists (e.g., `timeout 60 sh -ec 'until kubectl get hr/tenant-test -n tenant-root >/dev/null 2>&1; do sleep 2; done'`).
Signed-off-by: myasnikovdaniil <[email protected]>
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
LGTM. Three fixes all targeted, all correctly motivated.
The || true removal + 2m → 3m timeout pre-test delete is the right shape — kubectl apply on a resource still in finalizer-drain silently no-ops with "Detected changes to resource ... which is currently being deleted", which the previous code would then race against. Blocking the delete deterministically before apply removes that race.
Decoupling the second test's VMDisk into a distinct vmi-test name is the right structural fix — prior PV/PVC finalizer drain on the first test's test disk could collide with the second test's apply. Each test now owning its own resources is cleaner than tightening timeouts on the shared one.
20s → 120s VM-IP / 20s → 60s VM-ready are realistic for nested-virt under runner load (DHCP + virt-launcher + cloud-init combine to ~30-90s on cold boots). Comments explain the bound rather than just enlarge the number, which is what I want to see.
Existence backstop on vm-instance-$name HR slipping in here — correct call, same hunk, no reason to split it.
Ship it.
What this PR does
Three fixes for
hack/e2e-apps/vminstance.bats, surfaced once the 3x retry was dropped fromRun E2E tests:Disk delete-recreate race. The teardown used
kubectl delete vmdisk … || trueand immediately created a new disk; sometimes the new disk reconciled before the old finalizer completed, leaving a duplicate PVC and the new VM stuck. Drop the|| true, block until the old disk is gone, then create.VM IP wait timeout 20s → 120s. Under nested-virt + concurrent runner load, a fresh VM sometimes takes 60–90 s to acquire its IP via DHCP. The 20 s bound was tripping deterministically.
VM ready timeout 20s → 60s. Same root cause; the readiness probe needs the IP to be reachable, so the lower bound has to grow alongside the IP wait.
Existence backstop on the
vm-instance-$nameHR is added here too (it touches the same hunk as the timeout fix; cleaner to land them together).Surfaced from #2500.
Release note
Summary by CodeRabbit