Skip to content

fix(virtual-machine): vminstance disk race + bump VM IP/ready timeouts - #2555

Merged
myasnikovdaniil merged 5 commits into
mainfrom
daniil/split-vminstance
May 6, 2026
Merged

myasnikovdaniil merged 5 commits into
mainfrom
daniil/split-vminstance

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented May 2, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

Three fixes for hack/e2e-apps/vminstance.bats, surfaced once the 3x retry was dropped from Run E2E tests:

  1. Disk delete-recreate race. The teardown used kubectl delete vmdisk … || true and immediately created a new disk; sometimes the new disk reconciled before the old finalizer completed, leaving a duplicate PVC and the new VM stuck. Drop the || true, block until the old disk is gone, then create.

  2. VM IP wait timeout 20s → 120s. Under nested-virt + concurrent runner load, a fresh VM sometimes takes 60–90 s to acquire its IP via DHCP. The 20 s bound was tripping deterministically.

  3. VM ready timeout 20s → 60s. Same root cause; the readiness probe needs the IP to be reachable, so the lower bound has to grow alongside the IP wait.

Existence backstop on the vm-instance-$name HR is added here too (it touches the same hunk as the timeout fix; cleaner to land them together).

Surfaced from #2500.

Release note

NONE

Summary by CodeRabbit

  • Tests
    • Improved end-to-end test reliability for VM instance and disk flows.
    • Replaced short sleeps with explicit waits for upstream and downstream deployment artifacts.
    • Extended polling windows and timeouts for IP assignment and readiness checks.
    • Strengthened cleanup to reliably remove prior VM and disk resources before and after tests.

Three fixes for the vminstance bats, surfaced once the 3x retry was
dropped from `Run E2E tests`:

1. Disk delete-recreate race. The teardown used
   `kubectl delete vmdisk … || true` and immediately created a new
   disk; sometimes the new disk reconciled before the old finalizer
   completed, leaving a duplicate PVC and the new VM stuck. Drop the
   `|| true`, block until the old disk is gone, then create.

2. VM IP wait timeout 20s -> 120s. Under nested-virt + concurrent
   load, a fresh VM sometimes takes 60-90s to acquire its IP via
   DHCP. The 20s bound was tripping deterministically.

3. VM ready timeout 20s -> 60s. Same root cause; the readiness
   probe needs the IP to be reachable, so the lower bound has to
   grow alongside the IP wait.

Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-By: Claude <[email protected]>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses flakiness in the end-to-end testing suite for virtual machine instances. By improving resource deletion synchronization and adjusting timeouts to account for the performance characteristics of nested virtualization, the changes ensure more reliable test execution.

Highlights

  • Race Condition Resolution: Removed '|| true' from resource deletion commands and increased timeouts to 3m to ensure resources are fully cleaned up before re-creation, preventing finalizer-related race conditions.
  • Timeout Adjustments: Increased VM IP acquisition timeout from 20s to 120s and VM readiness timeout from 20s to 60s to accommodate nested virtualization overhead and runner load.
  • Improved Synchronization: Added explicit wait loops for HelmRelease existence before proceeding with downstream checks to prevent premature failures.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files area/virtualization Issues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import) kind/bug Categorizes issue or PR as related to a bug labels May 2, 2026
@coderabbitai

coderabbitai Bot commented May 2, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Strengthen e2e test determinism in hack/e2e-apps/vminstance.bats: remove error-masking on deletions, extend delete timeouts to 3m, add explicit waits for HelmRelease materialization, lengthen VMI IP polling to 120s, and increase readiness wait timeouts for VM Disk and VM Instance flows.

Changes

VM Disk & VM Instance e2e flow

Layer / File(s) Summary
Pre-apply Cleanup
hack/e2e-apps/vminstance.bats
Replace deletion commands that masked errors (e.g., `
HelmRelease Materialization
hack/e2e-apps/vminstance.bats
Add polling to wait for the operator to create the HelmRelease objects (vm-disk-$name, vm-instance-$name) before waiting on downstream resources (60s polling).
Downstream Readiness
hack/e2e-apps/vminstance.bats
After HelmRelease presence, wait for HelmRelease readiness and for DV/PVC readiness for VMDisk flows using explicit kubectl wait calls.
VM Startup & IP Polling
hack/e2e-apps/vminstance.bats
Increase VMI IP polling to a 120s cap with repeated checks; increase kubectl wait timeouts for HelmRelease and VM readiness to 60s.
Final Cleanup
hack/e2e-apps/vminstance.bats
Ensure end-of-test cleanup removes both VMInstance and VMDisk with --ignore-not-found --timeout=3m to reset state for subsequent tests.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

I nibble logs beneath the moon,
Timeouts stretch and waits are tuned.
Helm releases rise, IPs arrive,
Cleanups run so tests survive.
A rabbit cheers—CI dreams croon. 🐇

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix(virtual-machine): vminstance disk race + bump VM IP/ready timeouts' accurately summarizes the main changes: resolving a disk race condition and increasing timeouts for VM IP and readiness checks.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch daniil/split-vminstance

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@dosubot dosubot Bot added the area/testing Issues or PRs related to testing (e2e, bats, unit tests) label May 2, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves the reliability of E2E tests for VM instances and disks by replacing static sleep calls with dynamic waits for HelmRelease resources. It also increases timeouts for resource deletion, IP assignment, and VM readiness while removing error suppression during cleanup to prevent race conditions. The feedback suggests increasing the HelmRelease readiness wait timeout from 5 seconds to 60 seconds to ensure consistency and prevent intermittent failures in slow CI environments.

Comment thread hack/e2e-apps/vminstance.bats Outdated
# Wait for the operator to materialise the HelmRelease before kubectl wait
# kicks in (kubectl wait errors immediately if the object does not exist yet).
timeout 60 sh -ec "until kubectl -n tenant-test get hr vm-disk-$name >/dev/null 2>&1; do sleep 2; done"
kubectl -n tenant-test wait hr vm-disk-$name --timeout=5s --for=condition=ready

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The 5-second timeout for the HelmRelease readiness check is quite tight for slow CI environments or runners under heavy load. Given that other timeouts in this PR have been increased to 60s or 120s to improve reliability, it would be safer to increase this one as well to prevent intermittent failures.

  kubectl -n tenant-test wait hr vm-disk-$name --timeout=60s --for=condition=ready

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — 5s was meant to catch a hung operator quickly, but under runner load the readiness condition can legitimately lag a few seconds past the existence poll. Bumped to 60s in 25180c5 for parity with the rest of the bumps in this PR.

Comment thread hack/e2e-apps/vminstance.bats Outdated
# routinely takes 30-60s under runner load; the previous 20s was unrealistic
# and produced flakes. 120s is a comfortable upper bound for nested virt.
timeout 120 sh -ec "until kubectl -n tenant-test get vmi vm-instance-$name -o jsonpath='{.status.interfaces[0].ipAddress}' | grep -q '[0-9]'; do sleep 2; done"
kubectl -n tenant-test wait hr vm-instance-$name --timeout=5s --for=condition=ready

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similar to the vm-disk check, a 5-second timeout for the HelmRelease readiness might lead to flakes on slow runners. Increasing this to 60s ensures consistency with the other timeout adjustments and improves the robustness of the E2E test.

  kubectl -n tenant-test wait hr vm-instance-$name --timeout=60s --for=condition=ready

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same fix as the vm-disk thread — bumped to 60s in 25180c5. In practice the HR is ready well before this line (the IP-assignment poll above implies the VM has launched), but there's no upside to keeping this aggressive.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
hack/e2e-apps/vminstance.bats (1)

83-84: ⚡ Quick win

Move end-of-test cleanup to a teardown() function and add --ignore-not-found.

bats' teardown() runs after each test regardless of whether it passed or failed. As these delete calls are in the test body, any failure between kubectl apply (line 42) and here (e.g., the 120 s IP-assignment timeout on line 79 expiring) leaves the VMInstance and VMDisk in the cluster. The next cleanup only covers vminstances (line 41); the vmdisk survives until "Create a VM Disk" runs, meaning a standalone re-run of just this test accumulates leaked disks.

Additionally, lines 83-84 lack --ignore-not-found. If either resource is gone by the time cleanup runs, the delete fails and the test is reported as failed despite having exercised its actual assertions successfully.

♻️ Proposed refactor: move cleanup to `teardown()`
+teardown() {
+  kubectl -n tenant-test delete vminstances.apps.cozystack.io test --ignore-not-found --timeout=3m || true
+  kubectl -n tenant-test delete vmdisks.apps.cozystack.io test --ignore-not-found --timeout=3m || true
+}
+
 `@test` "Create a VM Disk" {
-  kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --timeout=3m
-  kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --timeout=3m
 }

The || true in teardown() is intentional: a failed cleanup must not mask a passing test, and repeated teardown calls on already-deleted resources should not block the next test. The pre-run deletes at lines 12-13 and 41 can remain as a defensive "flush stale state before setup" guard, or be folded into setup() for the same reason.

Based on learnings: the codebase convention for E2E bats tests is to replace inline cleanup with framework-level teardown()/teardown_file() rather than embedding kubectl delete in the test body.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@hack/e2e-apps/vminstance.bats` around lines 83 - 84, Move the two inline
cleanup kubectl delete calls for vminstances.apps.cozystack.io and
vmdisks.apps.cozystack.io out of the test body and into a bats teardown()
function so they always run after each test; in teardown() call kubectl delete
for $name and $diskName with the --ignore-not-found flag and ensure the commands
are resilient (e.g., append || true) so cleanup failures or already-removed
resources don't cause test failures, and remove the original delete lines from
the test body.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@hack/e2e-apps/vminstance.bats`:
- Around line 83-84: Move the two inline cleanup kubectl delete calls for
vminstances.apps.cozystack.io and vmdisks.apps.cozystack.io out of the test body
and into a bats teardown() function so they always run after each test; in
teardown() call kubectl delete for $name and $diskName with the
--ignore-not-found flag and ensure the commands are resilient (e.g., append ||
true) so cleanup failures or already-removed resources don't cause test
failures, and remove the original delete lines from the test body.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 2d448539-abf7-4a16-85f7-fd6612b4fb2e

📥 Commits

Reviewing files that changed from the base of the PR and between 5786afe and 4a5946a.

📒 Files selected for processing (1)
  • hack/e2e-apps/vminstance.bats

@github-actions github-actions Bot removed the size:M label May 4, 2026
5s was too tight under runner load — both vm-disk-$name and
vm-instance-$name HR waits could miss the operator's reconcile
window. Aligns with the other timeout bumps in this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
@dosubot dosubot Bot added size:M and removed size/M This PR changes 30-99 lines, ignoring generated files labels May 5, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
hack/e2e-apps/vminstance.bats (1)

12-13: 🏗️ Heavy lift

Move inline cleanup to framework-level teardown for failure-safe cleanup.

Cleanup is still embedded in test bodies, so failures before the final delete path can leave residue. Please migrate these deletes to Bats teardown/global cleanup to guarantee execution regardless of where a test aborts.

Based on learnings: “In E2E bats tests … replace inline cleanup … with framework-level teardown/cleanup … apply this pattern to all bats tests under hack/e2e-apps.”

Also applies to: 41-42, 83-84

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-apps/vminstance.bats` around lines 12 - 13, The two inline kubectl
cleanup commands should be moved into a framework-level teardown to guarantee
execution on test failure: remove the inline deletes from test bodies and
implement a teardown_file (or teardown) function in vminstance.bats that
iterates over a shared list/array of resource names (e.g., resource_names or
created_vms) and runs kubectl -n tenant-test delete
vminstances.apps.cozystack.io <name> and kubectl -n tenant-test delete
vmdisks.apps.cozystack.io <name> with --ignore-not-found --timeout=3m; update
the test resource-creation code (places where name is assigned) to append the
name to that shared list so the teardown can clean them up, and apply the same
refactor for the other inline deletes referenced (lines 41-42 and 83-84).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@hack/e2e-apps/vminstance.bats`:
- Around line 12-13: The two inline kubectl cleanup commands should be moved
into a framework-level teardown to guarantee execution on test failure: remove
the inline deletes from test bodies and implement a teardown_file (or teardown)
function in vminstance.bats that iterates over a shared list/array of resource
names (e.g., resource_names or created_vms) and runs kubectl -n tenant-test
delete vminstances.apps.cozystack.io <name> and kubectl -n tenant-test delete
vmdisks.apps.cozystack.io <name> with --ignore-not-found --timeout=3m; update
the test resource-creation code (places where name is assigned) to append the
name to that shared list so the teardown can clean them up, and apply the same
refactor for the other inline deletes referenced (lines 41-42 and 83-84).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9c70cbdd-31a1-4c2e-8b9a-a9bb0f18f5f5

📥 Commits

Reviewing files that changed from the base of the PR and between 4a5946a and 25180c5.

📒 Files selected for processing (1)
  • hack/e2e-apps/vminstance.bats

@myasnikovdaniil

Copy link
Copy Markdown
Contributor Author

Re: the teardown() nitpick on lines 83–84 from CodeRabbit —

Good catch on the leak path. Confirmed: if kubectl wait vm … --timeout=60s on line 82 fails, the trailing deletes don't run, and a standalone re-run of just "Create a VM Instance" (skipping "Create a VM Disk") would find a leftover vmdisk, because the in-test pre-cleanup on line 41 only covers vminstances.

I'd rather keep this PR scoped to the disk race + timeout bumps it advertises, and address the teardown() refactor in a follow-up that applies the same pattern across vmdisk.bats and the other bats suites consistently. Will file the follow-up separately.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Business context: vminstance.bats fails non-deterministically once the 3× retry on Run E2E tests is dropped (in #2500). Three classes of flake: (1) old vmdisk in finalizer drain races a fresh apply because || true swallowed the 2m delete timeout, (2) nested-virt VM IP DHCP routinely takes 60-90s but the wait was capped at 20s, (3) HR-ready window after IP assignment was 20s but qemu-guest-agent + readiness probe needs longer.

Both commits address those classes; Gemini's 5s HR-ready timeout was bumped to 60s in 25180c5.

Non-blocking follow-ups

  1. Final-cleanup timeout couples teardown latency to test result (lines 83-84). The post-success kubectl delete vminstances/vmdisks --timeout=3m (no --ignore-not-found, no || true) turns slow but correct finalizer drain into a test failure after the actual assertions have passed. The pre-PR shape was also blocking (kubectl delete defaults to --wait=true with a size-based timeout) but had no explicit cap, so a long teardown blocked CI rather than failing it. The 3m cap on lines 83-84 plus the absence of --ignore-not-found introduces two new failure modes: (a) finalizer drain >3m on a loaded runner fails the suite, (b) any race that already deleted the resource fails the suite. Symmetrize with the pre-clean shape on lines 12-13/41: kubectl delete ... --ignore-not-found --timeout=3m. Optional but cheap.

  2. Pre-clean asymmetry on line 41. "Create a VM Instance" pre-deletes only vminstances, not vmdisks. Inert in CI because the whole .bats file runs in order and lines 12-13 cover both, but a standalone re-run of just the second test would find a stale vmdisk. Acknowledged in the PR thread; cross-suite teardown is a separate refactor. Optional one-liner if it's cheap: kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --ignore-not-found --timeout=3m next to line 41.

  3. CodeRabbit's teardown() suggestion is a no-op here. hack/cozytest.sh is pure shell, not real bats — the @test "..." blocks become regular shell functions running under set -e, and a teardown() definition would not auto-run between tests. Author was right to decline. (Minor follow-up worth tracking separately: a one-line comment in each hack/e2e-apps/*.bats clarifying the shim status would prevent reviewers from suggesting the same refactor on every PR. Out of scope here.)

  4. Two commits could squash before merge. 25180c5 (bump HR readiness wait from 5s to 60s) responds to bot feedback on 4a5946a and is part of the same logical "right-size the timeouts in this test" change. The current pair isn't bad — message 2 stands on its own — but a single commit reads more cleanly in git log.

  5. Description nit. "Surfaced once the 3x retry was dropped from Run E2E tests" reads as if the retry has been dropped on main; it's still in .github/workflows/pull-requests.yaml:267 and is dropped in #2500. Tightening to "will surface once #2500 lands" or "surfaces under #2500" helps future bisecters.

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label May 5, 2026
Symmetrize the post-success cleanup on lines 83-84 with the pre-clean
shape on lines 12-13/41. Without --ignore-not-found, a race that already
removed the resource (e.g., a parallel teardown or finalizer-driven GC)
turns a passing test into a failure after assertions have already run.

Review feedback PR #2555 (lexfrei follow-up #1).

Signed-off-by: Myasnikov Daniil <[email protected]>
myasnikovdaniil added a commit that referenced this pull request May 5, 2026
The pre-clean on line 41 only covered vminstances, leaving a leak path
for standalone re-runs of the second test (e.g., bats --filter 'VM
Instance'): a stale vmdisk from a prior run would survive into kubectl
apply, racing the same delete-finalizer-drain that lines 12-13 already
guard against. Inert in CI (the file runs in order and lines 12-13
already pre-clean both kinds), but cheap to symmetrize.

Review feedback PR #2555 (lexfrei follow-up #2).

Signed-off-by: Myasnikov Daniil <[email protected]>
@github-actions github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files and removed size:M labels May 5, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
hack/e2e-apps/vminstance.bats (1)

86-87: ⚡ Quick win

Cleanup leak path: move final deletes to a teardown() function

If the IP-poll (line 82), HR-ready (line 83), or VM-ready (line 85) assertion times out, bats exits the test immediately and lines 86–87 never execute, leaving vminstances and vmdisks resources live in the cluster. A subsequent standalone or re-run invocation will then hit the pre-clean on lines 43–44 and wait up to 3 min each for those orphaned resources to finalise.

Moving cleanup into a teardown() guarantees it runs regardless of how the test exits.

♻️ Proposed refactor: `teardown()` function
+teardown() {
+  kubectl -n tenant-test delete vminstances.apps.cozystack.io test --ignore-not-found --timeout=3m || true
+  kubectl -n tenant-test delete vmdisks.apps.cozystack.io test --ignore-not-found --timeout=3m || true
+}
+
 `@test` "Create a VM Disk" {
-  kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m
-  kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --ignore-not-found --timeout=3m
 }

Note: teardown() must suppress errors (|| true) because bats marks the test as failed if teardown itself exits non-zero and the resource may already be gone.

Based on learnings: "In E2E bats tests like hack/e2e-apps/harbor.bats, replace inline cleanup (e.g., kubectl delete in the test body) with framework-level teardown/cleanup… Apply this pattern to all bats tests under hack/e2e-apps."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-apps/vminstance.bats` around lines 86 - 87, Move the two inline
cleanup kubectl delete calls that remove vminstances and vmdisks (the commands
using $name and $diskName) into a bats teardown() function so cleanup always
runs even if earlier assertions time out; implement teardown() to run kubectl -n
tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found
--timeout=3m and kubectl -n tenant-test delete vmdisks.apps.cozystack.io
$diskName --ignore-not-found --timeout=3m, ensure each delete command suppresses
errors (append || true) and remove the original inline deletes from the test
body so only teardown handles final deletion.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@hack/e2e-apps/vminstance.bats`:
- Around line 86-87: Move the two inline cleanup kubectl delete calls that
remove vminstances and vmdisks (the commands using $name and $diskName) into a
bats teardown() function so cleanup always runs even if earlier assertions time
out; implement teardown() to run kubectl -n tenant-test delete
vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m and kubectl
-n tenant-test delete vmdisks.apps.cozystack.io $diskName --ignore-not-found
--timeout=3m, ensure each delete command suppresses errors (append || true) and
remove the original inline deletes from the test body so only teardown handles
final deletion.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 23fabbe9-121a-4f8b-86d0-f865ad6724d3

📥 Commits

Reviewing files that changed from the base of the PR and between 25180c5 and 71219ab.

📒 Files selected for processing (1)
  • hack/e2e-apps/vminstance.bats

"Create a VM Instance" referenced the VMDisk created by "Create a VM
Disk" via the shared name "test". This implicit ordering meant any
defensive pre-clean of vmdisks in test 2 would destroy the disk test 1
just provisioned, and the vm-instance chart's lookup-or-fail on the
DataVolume would then surface as "Specified disk not exists in cluster:
test".

Make each test self-contained:
- "Create a VM Disk" cleans up its own VMDisk at the end.
- "Create a VM Instance" creates its own VMDisk inline (renamed to
  "vmi-test" to avoid colliding with any in-flight finalizer drain on
  test 1's resources) and waits HR -> DV -> PVC before applying the
  VMInstance.

Replaces the pre-delete approach from the prior followup commit, which
broke the data dependency.

Signed-off-by: Myasnikov Daniil <[email protected]>
@myasnikovdaniil
myasnikovdaniil force-pushed the daniil/split-vminstance branch from 71219ab to 209fa08 Compare May 6, 2026 04:05
@dosubot dosubot Bot added size:M and removed size/M This PR changes 30-99 lines, ignoring generated files labels May 6, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/e2e-apps/vminstance.bats`:
- Around line 100-111: The inline cleanup commands that delete
vminstances.apps.cozystack.io and vmdisks.apps.cozystack.io (the kubectl -n
tenant-test delete ... calls near the end of the test) can be skipped if earlier
waits fail, so move these deletions into the test framework's teardown or global
cleanup routine instead of inside the test body; remove the kubectl delete lines
from vminstance.bats and register the resource names (e.g., $name and $diskName
created by the test) with the framework-level teardown function so that teardown
always attempts deletion regardless of test failures, ensuring cleanup runs even
when waits like get hr/get vmi or wait vm fail.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 2aac7d69-a77e-4b34-b71e-c006d8c38dec

📥 Commits

Reviewing files that changed from the base of the PR and between 71219ab and 209fa08.

📒 Files selected for processing (1)
  • hack/e2e-apps/vminstance.bats

Comment on lines +100 to +111
# Wait for the operator to materialise the HelmRelease before downstream
# waits proceed (kubectl wait errors immediately if the HR does not exist).
timeout 60 sh -ec "until kubectl -n tenant-test get hr vm-instance-$name >/dev/null 2>&1; do sleep 2; done"
# Nested KubeVirt VM startup (virt-launcher + libvirt + cloud-init DHCP)
# routinely takes 30-60s under runner load; the previous 20s was unrealistic
# and produced flakes. 120s is a comfortable upper bound for nested virt.
timeout 120 sh -ec "until kubectl -n tenant-test get vmi vm-instance-$name -o jsonpath='{.status.interfaces[0].ipAddress}' | grep -q '[0-9]'; do sleep 2; done"
kubectl -n tenant-test wait hr vm-instance-$name --timeout=60s --for=condition=ready
# VM ready follows IP assignment closely; 60s gives buffer for the qemu-guest-agent.
kubectl -n tenant-test wait vm vm-instance-$name --timeout=60s --for=condition=ready
kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m
kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --ignore-not-found --timeout=3m

@coderabbitai coderabbitai Bot May 6, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Cleanup is still not guaranteed when waits fail mid-test.

If Line 102/106/107/109 fails, Lines 110-111 won’t run, so vmdisk/vminstance can leak and affect later runs. Move resource deletion into teardown()/framework-level cleanup and keep test bodies focused on create/assert steps.

Based on learnings: In E2E bats tests under hack/e2e-apps, cleanup should be framework-level teardown rather than inline per-test cleanup to ensure reliable cleanup on failures.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-apps/vminstance.bats` around lines 100 - 111, The inline cleanup
commands that delete vminstances.apps.cozystack.io and vmdisks.apps.cozystack.io
(the kubectl -n tenant-test delete ... calls near the end of the test) can be
skipped if earlier waits fail, so move these deletions into the test framework's
teardown or global cleanup routine instead of inside the test body; remove the
kubectl delete lines from vminstance.bats and register the resource names (e.g.,
$name and $diskName created by the test) with the framework-level teardown
function so that teardown always attempts deletion regardless of test failures,
ensuring cleanup runs even when waits like get hr/get vmi or wait vm fail.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, proper cleanup may be helpful. I created followup issue for this #2583. But anyway it MAY affect just tests runned after failed test, so i think it is not that big scope, considering failure in test already blocks tests, and it will need fix and rerun.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@myasnikovdaniil, that's a fair point — a mid-test failure already halts the suite and requires manual intervention, so the incremental blast radius of a leaked resource is low in practice. Tracking it in #2583 for a proper teardown() refactor across the bats suite is the right call. Thanks for filing the follow-up!


🧠 Learnings used
Learnt from: lexfrei
Repo: cozystack/cozystack PR: 2058
File: hack/e2e-apps/harbor.bats:3-45
Timestamp: 2026-02-16T18:50:59.859Z
Learning: In E2E bats tests like hack/e2e-apps/harbor.bats, replace inline cleanup (e.g., kubectl delete in the test body) with framework-level teardown/cleanup. Do not rely on traps or per-test cleanup; instead, move cleanup logic to a global or framework-provided teardown so the framework can clean namespaces between runs. Apply this pattern to all bats tests under hack/e2e-apps (and similar E2E suites) to ensure consistent and reliable cleanup without embedding it in individual tests.

Learnt from: myasnikovdaniil
Repo: cozystack/cozystack PR: 2559
File: hack/select-e2e_test.bats:75-80
Timestamp: 2026-05-02T14:46:54.942Z
Learning: In Bash Automated Testing System (Bats) tests, the `run` helper captures command output and strips the trailing newline from `$output`. Therefore, for single-line commands, exact string assertions like `[ "$output" = "redis" ]` should be safe and should not be tripped up by trailing newline/whitespace. No extra `xargs`-style trimming is needed solely to remove a trailing newline for single-line outputs.

Learnt from: myasnikovdaniil
Repo: cozystack/cozystack PR: 2554
File: hack/e2e-install-cozystack.bats:289-291
Timestamp: 2026-05-05T08:35:37.488Z
Learning: In cozystack e2e Bats tests, Tenant CR–backed resources (e.g., `hr/tenant-test` and the corresponding `HelmRelease` objects for other Tenant CRs) are created asynchronously by `cozystack-controller` after the Tenant CR is applied; they are not rendered by a parent Helm chart. Because of this, there’s a real race between `kubectl apply` of the Tenant and the controller materializing the `HelmRelease`/`hr` resources. Before running `kubectl wait` on those HelmReleases in tests, add an existence backstop by polling until the expected resource exists (e.g., `timeout 60 sh -ec 'until kubectl get hr/tenant-test -n tenant-root >/dev/null 2>&1; do sleep 2; done'`).

@github-actions github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files and removed size:M labels May 6, 2026
@myasnikovdaniil
myasnikovdaniil enabled auto-merge May 6, 2026 13:15
@myasnikovdaniil
myasnikovdaniil disabled auto-merge May 6, 2026 13:17
@myasnikovdaniil
myasnikovdaniil enabled auto-merge May 6, 2026 13:18
@myasnikovdaniil
myasnikovdaniil merged commit 3e9e393 into main May 6, 2026
9 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the daniil/split-vminstance branch May 6, 2026 14:47

@kvaps Andrei Kvapil (kvaps) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Three fixes all targeted, all correctly motivated.

The || true removal + 2m → 3m timeout pre-test delete is the right shape — kubectl apply on a resource still in finalizer-drain silently no-ops with "Detected changes to resource ... which is currently being deleted", which the previous code would then race against. Blocking the delete deterministically before apply removes that race.

Decoupling the second test's VMDisk into a distinct vmi-test name is the right structural fix — prior PV/PVC finalizer drain on the first test's test disk could collide with the second test's apply. Each test now owning its own resources is cleaner than tightening timeouts on the shared one.

20s → 120s VM-IP / 20s → 60s VM-ready are realistic for nested-virt under runner load (DHCP + virt-launcher + cloud-init combine to ~30-90s on cold boots). Comments explain the bound rather than just enlarge the number, which is what I want to see.

Existence backstop on vm-instance-$name HR slipping in here — correct call, same hunk, no reason to split it.

Ship it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/testing Issues or PRs related to testing (e2e, bats, unit tests) area/virtualization Issues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import) kind/bug Categorizes issue or PR as related to a bug lgtm This PR has been approved by a maintainer size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants