fix(tests): support skipped tests in cozytest by immanuwell · Pull Request #3933 · cozystack/cozystack

Immanuel Tikhonov (immanuwell) · 2026-08-21T10:01:26Z

What this PR does

cozytest.sh currently treats Bats skip as a missing command. Root runs hit this in run-kubernetes-kvm-exits_test.bats and fail instead of skipping. Repro on main:

sudo env PATH="$PATH" ./hack/cozytest.sh hack/run-kubernetes-kvm-exits_test.bats 'a file that cannot be read is skipped as unread'

hack/cozytest.sh: skip: not found

This adds proper skip handling, reports the reason, and continues with the next test.
A small end to end test covers it too
Fixes #3827

Screenshots

Not a UI change.

Downstream repositories

Release note

NONE

Assisted-By: GPT-5 <[email protected]> Signed-off-by: immanuwell <[email protected]>

coderabbitai · 2026-08-21T10:02:11Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: b7aa7c1e-365d-4959-8722-acd71068da90

📥 Commits

Reviewing files that changed from the base of the PR and between 3c780ac and 6bb1821.

📒 Files selected for processing (2)

hack/cozytest-skip.bats
hack/cozytest.sh

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📝 Walkthrough

Walkthrough

cozytest.sh now supports a Bats-compatible skip function. The runner records skip status and reports skipped tests separately from successful tests. A new Bats test validates skip behavior, output, and cleanup.

Changes

Skip handling

Layer / File(s)	Summary
Skip marker and runner reporting `hack/cozytest.sh`	Adds `skip()` with an optional reason, creates a per-test marker path, and reports skipped tests separately from successful tests.
End-to-end skip validation `hack/cozytest-skip.bats`	Runs a temporary fixture and verifies successful completion, skip output, skipped-body suppression, subsequent test success, and cleanup.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 6bb18

The change makes skipped tests report their reason and continue correctly, with coverage for the behavior; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: kvaps

Sequence Diagram(s)

sequenceDiagram
  participant BatsTest
  participant TestFixture
  participant cozytest.sh
  BatsTest->>TestFixture: create skipped and passing tests
  BatsTest->>cozytest.sh: run fixture
  cozytest.sh->>TestFixture: execute test subshell
  TestFixture->>cozytest.sh: call skip with reason
  cozytest.sh->>cozytest.sh: record marker and report skipped test
  cozytest.sh->>TestFixture: execute following passing test
  cozytest.sh-->>BatsTest: return successful suite status
  BatsTest->>BatsTest: verify output and skipped-body suppression

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The PR adds skip handling and coverage, but it does not update the inaccurate API availability comment required by issue `#3827`.	Update the comment in hack/e2e-install-cozystack.bats to describe the kubectl api-resources output check.
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Out of Scope Changes check	✅ Passed	The changes are limited to cozytest skip handling and its end-to-end test, with no unrelated code changes identified.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: adding skipped-test support to cozytest.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Aleksei Sviridkin (lexfrei)

Immanuel Tikhonov (@immanuwell) NOT LGTM: the change is useful, but it needs a rebase and the commit trailer has to change.

The problem is real on main: run-kubernetes-kvm-exits_test.bats calls skip when run as root, and e2e-install-cozystack.bats calls it on clusters without the ValidatingAdmissionPolicy API. cozytest.sh defines no skip, so both fail with exit 127. #3849 moves the unit lane to real Bats, but cozytest.sh stays as the runner for the live-cluster suites and the POSIX pass, so this is not moot. Your test passes on the PR head.

Two blockers:

The commit carries Assisted-By: GPT-5 <[email protected]>. The repo accepts only Assisted-by: LLM, with no model or vendor name (docs/agents/contributing.md, AI Agent Attribution). The Commit trailers check fails on any other value.
hack/cozytest.sh has changed a lot since your base and the branch conflicts with main. Please rebase and re-run hack/cozytest-skip.bats against the new runner.

A small one: the test only runs rm -rf "$dir" on success, so a failing run leaves its temp dir behind.

Aleksei Sviridkin (lexfrei) · 2026-10-10T20:05:38Z

Immanuel Tikhonov (@immanuwell) Since #3849 the unit suites run under real Bats, which has skip built in, so the cozytest shim is no longer needed for this. Thanks for the PR. Closing.

fix(tests): support skipped tests in cozytest

6bb1821

Assisted-By: GPT-5 <[email protected]> Signed-off-by: immanuwell <[email protected]>

Immanuel Tikhonov (immanuwell) requested review from Andrei Kvapil (kvaps), Aleksei Sviridkin (lexfrei), Timofei Larkin (lllamnyp) and myasnikovdaniil as code owners August 21, 2026 10:01

github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug labels Aug 21, 2026

Aleksei Sviridkin (lexfrei) requested changes Sep 30, 2026

View reviewed changes

Aleksei Sviridkin (lexfrei) mentioned this pull request Sep 30, 2026

test: cozytest.sh stops a bats file at its first failing test #4620

Open

myasnikovdaniil mentioned this pull request Oct 2, 2026

[Tracking] Shell test architecture: vanilla BATS for unit tests, assert library, testable suite shell #3453

Open

2 of 8 tasks

Aleksei Sviridkin (lexfrei) closed this Oct 10, 2026

Navigation Menu

Repository navigation

fix(tests): support skipped tests in cozytest - #3933

fix(tests): support skipped tests in cozytest#3933
Immanuel Tikhonov (immanuwell) wants to merge 1 commit into
cozystack:mainfrom
immanuwell:fix/cozytest-skip

Immanuel Tikhonov (immanuwell) commented Aug 21, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 21, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (2 warnings)

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Uh oh!

Aleksei Sviridkin (lexfrei) commented Oct 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Immanuel Tikhonov (immanuwell) commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Screenshots

Downstream repositories

Release note

Uh oh!

coderabbitai Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (2 warnings)

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Choose a reason for hiding this comment

Uh oh!

Aleksei Sviridkin (lexfrei) commented Oct 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Immanuel Tikhonov (immanuwell) commented Aug 21, 2026 •

edited

Loading

coderabbitai Bot commented Aug 21, 2026 •

edited

Loading