Skip to content

fix(e2e): improve failure diagnostics - #3421

Merged
myasnikovdaniil merged 4 commits into
mainfrom
fix/e2e-diagnostics-gaps
Jul 23, 2026
Merged

myasnikovdaniil merged 4 commits into
mainfrom
fix/e2e-diagnostics-gaps

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

Improves E2E failure diagnostics tracked in #3368:

  • preserves scheduled, non-terminal NotReady pods before a pod IP is assigned
  • always captures load-balancer state while gating IP-dependent diagnostics
  • collects Talos dmesg, kubelet, and containerd logs per host node using explicit endpoint and node arguments
  • renders COSI Bucket catch output as YAML
  • makes pre-commit generation locking work from linked worktrees
  • adds regression coverage for dataplane and Talos collection

Tenant-worker Talos logs remain outside this PR because the runner does not currently have a tenant talosconfig.

Part of #3368.

Testing

  • hack/cozytest.sh hack/capture-dataplane.bats
  • hack/cozytest.sh hack/cozyreport-talos.bats
  • sh -n hack/e2e-capture-dataplane.sh
  • sh -n hack/cozyreport.sh
  • chainsaw test --no-cluster --config hack/e2e-chainsaw/.chainsaw.yaml --include-test-regex '^$' hack/e2e-chainsaw/bucket hack/e2e-chainsaw/harbor
  • pre-commit run --all-files

Screenshots

Not applicable.

Downstream repositories

Release note

fix(e2e): improve diagnostic capture for dataplane, Talos, and COSI failures

Summary by CodeRabbit

  • Bug Fixes

    • Improved dataplane diagnostics for scheduled pods that are NotReady, including cases without an assigned IP.
    • LoadBalancer checks now proceed even when no affected pods are found.
    • Standardized Talos node diagnostics collection for consistent kernel, kubelet, and containerd logs.
    • Improved reliability of generation hooks by using a git-common lockfile path for concurrent runs.
  • Tests

    • Added/expanded Bats coverage for dataplane pod filtering and Talos diagnostics argument correctness.
    • Improved end-to-end bucket test retrieval/formatting for clearer failure output.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@github-actions github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug labels Jul 22, 2026
@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 607cfc93-d0c0-4cfc-b108-0b847543b764

📥 Commits

Reviewing files that changed from the base of the PR and between b11e870 and 6a41d65.

📒 Files selected for processing (7)
  • .pre-commit-config.yaml
  • hack/capture-dataplane.bats
  • hack/cozyreport-talos.bats
  • hack/cozyreport.sh
  • hack/e2e-capture-dataplane.sh
  • hack/e2e-chainsaw/bucket/chainsaw-test.yaml
  • hack/e2e-chainsaw/harbor/chainsaw-test.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
  • hack/e2e-chainsaw/harbor/chainsaw-test.yaml
  • hack/e2e-chainsaw/bucket/chainsaw-test.yaml
  • .pre-commit-config.yaml
  • hack/cozyreport-talos.bats
  • hack/e2e-capture-dataplane.sh
  • hack/cozyreport.sh

📝 Walkthrough

Walkthrough

The PR updates dataplane capture handling for scheduled NotReady pods, centralizes Talos diagnostics collection with regression tests, changes pre-commit locking to use the Git common directory, and sets YAML output for two Chainsaw Bucket retrieval steps.

Changes

Dataplane capture handling

Layer / File(s) Summary
Affected pod filtering and capture flow
hack/e2e-capture-dataplane.sh, hack/capture-dataplane.bats
Scheduled NotReady pods are retained without podIP; IP-specific captures are conditional, LoadBalancer checks continue when no affected pods exist, and Bats tests cover the filtering and runtime behavior.

Talos diagnostics collection

Layer / File(s) Summary
Talos diagnostics helper and integration
hack/cozyreport.sh, hack/cozyreport-talos.bats
A reusable helper captures Talos dmesg and service logs, supports source-only mode, replaces inline collection, and is covered by argument-level regression tests.

Workflow support updates

Layer / File(s) Summary
Hook locking and Chainsaw retrieval formats
.pre-commit-config.yaml, hack/e2e-chainsaw/*/chainsaw-test.yaml
Generation hooks lock through the Git common directory, and Bucket retrieval steps explicitly request YAML format.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested reviewers: kvaps

Sequence Diagram(s)

sequenceDiagram
  participant kubectl
  participant pod_filter_affected
  participant dataplane_capture
  participant LoadBalancer_checks
  kubectl->>pod_filter_affected: provide pod status rows
  pod_filter_affected->>dataplane_capture: return scheduled NotReady pods
  dataplane_capture->>LoadBalancer_checks: continue when no affected pods
Loading
sequenceDiagram
  participant cozyreport.sh
  participant cozyreport_collect_talos_node
  participant talosctl
  cozyreport.sh->>cozyreport_collect_talos_node: pass config, node, and output directory
  cozyreport_collect_talos_node->>talosctl: collect dmesg and service logs
  talosctl-->>cozyreport_collect_talos_node: write diagnostic output
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately reflects the PR’s main goal of improving E2E failure diagnostics.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/e2e-diagnostics-gaps

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Checkov (3.3.8)
.pre-commit-config.yaml

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'

hack/e2e-chainsaw/bucket/chainsaw-test.yaml

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'

hack/e2e-chainsaw/harbor/chainsaw-test.yaml

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dosubot dosubot Bot added the area/ci Issues or PRs related to CI workflows, GitHub Actions, automation label Jul 22, 2026

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

LGTM with non-blocking notes

Diagnostics/CI-only change (no packages/ touched); the three new behavior tests are non-vacuous and pass under the CI dash path, the shell is portable, and the pre-commit worktree fix is behavior-preserving. Two minor items could not be verified offline and are noted below.

Caveats

  • Phase 5b/5c (upgrade + fresh-install / config-matrix): not applicable. The PR touches only hack/* diagnostics + hack/e2e-chainsaw/* test manifests + .pre-commit-config.yaml; no chart, values.yaml, migration, CRD, RBAC, PackageSource, bundle, or ApplicationDefinition. No customer upgrade or install surface is affected.

  • Shell portability (dash vs ash vs bash): verified, no defect. hack/cozyreport.sh and hack/e2e-capture-dataplane.sh are #!/bin/sh, contain no bashisms (grep for set -o pipefail/[[/arrays/local/<<</${x,,} returned only comment-text false hits), pass dash -n, and both new bats suites run green when driven through hack/cozytest.sh under /bin/sh=dash (the CI runner path). The sourcing guards (return 0 2>/dev/null in cozyreport.sh:16-18 of the diff; E2E_CAPTURE_DATAPLANE_LIB guard at hack/e2e-capture-dataplane.sh:219) behave correctly: the guard variable is a non-exported shell var, so the subprocess-invoked full-script test (capture-dataplane.bats:66-84) is not short-circuited by it.

  • cozytest.sh hook fidelity: verified, no dead code. Neither new .bats file defines setup/teardown/setup_file/teardown_file (those are silently never run by hack/cozytest.sh). The top-level talosctl() mock in hack/cozyreport-talos.bats:14-16 is correctly placed at file scope, not nested in the @test, matching the runner's bare-} block-terminator constraint documented in its comment (hack/cozyreport-talos.bats:12-13); the test passes, confirming the mock is in scope at call time.

  • Test adequacy (Phase 5d): required regression tests present and non-vacuous, verified by mutation. Reverting pod_filter_affected to require a podIP ($3!="") turns capture-dataplane.bats:45 RED; reverting cozyreport_collect_talos_node to the old -n-only + dmesg --tail=200 shape turns cozyreport-talos.bats:18 RED; re-adding the pre-fix early exit 0 on empty affected turns the LB-independence test capture-dataplane.bats:66 RED. All three mutations produced the expected failure; the fixes are protected.

  • hack/e2e-chainsaw/{bucket,harbor}/chainsaw-test.yaml format: yaml on the get collector: not offline-schema-validated (no chainsaw binary in the hermetic environment). Low risk: .chainsaw.yaml:195 already uses format: XML, so format is a known field in this tree, and the PR body's chainsaw test --no-cluster --include-test-regex '^$' run exercises manifest parsing. An operator running the chainsaw suite on a dev cluster confirms it live.

  • .pre-commit-config.yaml worktree lock fix (sh -c 'exec flock -x "$(git rev-parse --git-common-dir)/pre-commit.lock" "$@"' -- sh -c '...'): traced the argv wrapping — "$@" resolves to sh -c '<inner>' <appended-files>, so flock acquires the shared lock then execs the inner generator; appended filenames land as ignored positionals exactly as in the pre-change form. Behavior-preserving except for the intended common-dir path. Confirmed git rev-parse --git-common-dir returns .git in a normal checkout (identical to the old literal). No automated regression test accompanies this (inherent to a pre-commit entry); manual trace only.

@IvanHunters

Copy link
Copy Markdown
Collaborator

Verdict

LGTM with non-blocking notes

Diagnostics/CI-only change (no packages/ touched); the three new behavior tests are non-vacuous and pass under the CI dash path, the shell is portable, and the pre-commit worktree fix is behavior-preserving. Two minor items could not be verified offline and are noted below.

Caveats

  • Phase 5b/5c (upgrade + fresh-install / config-matrix): not applicable. The PR touches only hack/* diagnostics + hack/e2e-chainsaw/* test manifests + .pre-commit-config.yaml; no chart, values.yaml, migration, CRD, RBAC, PackageSource, bundle, or ApplicationDefinition. No customer upgrade or install surface is affected.

  • Shell portability (dash vs ash vs bash): verified, no defect. hack/cozyreport.sh and hack/e2e-capture-dataplane.sh are #!/bin/sh, contain no bashisms (grep for set -o pipefail/[[/arrays/local/<<</${x,,} returned only comment-text false hits), pass dash -n, and both new bats suites run green when driven through hack/cozytest.sh under /bin/sh=dash (the CI runner path). The sourcing guards (return 0 2>/dev/null in cozyreport.sh:16-18 of the diff; E2E_CAPTURE_DATAPLANE_LIB guard at hack/e2e-capture-dataplane.sh:219) behave correctly: the guard variable is a non-exported shell var, so the subprocess-invoked full-script test (capture-dataplane.bats:66-84) is not short-circuited by it.

  • cozytest.sh hook fidelity: verified, no dead code. Neither new .bats file defines setup/teardown/setup_file/teardown_file (those are silently never run by hack/cozytest.sh). The top-level talosctl() mock in hack/cozyreport-talos.bats:14-16 is correctly placed at file scope, not nested in the @test, matching the runner's bare-} block-terminator constraint documented in its comment (hack/cozyreport-talos.bats:12-13); the test passes, confirming the mock is in scope at call time.

  • Test adequacy (Phase 5d): required regression tests present and non-vacuous, verified by mutation. Reverting pod_filter_affected to require a podIP ($3!="") turns capture-dataplane.bats:45 RED; reverting cozyreport_collect_talos_node to the old -n-only + dmesg --tail=200 shape turns cozyreport-talos.bats:18 RED; re-adding the pre-fix early exit 0 on empty affected turns the LB-independence test capture-dataplane.bats:66 RED. All three mutations produced the expected failure; the fixes are protected.

  • hack/e2e-chainsaw/{bucket,harbor}/chainsaw-test.yaml format: yaml on the get collector: not offline-schema-validated (no chainsaw binary in the hermetic environment). Low risk: .chainsaw.yaml:195 already uses format: XML, so format is a known field in this tree, and the PR body's chainsaw test --no-cluster --include-test-regex '^$' run exercises manifest parsing. An operator running the chainsaw suite on a dev cluster confirms it live.

  • .pre-commit-config.yaml worktree lock fix (sh -c 'exec flock -x "$(git rev-parse --git-common-dir)/pre-commit.lock" "$@"' -- sh -c '...'): traced the argv wrapping — "$@" resolves to sh -c '<inner>' <appended-files>, so flock acquires the shared lock then execs the inner generator; appended filenames land as ignored positionals exactly as in the pre-change form. Behavior-preserving except for the intended common-dir path. Confirmed git rev-parse --git-common-dir returns .git in a normal checkout (identical to the old literal). No automated regression test accompanies this (inherent to a pre-commit entry); manual trace only.

@myasnikovdaniil
myasnikovdaniil merged commit 18e459a into main Jul 23, 2026
15 of 16 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the fix/e2e-diagnostics-gaps branch July 23, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci Issues or PRs related to CI workflows, GitHub Actions, automation area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants