Repository navigation
test(e2e): capture LINSTOR satellite ErrorReports in cozyreport - #3209
Merged
myasnikovdaniil merged 1 commit intoJul 6, 2026
Merged
Conversation
github-actions
Bot
added
the
area/testing
Contributor
📝 WalkthroughWalkthroughAdds LINSTOR ErrorReport collection to the cozyreport script: exports a controller-side error-reports index, packages controller ErrorReport-*.log files into a validated tarball, and creates per-satellite-node tarballs while cleaning up empty or invalid bundles. ChangesLINSTOR ErrorReport Collection
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
the
size/M
IvanHunters
force-pushed
the
test/cozyreport-linstor-satellite-errorreports
branch
2 times, most recently
from
July 5, 2026 15:19
b8c9f48 to
9cc4fa1
Compare
IvanHunters
marked this pull request as ready for review
July 5, 2026 15:27
Contributor
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request improves the diagnostic capabilities of the LINSTOR e2e testing infrastructure by ensuring that detailed error reports are preserved. By extending the existing collection script to bundle and export log files from ephemeral pod storage, it enables better post-mortem analysis of storage-related failures that were previously obscured by limited log visibility. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the hack/cozyreport.sh script to collect LINSTOR controller and satellite error reports, ensuring critical diagnostic logs are captured before ephemeral environments are torn down. The review feedback suggests several shell scripting improvements, including using POSIX shell built-ins to check for the existence of error logs before running tar (preventing empty or invalid archives), replacing external basename subprocesses with shell parameter expansion, and avoiding the reassignment of the shared DIR variable.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Comment on lines
+280
to
+283
| kubectl -n cozy-linstor exec deploy/linstor-controller --container=linstor-controller -- sh -c ' | ||
| cd /var/log/linstor-controller 2>/dev/null || exit 0 | ||
| tar -czf - ErrorReport-*.log 2>/dev/null || true | ||
| ' > "$DIR/controller-error-reports.tgz" 2>/dev/null || true |
Contributor
There was a problem hiding this comment.
If there are no files matching ErrorReport-*.log, running tar will fail and might produce an empty or invalid archive that then has to be cleaned up. We can use standard POSIX shell built-ins to check if any matching files exist before running tar. This avoids executing tar unnecessarily and prevents creating empty files.
Suggested change
| kubectl -n cozy-linstor exec deploy/linstor-controller --container=linstor-controller -- sh -c ' | |
| cd /var/log/linstor-controller 2>/dev/null || exit 0 | |
| tar -czf - ErrorReport-*.log 2>/dev/null || true | |
| ' > "$DIR/controller-error-reports.tgz" 2>/dev/null || true | |
| kubectl -n cozy-linstor exec deploy/linstor-controller --container=linstor-controller -- sh -c ' | |
| cd /var/log/linstor-controller 2>/dev/null || exit 0 | |
| set -- ErrorReport-*.log | |
| [ -e "$1" ] || exit 0 | |
| tar -czf - "$@" 2>/dev/null || true | |
| ' > "$DIR/controller-error-reports.tgz" 2>/dev/null || true |
Comment on lines
+298
to
+324
| DIR=$REPORT_DIR/linstor/error-reports | ||
| mkdir -p "$DIR" | ||
| for pod in $(kubectl -n cozy-linstor get pods -l app.kubernetes.io/component=linstor-satellite -o name 2>/dev/null); do | ||
| # Read the node name directly off the pod spec so the tarball key | ||
| # is stable across piraeus DaemonSet regenerations. Fallback to the | ||
| # pod name if .spec.nodeName is not readable for any reason (last | ||
| # resort; collisions between DS pod generations are theoretically | ||
| # possible but the bundle would still land in a distinct file). | ||
| node=$(kubectl -n cozy-linstor get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null) | ||
| [ -z "$node" ] && node=$(basename "$pod") | ||
| # Tar the ErrorReport-*.log files into a per-satellite bundle so a | ||
| # burst of retry-loop reports (dozens per incident) does not explode | ||
| # the artefact tree, and a missing directory or empty set never | ||
| # fails the whole cozyreport run. | ||
| kubectl -n cozy-linstor exec "$pod" --container=linstor-satellite -- sh -c ' | ||
| cd /var/log/linstor-satellite 2>/dev/null || exit 0 | ||
| tar -czf - ErrorReport-*.log 2>/dev/null || true | ||
| ' > "$DIR/$node.tgz" 2>/dev/null || true | ||
| # Drop the bundle on failure so an operator does not have to unpack | ||
| # a stub. A satellite with zero ErrorReports is the healthy case and | ||
| # should leave nothing in the tree. | ||
| if [ ! -s "$DIR/$node.tgz" ] || ! tar -tzf "$DIR/$node.tgz" >/dev/null 2>&1; then | ||
| rm -f "$DIR/$node.tgz" | ||
| fi | ||
| done | ||
| # Drop the empty parent directory when no satellite had reports. | ||
| rmdir "$DIR" 2>/dev/null || true |
Contributor
There was a problem hiding this comment.
We can optimize this block in three ways:
- Avoid reassigning the shared
DIRvariable: This prevents potential bugs if more commands are added to thelinstorblock later. Use a local/distinct variable likelocal_err_dirinstead. - Use POSIX shell parameter expansion: Use
${pod##*/}instead of$(basename "$pod")to avoid spawning an external subprocess inside the loop. - Use standard POSIX shell built-ins: Use
set -- ErrorReport-*.log; [ -e "$1" ]to check if any matching files exist before runningtar, avoiding unnecessary execution and empty archive creation.
local_err_dir="$DIR/error-reports"
mkdir -p "$local_err_dir"
for pod in $(kubectl -n cozy-linstor get pods -l app.kubernetes.io/component=linstor-satellite -o name 2>/dev/null); do
# Read the node name directly off the pod spec so the tarball key
# is stable across piraeus DaemonSet regenerations. Fallback to the
# pod name if .spec.nodeName is not readable for any reason (last
# resort; collisions between DS pod generations are theoretically
# possible but the bundle would still land in a distinct file).
node=$(kubectl -n cozy-linstor get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
[ -z "$node" ] && node=${pod##*/}
# Tar the ErrorReport-*.log files into a per-satellite bundle so a
# burst of retry-loop reports (dozens per incident) does not explode
# the artefact tree, and a missing directory or empty set never
# fails the whole cozyreport run.
kubectl -n cozy-linstor exec "$pod" --container=linstor-satellite -- sh -c '
cd /var/log/linstor-satellite 2>/dev/null || exit 0
set -- ErrorReport-*.log
[ -e "$1" ] || exit 0
tar -czf - "$@" 2>/dev/null || true
' > "$local_err_dir/$node.tgz" 2>/dev/null || true
# Drop the bundle on failure so an operator does not have to unpack
# a stub. A satellite with zero ErrorReports is the healthy case and
# should leave nothing in the tree.
if [ ! -s "$local_err_dir/$node.tgz" ] || ! tar -tzf "$local_err_dir/$node.tgz" >/dev/null 2>&1; then
rm -f "$local_err_dir/$node.tgz"
fi
done
# Drop the empty parent directory when no satellite had reports.
rmdir "$local_err_dir" 2>/dev/null || true![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/cozyreport.sh`:
- Around line 280-286: The archive validation in the controller error report
block still keeps empty .tgz files when ErrorReport-*.log matches nothing;
update the check after the kubectl/tar pipeline to verify the archive has at
least one entry instead of only checking file size or tar readability. Use the
existing "$DIR/controller-error-reports.tgz" cleanup path, and apply the same
entry-count-based validation to the satellite bundle logic below so both blocks
discard zero-entry archives.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: d90a634e-5c68-4ab1-8a99-c7dab76f5eb9
📒 Files selected for processing (1)
hack/cozyreport.sh
The linstor-Satellite pod stdout log references its error reports
only by ID:
[DeviceManager] ERROR LINSTOR/Satellite/... SYSTEM - Failed to
create zfsvolume [Report number 6A4A394E-E4EA0-000000]
The body of that report (which carries the actual `zfs create -V …`
stderr, the DRBD adjust output, the storage-driver stack trace, and
the exit code) lives at `/var/log/linstor-satellite/ErrorReport-*.log`
on the satellite pod. crust-gather only captures pod stdout, so the
report body stays on the pod's ephemeral filesystem and is lost when
the e2e sandbox is torn down at end-of-job.
Consequence for post-mortem: a run that failed on a
`ResourceExhausted` / `Not enough available nodes` CreateVolume retry
loop shows only the driver-level symptom in cozyreport. Root cause
attribution (out-of-space, dataset name conflict, ZFS module error,
kernel signal) is not distinguishable without the report body, and
without the sandbox is unrecoverable.
Extend the linstor module in `hack/cozyreport.sh` to tar the
`ErrorReport-*.log` set off every `linstor-satellite` pod into
`linstor/error-reports/<node>.tgz`. Guarded so a satellite with zero
reports (the healthy case) leaves nothing in the tree, and a
transient exec / tar failure never fails the surrounding cozyreport
run.
Verified against a live dev cluster: label selector
`app.kubernetes.io/component=linstor-satellite` matches all three
satellite pods, and the tar bundle for one active satellite comes
out at ~60 KiB. No product-side change.
Signed-off-by: Ivan Okhotnikov <[email protected]>
IvanHunters
force-pushed
the
test/cozyreport-linstor-satellite-errorreports
branch
from
July 5, 2026 18:50
9cc4fa1 to
d5098d9
Compare
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (2)
hack/cozyreport.sh (2)
275-290: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicated tar-and-prune logic between controller and satellite bundles.
The "tar to file, then drop if empty/invalid" block is repeated almost verbatim for the controller bundle and each satellite bundle. Extracting it into a small shell function (taking the exec target, container, remote dir, and destination path) would remove the duplication and keep future edits (e.g. the entry-count check) in one place.
♻️ Example helper extraction
+bundle_error_reports() { + # $1=namespace $2=pod/deploy-ref $3=container $4=remote-log-dir $5=dest-tgz + kubectl -n "$1" exec "$2" --container="$3" -- sh -c " + cd $4 2>/dev/null || exit 0 + tar -czf - ErrorReport-*.log 2>/dev/null || true + " > "$5" 2>/dev/null || true + if [ ! -s "$5" ] || [ -z "$(tar -tzf "$5" 2>/dev/null | head -n 1)" ]; then + rm -f "$5" + fi +}Also applies to: 292-327
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/cozyreport.sh` around lines 275 - 290, The controller bundle repeats the same tar-to-file and prune-if-empty logic used for the satellite bundles, so extract that shared flow into a small shell helper to avoid duplication. Introduce a reusable function in cozyreport.sh that takes the kubectl exec target, container name, remote log directory, and output archive path, then move the existing controller-side ErrorReport and satellite archive creation into calls to that helper. Keep the current empty-archive/valid-member check inside the helper so future changes only need to be made once.
280-283: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAll exec/tar stderr is discarded, hiding real failures behind "no reports".
Both bundling subshells redirect
stderrto/dev/nullat every stage, so a genuinekubectl execconnectivity failure (RBAC, pod terminating, container not found) is indistinguishable from a satellite that legitimately has zero ErrorReports — the only symptom is a missing.tgz. Consider capturing exec/tar stderr to a small sidecar log (e.g.$node.err) only when non-empty, so operators can tell "no reports" apart from "collection failed."Also applies to: 316-319
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/cozyreport.sh` around lines 280 - 283, The error report bundling in the kubectl exec/tar flow is swallowing all stderr, making real collection failures look like “no reports.” Update the two exec/tar subshells in cozyreport.sh to preserve stderr output for the kubectl exec and tar steps, writing any non-empty error details to a small per-node sidecar log such as $node.err instead of /dev/null. Keep the current stdout archive behavior, but ensure failures in the controller and satellite report collection paths can be distinguished from legitimately empty ErrorReport sets.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@hack/cozyreport.sh`:
- Around line 275-290: The controller bundle repeats the same tar-to-file and
prune-if-empty logic used for the satellite bundles, so extract that shared flow
into a small shell helper to avoid duplication. Introduce a reusable function in
cozyreport.sh that takes the kubectl exec target, container name, remote log
directory, and output archive path, then move the existing controller-side
ErrorReport and satellite archive creation into calls to that helper. Keep the
current empty-archive/valid-member check inside the helper so future changes
only need to be made once.
- Around line 280-283: The error report bundling in the kubectl exec/tar flow is
swallowing all stderr, making real collection failures look like “no reports.”
Update the two exec/tar subshells in cozyreport.sh to preserve stderr output for
the kubectl exec and tar steps, writing any non-empty error details to a small
per-node sidecar log such as $node.err instead of /dev/null. Keep the current
stdout archive behavior, but ensure failures in the controller and satellite
report collection paths can be distinguished from legitimately empty ErrorReport
sets.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: e50bab59-4483-42f7-9e88-6f77b826ec97
📒 Files selected for processing (1)
hack/cozyreport.sh
myasnikovdaniil
approved these changes
Jul 6, 2026
myasnikovdaniil
added a commit
that referenced
this pull request
Jul 6, 2026
…tes tests (#3210) ## What this PR does Adds a companion `cozy_wait_linstor_pool_free` helper to `hack/e2e-apps/run-kubernetes.sh` and calls it from `cozy_cleanup` right after `cozy_wait_tenant_drained`. The helper polls LINSTOR for the smallest ZFS pool `FreeCapacity` across every satellite and blocks until it reaches a per-satellite threshold (default 90 GiB), bounded to 300 s with a loud warning on timeout. ### Root cause (proven) `kubernetes-previous.bats` intermittently failed with `zfs create -V 20976048KB data/pvc-... returned with exitcode 1: cannot create '...': out of space` on a satellite that measured **132 GiB free 35 minutes later** — so this is not an absolute capacity issue but a timing race. Root cause is captured in the cozyreport artefact of run 28751310913 (PR #3044), satellite ErrorReport `6A4AADFD-349B2-000000`. The previous tenant Kubernetes test's `cozy_wait_tenant_drained` gate returned as soon as its PVCs disappeared from the API, but the satellite-side ZFS `zvol destroy` for the just-deleted replicated (autoPlace=3, DRBD) 20 GiB worker disks lagged tens of seconds behind that signal as DRBD adjusted, unref counts drained, and ZFS batch-destroyed the datasets. The next tenant test's `CreateVolume` landed inside that window, LINSTOR-CSI retried autoplace against the still-being-torn-down previous placement (`Do not place with resource: pvc-<same-uid>` implicit anti-collocation, no CSI bug), and worker Machine bringup stretched past the MHC `nodeStartupTimeout` budget with no cluster-side fault. The proximate observation (`zfs create out of space`) surfaced only because of PR #3209 which teaches `hack/cozyreport.sh` to bundle satellite ErrorReports into the artefact. Without that, the root cause was ambiguous between several plausible LINSTOR internal races and I nearly opened a wrong upstream issue. ### Threshold rationale Two 20 GiB replicated worker targets = 60 GiB per satellite (autoPlace=3 places one replica per node) plus two 21 GiB CDI scratch PVCs (worst case both landing on the same node via the `local` storageClass) = ~82 GiB per-satellite peak footprint. 90 GiB default threshold covers that with margin. ### Scope `hack/e2e-apps/run-kubernetes.sh` only. E2E-scope, no product or chart change. Threshold and timeout are function args with sensible defaults; caller can override. ### Verified `jq` expression against a live LINSTOR 1.33.2 cluster: ``` $ kubectl -n cozy-linstor exec deploy/linstor-controller -- sh -c 'linstor --machine-readable sp l 2>/dev/null | jq -r "first | .[] | select((.provider_kind | test(\"^ZFS\")) and .free_capacity != null) | \"\(.free_capacity):\(.node_name)\"" | sort -n' 3088864:talos-5jt-iq3 42483036:talos-te2-p24 197991872:talos-6cn-cgg ``` Matches `linstor sp l` output exactly (2.95 GiB / 40.51 GiB / 188.82 GiB). ZFS-only filter excludes DISKLESS `Long.MAX_VALUE` sentinel; `and .free_capacity != null` guard prevents OFFLINE satellite pool objects (whose `free_capacity` is optional per REST schema) from ranking as `null` in the sort and hanging the loop. Contract mirrors `cozy_wait_tenant_drained`: bounded deadline, `|| true` at the call site, loud timeout warning that includes the smallest observed pool and its satellite. Timeout can never wedge the suite worse than the status quo. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved cleanup reliability by waiting for storage capacity to be released before finishing teardown, reducing cases where resources appear deleted but storage is still unavailable. * Made teardown more resilient with a configurable timeout to avoid premature completion. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
pull Bot
pushed a commit
to medampudi/cozystack
that referenced
this pull request
Jul 6, 2026
…tes tests kubernetes-previous.bats intermittently fails with `zfs create -V ... returned with exitcode 1: cannot create '...': out of space` on a satellite even though the LINSTOR pool measured 30+ minutes later shows 100+ GiB free per satellite. Root cause proven from a satellite ErrorReport captured in the cozyreport artefact (PR cozystack#3044 run 28751310913, report id 6A4AADFD-349B2-000000): the previous tenant Kubernetes test's cozy_wait_tenant_drained gate returned as soon as its PVCs disappeared from the API, but the satellite-side ZFS `zvol destroy` for the just-deleted replicated (autoPlace=3, DRBD) 20 GiB worker disks lagged tens of seconds behind that signal as DRBD adjusted, unref counts drained, and ZFS batch-destroyed the datasets. The next tenant test's CreateVolume then landed inside that window against a pool that had not yet reclaimed the space, LINSTOR-CSI retried autoplace against its own still-being-torn-down previous placement, and worker Machine bringup stretched past the MHC nodeStartupTimeout budget with no cluster-side fault. Add a companion `cozy_wait_linstor_pool_free` helper that polls the LINSTOR controller for the smallest FreeCapacity across every ZFS satellite pool and blocks until it reaches a per-satellite threshold (default 90 GiB — covers the ~82 GiB worst-case footprint of two 20 GiB replicated worker targets across all three satellites plus two 21 GiB CDI scratch PVCs landing on the same node via the local storageClass). Called from cozy_cleanup immediately after cozy_wait_tenant_drained. Bounded and best-effort: 300 s timeout, `|| true` at the call site, warning printed with the observed pool state if the threshold is not reached in budget so the next test does not misdiagnose the pressure. The threshold is a floor for the concrete failure mode observed in the E2E suite; the surrounding kubernetes-latest -> kubernetes-previous handoff is the only run in the current suite that immediately follows a heavy tenant teardown with another tenant provisioning, so this gate is the natural place for the check. E2E-scope only, no product or chart change. The proximate observation (`zfs create out of space`) surfaced because of PR cozystack#3209 which taught hack/cozyreport.sh to bundle satellite ErrorReports; without that artefact the root cause was ambiguous between multiple plausible LINSTOR internal races. Ships alongside that PR. Signed-off-by: Ivan Okhotnikov <[email protected]>
myasnikovdaniil
deleted the
test/cozyreport-linstor-satellite-errorreports
branch
July 6, 2026 05:00
IvanHunters
added a commit
that referenced
this pull request
Jul 22, 2026
Follows #3209 (LINSTOR ErrorReports) with the second half of the "cozyreport is blind to the host-networking path" gap surfaced by the PR #3044 vlselect flake investigation (run 28772259934): `vlselect-generic-*` on one satellite hit `dial tcp <podIP>:9471: connect: connection refused` on every livenessProbe for 49s and got CrashLoopBackOff, while the sibling replica on another satellite — same pod spec, same probe config, same image — ran clean. The cozyreport artefact carried the CiliumEndpoint `state: ready` line, the pod-side listener log confirming the HTTP server was bound at 0.0.0.0:9471 within 5 ms of start, and the kube-ovn CNI ADD trace completing for the pod IP well before the first probe — every one of which says the path SHOULD have worked. Attribution of the drop to a specific host-side cause (stale conntrack from a previous pod that shared the IP, a Cilium policy drop that only Hubble sees, or an OVS-flow race between the CNI ADD and the iptables NAT chain) was not possible from the artefact because none of the host-side networking state is captured. Talos-side kubelet logs come out as one-line stubs (`error constructing client: failed to determine endpoints` — a separate cozyreport bug), so kubelet-view evidence is unavailable too. Add a `net-diag` section that runs after the sandbox-host block and copies four artefacts per satellite node: - `<node>-iptables.txt` — `iptables-save` from `kube-ovn-cni` (all tables in one blob so grepping across chains does not need a per-table unpack step). - `<node>-conntrack.txt` — `conntrack -L` first 5000 lines, from the same pod; the head cap keeps the file small on nodes with heavy tenant traffic and preserves the failure-window entries on the first-in-first-out flush semantics of the kernel table. - `<node>-hubble-flows.txt` — `hubble observe --last 2000 --output=compact` from `cilium-agent`. Hubble flow log carries the verdict (FORWARDED / DROPPED / ERROR) with reason field per flow, which is the only source that tells apart a Cilium policy drop from a datapath race. - `<node>-cilium-endpoints.txt` — `cilium-dbg endpoint list` for the per-endpoint policy state (identity, enforcement, controllers, status) that resolves "was the drop a policy decision" once hubble flags one. Every collection call uses a DaemonSet pod that already exists on the node with hostNetwork and the required binary preinstalled: `iptables-save` + `conntrack` live in `kube-ovn-cni`'s cni-server container; `hubble observe` + `cilium-dbg` live in the `cilium-agent` container. Both label selectors were confirmed against a live production cluster; both tool paths were confirmed present in the current piraeus / cilium image tags. Every capture is guarded with `|| true` and stderr redirection so a broken pod / missing tool / transient exec failure never fails the surrounding cozyreport run, and the parent directory is pruned when neither CNI is present on the cluster (e.g. a cozyreport captured against a non-cozystack cluster). Shipped as diagnostic-only test infra; no product-side change, no schema change, no chart change. Signed-off-by: Ivan Okhotnikov <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Extends
hack/cozyreport.shlinstor module to copy/var/log/linstor-satellite/ErrorReport-*.logoff everylinstor-satellitepod intolinstor/error-reports/<node>.tgzin the cozyreport archive.Why
The linstor-Satellite pod stdout log references its error reports only by ID:
The report body (actual
zfs create -V …stderr, DRBD adjust output, storage-driver stack trace, exit code) lives on the satellite pod's ephemeral filesystem at/var/log/linstor-satellite/ErrorReport-*.log. crust-gather only captures pod stdout, so the report body is lost when the e2e sandbox is torn down at end-of-job.Consequence for post-mortem of
ResourceExhausted/Not enough available nodesCreateVolume retry loops (see e.g. PR #3044 CI history): cozyreport shows only the driver-level symptom. Root cause distinction (out-of-space vs dataset name conflict vs ZFS module error vs kernel signal) is not possible from the artefact alone, and the sandbox is unrecoverable by that point.Scope
hack/cozyreport.shonly, +32 lines under the existinglinstor moduleblock. Guarded so:Verified on a live cluster
app.kubernetes.io/component=linstor-satellitematches all 3 satellite pods on the dev cluster used for verification.No product-side change, no dependency on capture happening during a specific test phase; runs as part of the existing end-of-job cozyreport collection.
Release note
Summary by CodeRabbit