Repository navigation
check-rd-presets.sh reports a present preset as missing when grep short-circuits under pipefail #4178
Description
Activity
Second observation, and it settles that the failure is a race rather than something wrong with one file.
The same run, 34346366626, reported two different victims at two different times. At 11:40:31 it was
FAIL: packages/system/opensearch-rd/cozyrds/opensearch.yaml resourcesPreset enum missing: u1.medium. At 13:10:11 it wasFAIL: packages/system/rabbitmq-rd/cozyrds/rabbitmq.yaml resourcesPreset enum missing: s1.xlarge. Both carry theprintf: write error: Broken pipeline immediately before the FAIL, and in both cases the named preset is present in the named file.So the pair that gets reported is not a property of any chart. Which preset loses depends on when grep happens to exit relative to printf's write, which is why re-running turns one red file into a different red file rather than into green.
Practical consequence for anyone triaging: a PR touching neither the named
-rdpackage nor the script can still fail this check, and the same PR can fail it on a different package on the next attempt. Matching the failure against what the PR changed will not clear it, and re-running is not a fix — it just reshuffles which name appears.Seen again today on #4171, whose branch touches neither
rabbitmq-rdnorhack/check-rd-presets.sh, and where the redE2E Testsis the downstream cascade described above rather than a suite result.Two clarifications, so nobody re-derives them.
The line attribution is right, and the other pipe in that loop is ruled out. The same function has a second writer into a pipe at lines 27-32,
enums=$(printf '%s' "$schema" | jq -r '...' 2>/dev/null || true), and it is a much more plausible EPIPE site by size, since it writes the whole schema blob rather than the few hundred bytes of enum values. It is not the one. Bash attributes an error inside a multi-line command to that command's last line, so an EPIPE there would be reported as line 32. The log says line 39, which is theprintf | grep -Fqxin the preset loop. Note also that the2>/dev/nullon 32 applies to jq, not to printf, so a printf warning from that site would still have reached the log with a different line number.What does not reproduce, and where. Running the same pipeline by hand on macOS does not produce the failure: several hundred iterations at both the real input size and smaller, matching the first line so grep exits as early as possible, gives zero false failures. That is expected rather than contradictory. The enum list is a few kilobytes, one
write()well inside the 64K pipe buffer, so printf finishes before grep can close the read end and there is no window. Forcing the window open needs an artificially large writer, which demonstrates the mechanism but says nothing about the real size.So the honest summary for anyone picking this up: the mechanism is confirmed from the runner log, where
printf: write error: Broken pipesits immediately above the FAIL, and it is not reproducible by hand at the real input size on a developer machine. Expect green when you try it locally. That is not evidence against the bug, and the fix does not depend on reproducing it — feeding grep from a here-string removes the pipe, and with it the whole class.Duplicate of #4153, which described the same SIGPIPE race four days earlier. I have carried the two things this report added — that the named file and preset differ between attempts of the same run, and that the line-39 attribution rules out the larger jq pipe in the same loop — into that issue. Closing here.
Still live on main, and it reddened two PRs today in a batch of fifteen.
hack/check-rd-presets.shstill hasprintf '%s\n' "$enums" | grep -Fqx -- "$want"underset -euo pipefail.Neither fix landed: #4191 is open, #4194 is a draft untouched since 10 September. This issue and #4193 are both closed, so nothing tracks it.
Today's failures, a different file and a different preset each time, which is what the race looks like:
FAIL: packages/system/opensearch-rd/cozyrds/opensearch.yaml resourcesPreset enum missing: c1.medium FAIL: packages/system/kubernetes-rd/cozyrds/kubernetes.yaml resourcesPreset enum missing: t1.mediumBoth values are in those enums on main, I checked.
The cost is the one described above:
Unit & controller testsis inneedsfor the job that reportsE2E Tests, so a false failure turns a required context red and the pull request cannot merge until someone re-runs it.Reopening.
Re-ran the two failed jobs on the same commits, and the check reported a different set of presets missing each time. Five false reports across four files in two attempts:
attempt 2 opensearch.yaml c1.medium attempt 3 foundationdb.yaml large attempt 3 rabbitmq.yaml m1.xlarge attempt 2 kubernetes.yaml t1.medium attempt 3 redis.yaml s1.xlargeSame tree both times, nothing pushed between attempts. All five values are in those enums. That rules out stale RD files and leaves only the race.
Practically this means a re-run is not a workaround: the check fired on both attempts, just on different files. It is blocking a batch of pull requests right now, and #4191 is the fix sitting open.
Retracting both comments above. The fix is on main:
c2e7c86ff, 10 September, which replaced the pipeline with a here-string (grep -Fqx -- "$want" <<<"$enums"). I read the script out of a stale worktree and reported that as the state of main.What I actually ran into: the runs I was looking at were created on 9 September, a day before that commit, and a re-run is pinned to the original merge commit, so it re-executed the pre-fix tree. A different preset reported on each attempt is the old race doing exactly what this issue describes, not evidence that it survived the fix.
Those pull requests need a fresh run on a current base, not a re-run. Re-closing.
hack/check-rd-presets.sh:39reports a preset as missing when it is present, and the run goes red on a PR that touched nothing related.The line is
printf '%s\n' "$enums" | grep -Fqx -- "$want", and the script runs underset -euo pipefailfrom line 7.grep -qexits as soon as it matches,printfthen takes a broken pipe,pipefailmakes the whole pipeline non-zero, and the leading!inverts that into "not found".Seen on run 34346366626, job "Unit & controller tests", on a branch touching neither opensearch nor any RD:
That is the signature to look for: a broken-pipe warning immediately before the FAIL, and exactly one preset named out of the 47.
It does not stop at one red check.
finalizegates on the checks job, so E2E is skipped and the run is reported red without a single suite having run. Anyone reading that PR sees a red E2E that never executed.Worth saying that it does not reproduce on macOS. I ran the same pipeline 500 times locally and got zero false failures, because a different printf implementation and a different scheduler make the race impossible there. The log above is from the Linux runner, which is the environment that matters.
Fix shape: feed grep from a here-string,
grep -Fqx -- "$want" <<<"$enums", so there is no pipeline to fail. Droppingpipefailaround that one line works too but leaves the trap in place for the next person.