Repository navigation
[Bug]: Log pipes open even when no writer will attach, leading to deadlock during initdb #11436
Copy link
Copy link
Open
Labels
Description
Is there an existing issue already for this bug?
- I have searched for an existing issue, and could not find anything. I believe this is a new bug.
I have read the troubleshooting guide
- I have read the troubleshooting guide and I think this is a new bug.
I am running a supported version of CloudNativePG
- I have read the troubleshooting guide and I think this is a new bug.
Contact Details
Version
1.30.0
What version of Kubernetes are you using?
1.36
What is your Kubernetes environment?
Self-managed: kind (evaluation)
How did you install the operator?
Helm
What happened?
A bug in PR #10043 causes a leaked OS thread during initdb. This does not have a major impact on Linux systems where CNPG generally runs, but it's still a bug.
In that PR, jsonPipe := logpipe.NewJSONLineLogPipe(...) was added to WithActiveInstance() in pkg/management/postgres/instance.go.
1. `jsonPipe.Start()` creates `/controller/log/postgres.json` as a named pipe (FIFO) and calls `fileutils.OpenFileAsync(ctx, p.fileName, os.O_RDONLY, 0o600)`.
2. In Linux/POSIX, calling `open(fifo, O_RDONLY)` blocks in kernel `wait_for_partner` indefinitely until an external writer connects.
3. During `initdb` bootstrap, Postgres only opens `/controller/log/postgres.csv` (`log_destination: csvlog`). No Barman or external tools are running yet, so nothing ever opens `/controller/log/postgres.json` for write.
4. When `WithActiveInstance` finishes and calls `ctxCancel()`, `OpenFileAsync` returns on `ctx.Done()`, but the background goroutine running `os.OpenFile` **remains permanently stuck in the blocking kernel syscall on the OS thread**.
5. When `WithActiveInstance` runs `defer jsonPipe.GetExitedCondition().Wait()`, or when the Go runtime attempts process exit/thread synchronization, the blocked thread causes a deadlock.
6. Writing any byte to `/controller/log/postgres.json` from the host instantly unblocks the kernel `open()` and allows `initdb` to complete immediately.
As a consequence, under amd64 emulation using Rosetta on macOS, the initdb processes occasionally (about 5% of the time, in my testing) deadlock. Actually running the database is fine, because during that state the postgres.json named pipe does get open, unblocking the thread.
I have a suggested fix, which I think is unambiguous, and will submit a PR shortly.
Cluster resource
Any Cluster objectRelevant log output
{"level":"info","ts":"2026-08-22T14:14:08.290455636Z","msg":"Configuring new PostgreSQL instance","logging_pod":"grafana-db-1-initdb"}
{"level":"info","ts":"2026-08-22T14:14:08.729123974Z","msg":"Executing post-init SQL instructions","logging_pod":"grafana-db-1-initdb"}
{"level":"info","ts":"2026-08-22T14:14:08.76115752Z","msg":"Executing post-init template SQL instructions","logging_pod":"grafana-db-1-initdb"}
{"level":"info","ts":"2026-08-22T14:14:09.271973743Z","msg":"executing Application instructions","logging_pod":"grafana-db-1-initdb"}
<process hangs here indefinitely>
#### Kernel process call stacks (/proc/<pid>/task/*/stack):
=== TID 8362 (goroutine opening postgres.json) ===
[<0>] wait_for_partner+0x84/0x100
[<0>] fifo_open+0x1b8/0x2a8
[<0>] vfs_open+0x118/0x4a8
[<0>] path_openat+0x41c/0x1db0
[<0>] do_file_open+0x9c/0x160
[<0>] __arm64_sys_openat+0x120/0x1f8Code of Conduct
- I agree to follow this project's Code of Conduct
Reactions are currently unavailable
Activity
fix(logpipe): do not block threads on log FIFOs
[-][Bug]:[/-][+][Bug]: Log pipes open even when no writer will attach, leading to deadlock during initdb[/+]