Repository navigation
[Bug]: Cluster stuck in "Setting up primary" when instance creation Job fails #11042
Description
Is there an existing issue already for this bug?
- I have searched for an existing issue, and could not find anything. I believe this is a new bug.
I have read the troubleshooting guide
- I have read the troubleshooting guide and I think this is a new bug.
I am running a supported version of CloudNativePG
- I have read the troubleshooting guide and I think this is a new bug.
Contact Details
Version
1.30.0-rc1
What version of Kubernetes are you using?
1.36
What is your Kubernetes environment?
Self-managed: kind (evaluation)
How did you install the operator?
YAML manifest
What happened?
When restoring or bootstrapping a cluster, if the instance creation Job exhausts its backoff limit and terminates with a BackoffLimitExceeded failure, the cluster remains stuck in the "Setting up primary" phase indefinitely.
The operator counts the failed Job as still running, so reconciliation keeps waiting on it forever without surfacing the failure. Operators have no indication that the restore has failed — the cluster simply never progresses.
Expected Behavior
When an instance creation Job reaches a terminal JobFailed condition, the cluster should transition to the unrecoverable phase with a clear reason pointing to the failed Job and its logs. Jobs that are still legitimately retrying within their backoff limit should be left untouched — only terminal failures should trigger this transition.
Cluster resource
Relevant log output
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
dosubot commentedon Jun 25, 2026
This issue is already being addressed in PR #11035, which introduces explicit detection of terminal
JobFailedconditions (includingBackoffLimitExceeded) and transitions the cluster toPhaseUnrecoverablewith a clear failure reason pointing to the failed Job.The root cause is that the reconciliation loop's
runningJobNames()guard historically relied onStatus.Succeededcounts to determine Job completion — a terminal failure was indistinguishable from a still-running Job, causing the cluster to poll indefinitely at 5-second intervals [1].The fix adds a
JobHasFailed(job)utility inpkg/utils/job_conditions.gothat checks for thebatchv1.JobFailedcondition, and afailedJobNames()method to collect permanently failed Jobs. Failed Jobs are preserved for investigation rather than cleaned up [2].There's also a related feature request in #10733 proposing a
Provisioningcondition oncluster.status.conditionsto surface provisioning failures (quota exceeded, admission denials, scheduling constraints) that occur before the Job even creates a Pod — complementary to this fix which handles failures after the Job has started running.To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
fix: report failed instance creation jobs instead of waiting forever (#…
fix: report failed instance creation jobs instead of waiting forever (#…
fix: report failed instance creation jobs instead of waiting forever (#…
fix: report failed instance creation jobs instead of waiting forever (#…