Repository navigation
[fix][ml] Fail in-flight adds when a managed ledger is terminated - #26678
Merged
merlimat merged 1 commit intoSep 22, 2026
Merged
Conversation
Terminating a managed ledger closes the current BookKeeper ledger, and the close errors out the adds that are still in flight. Those adds were handed to ledgerClosed(), which had no branch for the Terminated state and returned without touching them, so they were never completed nor failed. In the broker this leaks PersistentTopic.pendingWriteOps. Any publish that reaches the terminated ledger fences the topic, and the topic is only unfenced once the pending writes drain to zero, so it stayed fenced forever and rejected every subscribe with "Topic is temporarily unavailable". Scalable topics seal segments by terminating them under producer load, so a stream consumer that has to drain the sealed segment could never attach and stopped making progress. Fail the pending adds with ManagedLedgerTerminatedException when the ledger is closed in the Terminated state, as already done for the Closed and Fenced states.
Merged
10 tasks
nodece
approved these changes
Sep 22, 2026
dao-jun
approved these changes
Sep 22, 2026
lhotari
approved these changes
Sep 22, 2026
lhotari
left a comment
Member
There was a problem hiding this comment.
LGTM. Thanks for fixing this termination race. The terminal-state branch now completes pending adds with the expected exception, and the managed-ledger and broker regression coverage verifies both callback completion and topic unfencing.
Member
|
for cherry-picking, #25240 is required before this one |
3 of 4 tasks
Radiancebobo
pushed a commit
to Radiancebobo/pulsar
that referenced
this pull request
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Terminating a managed ledger closes the current BookKeeper ledger, and that close errors out the adds that are still in flight (
LedgerHandle.doAsyncCloseInternalfails every un-acked pending add withLedgerClosedException).OpAddEntry.handleAddFailurehands those adds toManagedLedgerImpl.ledgerClosed(), which had no branch for theTerminatedstate and returned without touching them, so they were never completed nor failed.In the broker this leaks
PersistentTopic.pendingWriteOps. Any publish that reaches the terminated ledger fails withManagedLedgerTerminatedException, which fences the topic, and the topic is only unfenced once the pending writes drain to zero. With the leaked ops that never happens, so the topic stayed fenced forever and every subscribe was rejected withTopic is temporarily unavailable(topicFencingTimeoutSecondsis0by default, so nothing force-closes it either).Scalable topics seal segments by terminating them, routinely and under producer load, so this was hit easily: a v5
StreamConsumerthat must drain a sealed segment before its children are handed out could never subscribe to the fenced segment and stopped making progress. The broker log shows repeatedAttempting to subscribe to a fenced topicfor the sealedsegment://...topic.Modifications
ManagedLedgerImpl.ledgerClosed(): when the ledger is closed in theTerminatedstate, fail the pending adds withManagedLedgerTerminatedException, as is already done for theClosedandFencedstates. No new ledger will ever be created to retry them.Verifying this change
This change added tests and can be verified as follows:
ManagedLedgerBkTest.testTerminateFailsInFlightAdds(real bookies): suspends the bookie so an add stays in flight, terminates the ledger, and asserts the add fails withManagedLedgerTerminatedException. Before the fix the add callback never fires.BrokerBkEnsemblesTest.testConsumerCanSubscribeAfterTerminateWithInFlightPublish: delays the bookie response so a publish is in flight whenPersistentTopic.terminate()runs, sends one more publish to the terminated topic, and asserts the pending write count drains and a consumer can subscribe. Before the fix the count stays at 1 and the subscribe fails withTopic is temporarily unavailable.Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes