Skip to content

[fix] [client] Fix memory leak when publishing encountered a corner case error - #23738

Merged
poorbarcode merged 20 commits into
apache:masterfrom
poorbarcode:fix/publish_mem_leak
Dec 20, 2024
Merged

poorbarcode merged 20 commits into
apache:masterfrom
poorbarcode:fix/publish_mem_leak

Conversation

@poorbarcode

@poorbarcode poorbarcode commented Dec 16, 2024 •

Copy link
Copy Markdown
Contributor

Motivation

Issue 1: memory leak if get errors when publishing
Conditions:

  • Send queue is full
  • Or reaches the limitation named max message size
  • Or publishes after closing the producer
  • Or encounters an error when calling ProducerInterceptor. eligible
  • see testSendQueueIsFull, testSendMessageSizeExceeded, testSendAfterClosedProducer, and testInterceptorError

Modifications

  • Fix issues

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@poorbarcode poorbarcode added type/bug The PR fixed a bug or issue reported a bug release/3.0.9 release/3.3.4 release/4.0.2 labels Dec 16, 2024
@poorbarcode poorbarcode added this to the 4.1.0 milestone Dec 16, 2024
@poorbarcode poorbarcode self-assigned this Dec 16, 2024
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Dec 16, 2024
@poorbarcode

Copy link
Copy Markdown
Contributor Author

/pulsarbot rerun-failure-checks

@poorbarcode poorbarcode changed the title [fix] [client] Fix memory leak and publish stuck when publishing [fix] [client] Fix memory leak when publishing encountered a corner case error Dec 17, 2024
Comment thread pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java Outdated
@codecov-commenter

codecov-commenter commented Dec 17, 2024 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.11765% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 74.31%. Comparing base (bbc6224) to head (9f99fd2).
⚠️ Report is 1281 commits behind head on master.

Files with missing lines Patch % Lines
...va/org/apache/pulsar/client/impl/ProducerImpl.java 90.90% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #23738      +/-   ##
============================================
+ Coverage     73.57%   74.31%   +0.73%     
+ Complexity    32624     2374   -30250     
============================================
  Files          1877     1838      -39     
  Lines        139502   143053    +3551     
  Branches      15299    16232     +933     
============================================
+ Hits         102638   106304    +3666     
+ Misses        28908    28376     -532     
- Partials       7956     8373     +417     
Flag Coverage Δ
inttests 26.75% <23.52%> (+2.17%) ⬆️
systests 23.71% <23.52%> (-0.62%) ⬇️
unittests 73.66% <94.11%> (+0.82%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
.../pulsar/client/impl/BatchMessageContainerImpl.java 85.79% <100.00%> (+4.89%) ⬆️
...pache/pulsar/client/impl/ProducerInterceptors.java 72.50% <100.00%> (ø)
...va/org/apache/pulsar/client/impl/ProducerImpl.java 84.00% <90.90%> (+0.40%) ⬆️

... and 1007 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java Outdated
@poorbarcode
poorbarcode requested review from BewareMyPower and removed request for BewareMyPower December 19, 2024 07:24
@poorbarcode
poorbarcode merged commit ab1b5c0 into apache:master Dec 20, 2024
@lhotari

lhotari commented Dec 20, 2024

Copy link
Copy Markdown
Member

@poorbarcode @BewareMyPower Please review #23761, that addresses a long time issue where the completable futures of pending messages weren't completed when close was called. It's also a case which could result into a resource leak situation.

lhotari pushed a commit that referenced this pull request Dec 20, 2024
…ase error (#23738)

Co-authored-by: Yunze Xu <[email protected]>
(cherry picked from commit ab1b5c0)
lhotari pushed a commit that referenced this pull request Dec 20, 2024
…ase error (#23738)

Co-authored-by: Yunze Xu <[email protected]>
(cherry picked from commit ab1b5c0)
lhotari pushed a commit that referenced this pull request Dec 20, 2024
…ase error (#23738)

Co-authored-by: Yunze Xu <[email protected]>
(cherry picked from commit ab1b5c0)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 26, 2024
…ase error (apache#23738)

Co-authored-by: Yunze Xu <[email protected]>
(cherry picked from commit ab1b5c0)
(cherry picked from commit 7915b66)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 26, 2024
…ase error (apache#23738)

Co-authored-by: Yunze Xu <[email protected]>
(cherry picked from commit ab1b5c0)
(cherry picked from commit 7915b66)
hanmz pushed a commit to hanmz/pulsar that referenced this pull request Feb 12, 2025
nodece added a commit to nodece/pulsar that referenced this pull request Sep 14, 2026
…ls it

Two review findings on apache#26455, both queue/buffer paths left stranded by
a failure:

- processOpSendMsg's catch (from apache#23738) released only the permit and
  the memory when the body threw - realistically when the connection's
  event loop rejected the write task while shutting down. The op stayed
  in pendingMessages unrecycled, its cmd leaked both references (the
  op's own and the one retained for a write that never got queued), and
  a chunked op's ChunkedMessageCtx claim leaked with it. The catch now
  takes the op back out of the queue (OpSendMsgQueue gained a
  remove(OpSendMsg) that keeps the message-count accounting; the
  iterator's remove already did), drops the orphaned write reference,
  and releases the op's own command via releaseOpCmdAndRecycle, whose
  inline fallback covers the rejecting event loop.

- In the resend recovery loop, a rebuilt command that exceeds the max
  message size took the isMessageSizeExceeded path, which already
  released the accounting and completed the callback - but the bare
  continue left the op in pendingMessages with its fresh cmd, so it sat
  until the send timeout failed it and released the accounting a second
  time. The continue now removes the op from the iteration and releases
  its command and the op itself.

Regression test drives the real processOpSendMsg with an event loop
that rejects every task and asserts the queue is empty, the accounting
follows, and the cmd reaches refCnt 0 (fails with "expected [0] but
found [1]" queue residue on the previous code).

Assisted-by: Claude Code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants