[improve][broker] Implement PIP-430 Pulsar Broker cache improvements - #24623

Codecov Report

❌ Patch coverage is 70.53388% with 287 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.18%. Comparing base (829df71) to head (bbca447).
⚠️ Report is 49 commits behind head on master.

@@             Coverage Diff              @@
##             master   #24623      +/-   ##
============================================
- Coverage     74.26%   74.18%   -0.09%     
- Complexity    33213    33377     +164     
============================================
  Files          1885     1893       +8     
  Lines        146953   147722     +769     
  Branches      16928    17080     +152     
============================================
+ Hits         109136   109582     +446     
- Misses        29116    29395     +279     
- Partials       8701     8745      +44     

Here's the JMH results for the latest changes. The new getNumberOfCursorsAtSamePositionOrBefore behavior required for cacheEvictionByExpectedReadCount in ActiveManagedCursorContainerImpl is not causing an overhead compared to the previous ManagedCursorContainer tracking for slowest reader position.

Here's a visualization of the results:
JMH Visualizer: ActiveManagedCursorContainerBenchmark

text results

One interesting detail is that ActiveManagedCursorContainerImpl's algorithm would be faster than ManagedCursorContainer's algorithm for tracking the slowest cursor, assuming that most cursors are moving forward (which is the case).

The recently added "experimental" test case BrokerEntryCacheMultiBrokerTest.testTailingReadsRollingRestart shows the real value of PIP-430 changes. It demonstrates a rolling restart where a producer is producing to a topic and there are 10 connected consumers with unique subscriptions. During the 30 second test period the producer produces as fast as it can and the consumers consume as fast as they can. There are 2 brokers in the cluster and a broker is restarted 3 times during the test period.

The with PIP-430 defaults, Cache hits 98.56%, Cache misses 1.44% :

2025-08-18T19:39:44,894 - INFO  - [main:BrokerEntryCacheMultiBrokerTest] - Produced 21655 and Consumed 206425 messages (across 10 consumers with unique subscriptions) in total. Number of BK reads 74 with 2972 entries. Cache hits 98.56%, Cache misses 1.44%, Number of restarts 3

with cacheEvictionByExpectedReadCount=false, Cache hits 81.66%, Cache misses 18.34%

2025-08-18T19:41:17,119 - INFO  - [main:BrokerEntryCacheMultiBrokerTest] - Produced 20767 and Consumed 197466 messages (across 10 consumers with unique subscriptions) in total. Number of BK reads 848 with 36220 entries. Cache hits 81.66%, Cache misses 18.34%, Number of restarts 3

with cacheEvictionByExpectedReadCount=false and managedLedgerCacheEvictionExtendTTLOfRecentlyAccessed=false, Cache hits 72.97%, Cache misses 27.03%

2025-08-18T19:42:34,214 - INFO  - [main:BrokerEntryCacheMultiBrokerTest] - Produced 20888 and Consumed 198727 messages (across 10 consumers with unique subscriptions) in total. Number of BK reads 1309 with 53708 entries. Cache hits 72.97%, Cache misses 27.03%, Number of restarts 3

The results vary quite a lot and these are values from single runs. However, it demonstrates a significant improvement in reducing the amount of cache misses in rolling restarts when PIP-430 defaults are used.

While doing some validation, I noticed that Pulsar has a long time issue that consumers aren't able to keep with producers even when there are available resources unless dispatcherMaxReadBatchSize (and possibly also dispatcherMaxReadSizeBytes) are tuned. The default value for dispatcherMaxReadBatchSize is 100 which seems to be too small. Increasing it to 1000 could help in increasing cache hits and performance.
It's also useful to set preciseDispatcherFlowControl=true while increasing dispatcherMaxReadBatchSize so that unnecessary reads are avoided.

I created #24695 about this.