Add streaming dispatcher. by MarvinCai · Pull Request #9056 · apache/pulsar

MarvinCai · 2020-12-25T01:22:09Z

Related to #3804

Motivation

Trying to streamline the dispatcher's read requests to manager ledger instead of micro batch.

Modifications

Created a StreamingEntryReader that can streamline read request to managed ledger.
Created StreamingDispatcher interface with necessary method to interact with StreamingEntryReader.
Created PersistentStreamingDispatcherSingleActive/MultipleConsumer that make use of StreamingEntryReader to read entries from managed ledger.
Add config to use streaming dispatcher.

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change added tests and can be verified as follows:
Add unit tests.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API: no
The schema: no
The default values of configurations: no
The wire protocol: no
The rest endpoints: no
The admin cli options: no
Anything that affects deployment: no

Documentation

Does this pull request introduce a new feature? yes

Add debug print for ut.

MarvinCai · 2020-12-25T23:59:26Z

/pulsarbot run-failure-checks

MarvinCai · 2020-12-26T00:58:19Z

/pulsarbot run-failure-checks

sijie · 2020-12-29T17:34:17Z

        }
    }

+    void notifyWaitingEntryCallBack() {


Suggested change

void notifyWaitingEntryCallBack() {

void notifyWaitingEntryCallBacks() {

sijie · 2020-12-29T17:34:37Z

+        category = CATEGORY_SERVER,
+        doc = "Whether to use streaming read dispatcher."
+    )
+    private boolean streamingDispatch = false;


Can you mark this feature as a preview feature?

updated the doc.

sijie · 2020-12-29T17:34:50Z


+    @FieldContext(
+        category = CATEGORY_SERVER,
+        doc = "Whether to use streaming read dispatcher."


It would be great to add more documentation about this feature.

sijie · 2020-12-29T17:44:18Z

+public class PersistentStreamingDispatcherMultipleConsumers extends PersistentDispatcherMultipleConsumers
+    implements StreamingDispatcher {
+
+    private StreamingEntryReader streamingEntryReader = new StreamingEntryReader((ManagedCursorImpl) cursor,


…ispatcher

MarvinCai · 2021-01-08T07:08:24Z

/pulsarbot run-failure-checks

MarvinCai · 2021-01-08T14:13:36Z

/pulsarbot run-failure-checks

sijie

@MarvinCai Overall looks good to me. Can you add more tests to the newly introduced classes?

sijie · 2021-01-14T21:32:23Z

+ */
+@Slf4j
+@RequiredArgsConstructor
+public class StreamingEntryReader implements AsyncCallbacks.ReadEntryCallback, WaitingEntryCallBack {


@MarvinCai Can you write a test case for this class?

There's StreamingEntryReaderTests class, it's folded by github.

sijie · 2021-01-14T21:33:30Z

@codelipenghui @hangc0276 Can you review this pull request?

sijie · 2021-01-20T22:37:36Z

@codelipenghui Can you review this?

sijie · 2021-02-01T02:47:15Z

@MarvinCai Can you rebase this pull request?

@codelipenghui Can you review it?

…ispatcher

MarvinCai · 2021-02-01T19:55:00Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-01T21:20:37Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-01T21:46:37Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-01T22:36:48Z

@sijie Rebased to newer version of master and resolved conflict.
Also I saw original Dispatcher logic has some changes related to Transaction, will have separate pr to update streaming Dispatcher if needed to be compatible with transaction.

MarvinCai · 2021-02-01T23:02:39Z

/pulsarbot run-failure-checks

Related to #3804 Trying to streamline the dispatcher's read requests to manager ledger instead of micro batch. Created a StreamingEntryReader that can streamline read request to managed ledger. Created StreamingDispatcher interface with necessary method to interact with StreamingEntryReader. Created PersistentStreamingDispatcherSingleActive/MultipleConsumer that make use of StreamingEntryReader to read entries from managed ledger. Add config to use streaming dispatcher. (cherry picked from commit 8cfaf48)

MarvinCai added 2 commits December 24, 2020 17:14

Add streaming dispatcher.

41d44cf

Update situation reach end of topic.

b1bf5aa

Add debug print for ut.

sijie assigned MarvinCai Dec 27, 2020

sijie requested review from codelipenghui, jiazhai and sijie December 27, 2020 05:34

sijie added this to the 2.8.0 milestone Dec 27, 2020

sijie added area/broker component/storage labels Dec 27, 2020

sijie reviewed Dec 30, 2020

View reviewed changes

sijie added the doc-required Your PR changes impact docs and you will update later. label Dec 30, 2020

MarvinCai added 4 commits January 5, 2021 16:25

Merge remote-tracking branch 'apache-pulsar/master' into streamline-d…

242e61d

…ispatcher

Address comments.

474ed8b

Try remove unnecessary synchronization in StreamingEntryReader.

2f02621

Try run tests for streaming dispatcher as separate mvn command.

c296481

MarvinCai requested a review from sijie January 8, 2021 17:49

sijie requested changes Jan 14, 2021

View reviewed changes

sijie requested review from 315157973, hangc0276 and merlimat January 14, 2021 21:33

MarvinCai requested a review from sijie January 16, 2021 17:50

sijie approved these changes Jan 18, 2021

View reviewed changes

Merge remote-tracking branch 'apache-pulsar/master' into streamline-d…

884a49c

…ispatcher

Fix import for proto generated type.

081a569

sijie approved these changes Feb 2, 2021

View reviewed changes

codelipenghui approved these changes Feb 2, 2021

View reviewed changes

codelipenghui merged commit 8cfaf48 into apache:master Feb 2, 2021

MarvinCai mentioned this pull request Feb 14, 2021

[WIP][LedgerHandler] Try add read ahead capability to ledgerhandler. apache/bookkeeper#2494

Closed

lhotari mentioned this pull request Mar 11, 2021

Flaky-test: StreamingEntryReaderTests.testCanReadEntryFromMLedgerSizeExceededLimit #9885

Closed

Anonymitaet added doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. and removed doc-required Your PR changes impact docs and you will update later. labels Apr 9, 2021

merlimat mentioned this pull request Oct 1, 2026

[fix][broker] Fix Exclusive/Failover dispatch rate limiting using counts from another dispatch #26810

Merged

11 tasks

	void notifyWaitingEntryCallBack() {
	void notifyWaitingEntryCallBacks() {

Conversation

MarvinCai commented Dec 25, 2020

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

MarvinCai commented Dec 25, 2020

Uh oh!

MarvinCai commented Dec 26, 2020

Uh oh!

sijie Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

MarvinCai Jan 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sijie Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

MarvinCai Jan 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sijie Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

sijie Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

MarvinCai Jan 6, 2021

Choose a reason for hiding this comment

Uh oh!

MarvinCai commented Jan 8, 2021

Uh oh!

MarvinCai commented Jan 8, 2021

Uh oh!

sijie left a comment

Choose a reason for hiding this comment

Uh oh!

sijie Jan 14, 2021

Choose a reason for hiding this comment

Uh oh!

MarvinCai Jan 14, 2021

Choose a reason for hiding this comment

Uh oh!

sijie commented Jan 14, 2021

Uh oh!

sijie commented Jan 20, 2021

Uh oh!

sijie commented Feb 1, 2021

Uh oh!

MarvinCai commented Feb 1, 2021

Uh oh!

MarvinCai commented Feb 1, 2021

Uh oh!

MarvinCai commented Feb 1, 2021

Uh oh!

MarvinCai commented Feb 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarvinCai commented Feb 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MarvinCai Jan 6, 2021 •

edited

Loading

MarvinCai Jan 6, 2021 •

edited

Loading

MarvinCai commented Feb 1, 2021 •

edited

Loading