Skip to content

Repeated messages of shared dispatcher #16802

Description

@BewareMyPower

Describe the bug
Recently I found the C++ tests BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub and BasicEndToEndTest.testpatternMultiTopicsHttpConsumerPubSub became flaky. Eventually I found it's caused by the duplicated messages.

To Reproduce

First, start a standalone with latest master.

Then, run the following Java test.

    @Test(invocationCount = 10)
    public void test() throws Exception {
        @Cleanup final PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl("http://127.0.0.1:8080").build();
        final String topicBase = "persistent://public/default/my-topic-" + System.currentTimeMillis();
        for (int i = 0; i < 3; i++) {
            admin.topics().createPartitionedTopic(topicBase + "-" + i, i + 2);
        }
        @Cleanup final PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
        final Consumer<byte[]> consumer = client.newConsumer()
                .topicsPattern(topicBase + ".*")
                .subscriptionName("sub")
                .receiverQueueSize(10)
                .subscriptionType(SubscriptionType.Shared)
                .subscribe();
        final int numMessages = 100;
        for (int i = 0; i < 3; i++) {
            final Producer<byte[]> producer = client.newProducer()
                    .topic(topicBase + "-" + i)
                    .messageRoutingMode(MessageRoutingMode.RoundRobinPartition)
                    .create();
            for (int j = 0; j < numMessages; j++) {
                producer.send(("msg-content-" + i + "-" + j).getBytes());
            }
        }
        int numReceived = 0;
        while (true) {
            final Message<byte[]> msg = consumer.receive(1, TimeUnit.SECONDS);
            if (msg == null) {
                break;
            }
            numReceived++;
        }
        Assert.assertEquals(numReceived, numMessages * 3);
    }

Sometimes it failed. (You can add more logs or debugged code to see the received messages of all partitions)

Expected behavior
The tests should always succeed.

Screenshots
The failure in my local env.

截屏2022-07-26 23 00 19

image

Additional context
Adding some logs into C++ tests could make tests more stable to pass. And the Java test looks less flaky than the C++ tests.

After applying dispatcherDispatchMessagesInSubscriptionThread=false in conf/standalone.conf, the Java test could pass even if invocationCount = 50. Changing the subscription type to Exclusive also works.

image

From the test result, we can see the bug was introduced from #16603 that we should not run sendMessagesToConsumers in another thread.

Activity

  1. eolivelli commented on Jul 27, 2022

    @eolivelli
    Contributor

    I am working on a fix

  2. eolivelli commented on Jul 27, 2022

    @eolivelli
    Contributor

    in #16603 I had to change a test that was failing in a similar way.
    but the test was missing a "acknowledge(message)" so I thought that the test was wrong.

    https://github.com/apache/pulsar/pull/16603/files#diff-33303e92b2d03115c1bde9ad3af39f9127f6a25270855fb5b5c2eb7c3db74e50R281

    I will revert the change to the test and add your new test case

  3. tisonkun commented on Aug 9, 2022

    @tisonkun
    Member

    https://github.com/apache/pulsar/runs/7739162912?check_suite_focus=true

    This can be another failure case of BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub.

  4. tisonkun commented on Aug 9, 2022

    @tisonkun
    Member
    /pulsar/pulsar-client-cpp/tests/BasicEndToEndTest.cc:2184: Failure
    Expected equality of these values:
      ResultTimeout
        Which is: TimeOut
      consumer.receive(m, 1000)
        Which is: Ok
    [  FAILED  ] BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub (1173 ms)
    [----------] 1 test from BasicEndToEndTest (1173 ms total)
    
    [----------] Global test environment tear-down
    [==========] 1 test from 1 test suite ran. (1173 ms total)
    [  PASSED  ] 0 tests.
    [  FAILED  ] 1 test, listed below:
    [  FAILED  ] BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub
    
  5. tisonkun commented on Aug 9, 2022

    @tisonkun
    Member

    It seems the exception is timeout instead of unexpected result, reported at #17008.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

type/bugThe PR fixed a bug or issue reported a bug

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions