Describe the bug
Recently I found the C++ tests BasicEndToEndTest.testPatternMultiTopicsConsumerPubSub and BasicEndToEndTest.testpatternMultiTopicsHttpConsumerPubSub became flaky. Eventually I found it's caused by the duplicated messages.
To Reproduce
First, start a standalone with latest master.
Then, run the following Java test.
@Test(invocationCount = 10)
public void test() throws Exception {
@Cleanup final PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl("http://127.0.0.1:8080").build();
final String topicBase = "persistent://public/default/my-topic-" + System.currentTimeMillis();
for (int i = 0; i < 3; i++) {
admin.topics().createPartitionedTopic(topicBase + "-" + i, i + 2);
}
@Cleanup final PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
final Consumer<byte[]> consumer = client.newConsumer()
.topicsPattern(topicBase + ".*")
.subscriptionName("sub")
.receiverQueueSize(10)
.subscriptionType(SubscriptionType.Shared)
.subscribe();
final int numMessages = 100;
for (int i = 0; i < 3; i++) {
final Producer<byte[]> producer = client.newProducer()
.topic(topicBase + "-" + i)
.messageRoutingMode(MessageRoutingMode.RoundRobinPartition)
.create();
for (int j = 0; j < numMessages; j++) {
producer.send(("msg-content-" + i + "-" + j).getBytes());
}
}
int numReceived = 0;
while (true) {
final Message<byte[]> msg = consumer.receive(1, TimeUnit.SECONDS);
if (msg == null) {
break;
}
numReceived++;
}
Assert.assertEquals(numReceived, numMessages * 3);
}
Sometimes it failed. (You can add more logs or debugged code to see the received messages of all partitions)
Expected behavior
The tests should always succeed.
Screenshots
The failure in my local env.


Additional context
Adding some logs into C++ tests could make tests more stable to pass. And the Java test looks less flaky than the C++ tests.
After applying dispatcherDispatchMessagesInSubscriptionThread=false in conf/standalone.conf, the Java test could pass even if invocationCount = 50. Changing the subscription type to Exclusive also works.

From the test result, we can see the bug was introduced from #16603 that we should not run sendMessagesToConsumers in another thread.
Describe the bug
Recently I found the C++ tests
BasicEndToEndTest.testPatternMultiTopicsConsumerPubSubandBasicEndToEndTest.testpatternMultiTopicsHttpConsumerPubSubbecame flaky. Eventually I found it's caused by the duplicated messages.To Reproduce
First, start a standalone with latest master.
Then, run the following Java test.
Sometimes it failed. (You can add more logs or debugged code to see the received messages of all partitions)
Expected behavior
The tests should always succeed.
Screenshots
The failure in my local env.
Additional context
Adding some logs into C++ tests could make tests more stable to pass. And the Java test looks less flaky than the C++ tests.
After applying
dispatcherDispatchMessagesInSubscriptionThread=falseinconf/standalone.conf, the Java test could pass even ifinvocationCount = 50. Changing the subscription type toExclusivealso works.From the test result, we can see the bug was introduced from #16603 that we should not run
sendMessagesToConsumersin another thread.