Skip to content

[fix][broker] Do not start classic geo-replicators on scalable-topic segment topics - #26691

Merged
merlimat merged 1 commit into
apache:masterfrom
merlimat:mmerli/no-classic-replicator-on-segment-topics
Sep 25, 2026
Merged

merlimat merged 1 commit into
apache:masterfrom
merlimat:mmerli/no-classic-replicator-on-segment-topics

Conversation

@merlimat

Copy link
Copy Markdown
Contributor

Motivation

A segment:// topic backing a scalable topic (PIP-460) lives in an ordinary namespace, so when that namespace has more than one replication cluster the topic's replication check treats it like any other persistent topic: on load it opens a durable pulsar.repl.<remote> cursor and starts a GeoPersistentReplicator toward the same-named segment:// topic on the remote cluster.

That is never a valid target. Segment DAGs are independent per cluster (PIP-460, "Geo-Replication"), so the remote cluster either has no such segment or has one with a different hash range, owned by its own ScalableTopicController. Geo-replication for scalable topics needs a mechanism of its own and is not designed yet.

In practice the replicator never connects: its remote-topic pre-check asks the local admin for /admin/v2/segment/.../partitions, which has no REST resource, so producer creation fails and is retried forever (before #26681 it looped without backoff). The replicator stays in Starting, and the never-advancing cursor retains the segment's data and counts toward the backlog quota.

Modifications

  • PersistentTopic.startReplicator(): return early for segment topics, before the cursor is opened, with a debug log. This is the single method that creates both the cursor and the replicator, so the guard covers the initial check on load, policy updates and the unfence path alike. It is deliberately not an early return in internalCheckReplication(), so that removeTopicIfLocalClusterNotAllowed() keeps running for segments as for any other topic.
  • Shadow replication needs no guard: shadowTopics only comes from topic-level policies keyed by the segment's own name, which no admin path can set.
  • New ScalableTopicSegmentReplicationTest (two-cluster OneWayReplicatorTestBase): creates a scalable topic in a namespace replicated to two clusters, asserts that the namespace's replication clusters do resolve for each segment, and that after checkReplication() the segment has neither a replicator nor a pulsar.repl.* cursor. Fails on the unpatched code with Expecting empty but was: ["r2"].

Should land together with or after #26679: once no replicator is present on a segment, the ReplicatedSubscriptionsController would otherwise start writing snapshot markers into it, which that PR prevents.

…segment topics

A segment:// topic in a namespace with more than one replication cluster
started a GeoPersistentReplicator toward the same-named segment on the
remote cluster and opened a durable pulsar.repl.<remote> cursor. Segment
DAGs are independent per cluster (PIP-460), so that target never exists:
the replicator never connects and the cursor retains the segment's data.

Skip the replicator in PersistentTopic.startReplicator() for segment
topics, before the cursor is opened, and cover it with a two-cluster
test.

@lhotari lhotari left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@merlimat
merlimat merged commit bc8134c into apache:master Sep 25, 2026
44 checks passed
@lhotari lhotari added this to the 5.0.0 milestone Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants