[improve][broker] Allow disabling scalable topics when upgrading to 5.x - #26740

PIP: 460

Motivation

Brokers already provide enablePersistentTopics and enableNonPersistentTopics to control which topic types they support. Operators should also be able to disable scalable topics, introduced by PIP-460.

Existing deployments should be able to upgrade to Pulsar 5.x for its performance and reliability improvements while continuing to run their existing workloads. Adopting scalable topics, the V5 client API, or Oxia is a separate decision. Workloads using the existing 4.0.x feature set retain a supported downgrade path from 5.0.x to 4.0.x.

Setting scalableTopicsEnabled=false before upgrading prevents applications from creating or using scalable topics that 4.x cannot serve. Operators can validate the upgrade with their existing topics, subscriptions, and client API, then enable scalable topics when they are ready to use them. The default remains true.

The setting scalableTopicsEnabled was already added earlier to control capability advertisement and some protocol handlers, but scalable-topic resources and controllers still start when it is disabled, the admin API remains available, and existing segment topics can still load. This change makes those paths respect the setting as well. Disabling the feature retains existing scalable-topic data so it can be accessed again after re-enabling it.

Modifications

• Apply scalableTopicsEnabled=false when initializing broker services. Do not instantiate ScalableTopicResources, start ScalableTopicService, or start the V5 transaction coordinator. Do not register the /admin/v2/scalable REST resource.
• Reject scalable-topic protocol requests, including watch-close and unsubscribe commands, and prevent the separate V5 transaction setting from bypassing scalableTopicsEnabled=false. Advertise the existing scalable-topic and V5 transaction-discovery capability flags as disabled.
• Reject binary and HTTP lookups, producer/consumer attachment, and loading of topic:// and segment:// topics, including topics created before the feature was disabled.
• Fail V5 transaction discovery immediately when its capability is absent. V5 producer and consumer creation against a disabled broker fails with the broker lookup error explaining that scalable topics are disabled.
• Document the setting in the broker and standalone configuration files. The default remains true; changing it requires a broker restart. Existing scalable-topic data is retained, and ordinary topics remain available through the classic client API.

Verifying this change

• ./gradlew quickCheck passed.
• 14 targeted tests passed with retries disabled, with no skipped tests:
  • ScalableTopicBinaryAuthZTest: capability advertisement and rejection of raw protocol requests, lookups, and producer/consumer attachment when disabled.
  • V5TransactionRecoveryTest: enabled transaction recovery, plus a disable/re-enable restart test covering absent resources/services, unavailable admin endpoints, rejected lookups and loads, V5 client errors, ordinary-topic availability, and recovery of previously written scalable-topic data.
  • ClientCnxTest.testScalableTopicSupportRequiresExplicitFeatureFlag and WatchTcAssignmentsDiscoveryTest: missing/disabled capability flags and fail-fast transaction discovery.

Does this pull request potentially affect one of the following parts:

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol — reuse existing capability flags and error responses; reject scalable-topic requests when disabled. No wire-format changes.
• The REST endpoints — the scalable-topic admin resource is unavailable when disabled; scalable-topic HTTP lookups fail.
• The admin CLI options
• The metrics
• Anything that affects deployment — operators can set scalableTopicsEnabled=false before a 4.x-to-5.x migration; changing the setting requires a broker restart.