[improve][broker] Make AvgShedder the default load shedding and placement strategy - #26609

PIP: PIP-364 introduced AvgShedder; this PR changes the default. The previous default change (ThresholdShedder) went through PIP-122, so this may need a PIP as well — opened for discussion and CI first.

Motivation

The modular load manager (ModularLoadManagerImpl) still defaults to ThresholdShedder for shedding and LeastLongTermMessageRate for placement. The two score brokers differently — resource usage vs. message rate — so a bundle shed from a hot broker is routinely placed on another hot broker and shed again on the next cycle. ThresholdShedder's EMA (loadBalancerHistoryResourcePercentage=0.9) is updated once per shedding cycle on the leader, so its scores lag actual load by minutes and it keeps unloading a broker after the load has already moved; and it never sheds towards an idle broker unless lowerBoundarySheddingEnabled is set, which is the shape a cluster is in after every rolling restart.

AvgShedder (PIP-364, available since 3.0.6 / 3.2.4 / 3.3.1) addresses all three: it scores by resource usage, pairs the highest and the lowest loaded brokers and moves load between them, pre-plans the destination of every bundle it unloads so placement cannot undo the decision, and only acts after the gap has exceeded a threshold for several consecutive checks (8 for a 15-point gap, 2 for a 40-point gap), which keeps short traffic spikes from triggering moves.

Modifications

• ServiceConfiguration, conf/broker.conf, conf/standalone.conf:
  • loadBalancerLoadSheddingStrategy and loadBalancerLoadPlacementStrategy default to org.apache.pulsar.broker.loadbalance.impl.AvgShedder.
  • maxUnloadPercentage defaults to 0.5 (was 0.2), so AvgShedder equalizes a broker pair in one cycle instead of moving a fifth of the gap per cycle.
  • loadBalancerDistributeBundlesEvenlyEnabled defaults to false (was true) and is now listed in broker.conf for the first time: the per-namespace bundle-count filter runs before the placement strategy and can discard the destination AvgShedder planned for a bundle. System-namespace bundles are still distributed evenly.
  • standalone.conf gains the strategy keys and the AvgShedder settings it did not list; the field docs describe the old and the new default.
• ModularLoadManagerImpl: the strategy pairing is extracted into createLoadBalanceStrategies() (package-private, LoadBalanceStrategies record). A configuration that sets a classic shedder (ThresholdShedder, UniformLoadShedder, OverloadShedder) while leaving the placement strategy at the new AvgShedder default previously threw IllegalArgumentException at start-up; it now keeps the configured shedder and falls back to LeastLongTermMessageRate placement with a warning, so existing configurations that only set loadBalancerLoadSheddingStrategy keep exactly the behavior they had. AvgShedder shedding paired with a different placement strategy logs a warning that its planned destinations are not honored (this combination was already accepted before).
• New ModularLoadManagerImplStrategyTest.

The extensible load manager is not affected: UnloadScheduler already falls back to TransferShedder (with an error log) for any configured class that is not a NamespaceUnloadStrategy, exactly as with the previous ThresholdShedder default.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• ModularLoadManagerImplStrategyTest covers the new defaults (shedding and placement are the same AvgShedder instance), the fallback for a classic shedder, an explicit classic pairing, and AvgShedder shedding with a different placement strategy.
• ModularLoadManagerImplTest (24 tests) configures its brokers with OverloadShedder only and therefore exercises the fallback path end to end.
• AvgShedderTest, ThresholdShedderTest, UniformLoadShedderTest, AntiAffinityNamespaceGroupTest, NamespaceServiceTest, BrokerServiceLookupTest, BundleSplitterTaskTest and BrokerBookieIsolationTest pass locally.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Default values changed: loadBalancerLoadSheddingStrategy (ThresholdShedder → AvgShedder), loadBalancerLoadPlacementStrategy (LeastLongTermMessageRate → AvgShedder), maxUnloadPercentage (0.2 → 0.5), loadBalancerDistributeBundlesEvenlyEnabled (true → false). Deployments that set only loadBalancerLoadSheddingStrategy keep their previous behavior through the placement fallback. maxUnloadPercentage=0.5 also applies to UniformLoadShedder deployments that did not set it. The pulsar-site load-balancing pages will need the new defaults.