[fix][broker] Fix assignment and ownership cleanup races in the extensible load manager - #26520

Motivation

The extensible load manager can lose ownership cleanup updates to concurrent assignments, and can accept late assignments after shutdown cleanup has started. This CI job exposed these races through background topic-policy initialization assigning the __change_events bundle:

• Cleanup reads Assigning(version 1) and publishes an override at version 2. If the assignment's Owned(version 2) update wins, the conflict resolver rejects the cleanup override. Cleanup previously only waited, leaving the bundle owned by the broker being cleaned up and consuming its five-second wait.
• A lookup that finishes broker selection after the channel is disabled can still publish a new assignment. That assignment can arrive after cleanup has scanned ownership, leaving later lookups pointing to a stopped broker.

The cleanup retry and polling changes apply both to graceful shutdown and to the leader's cleanup of inactive brokers during failure recovery. Topic policies exposed the races, but other concurrent bundle lookups can trigger them as well. The changes are confined to the extensible load manager.

The reported test initially exceeded its five-second shutdown assertion, then its retry failed with HTTP 409 because it reused the same topic name.

Modifications

• Track publication of every new bundle assignment and serialize its admission with channel disable using a short critical section. Reject new assignments once disabled, and wait for accepted assignment writes before shutdown scans ownership. Storage calls and future completion run outside the admission lock; waiting uses the existing metadata-operation timeout.
• Retry cleanup overrides for non-system bundles that remain owned by the broker being cleaned up, using their current state and retaining version conflict checks. This shared cleanup path handles both graceful shutdown and inactive-broker recovery; graceful shutdown continues to use graceful transfer when another broker is available.
• Use the intended cleanup retry interval instead of half the overall cleanup timeout in both cleanup paths.
• Exercise both shutdown scenarios with topic policies enabled and disabled, retaining the five-second assertions. Use unique lookup topic names and clean up brokers after every invocation, including failures.
• Add regression tests that control the assignment-write ordering and exercise both ownership-store implementations, with and without a replacement broker. Use package-private testing accessors instead of reflection to inject the delayed table view.

Verifying this change

• Make sure that the change passes the CI checks.

Both regression tests were checked against the previous production code and failed:

• testCleanupRetriesConcurrentAssignment: the bundle remained Owned after the same-version cleanup override was rejected.
• testCleanupDrainsAssignmentsAndRejectsNewOnes: the disabled channel accepted a late assignment.

With the fix and -PtestRetryCount=0 -PtestFailFast=false:

• Full ServiceUnitStateChannelTest: 66 tests passed across system-topic and metadata-store ownership tables.
• ExtensibleLoadManagerCloseTest: 40 invocations passed, covering both topic-policy settings and both shutdown scenarios. The temporary tenfold repetition on testLookup was removed afterward; the pre-existing repetition on testCloseAfterLoadingBundles remains.
• Final targeted regression and lookup tests passed.
• ./gradlew quickCheck passed.

Does this pull request potentially affect one of the following parts:

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Normal bundle assignments now track their publication and briefly acquire the same lock used by channel disable. Shutdown waits for already-accepted assignment writes before ownership cleanup; no storage operation or future completion runs under that lock.