[fix][broker] Fix TableViewLoadDataStoreImpl close deadlock that stalls broker shutdown - #26243

Fixes #24628

Motivation

ExtensibleLoadManagerCloseTest.testCloseAfterLoadingBundles has been flaky in CI for a long time (#24628), with several failure signatures. Analysis of a recent failure (test-report artifact of this run) shows the reported deferGetOwner TimeoutException is only fallout: one invocation earlier, broker.close() hung for 300 seconds until the TestNG method timeout killed the test thread, leaving half-closed zombie brokers that poisoned the subsequent invocation's lookups.

The hang is a deadlock in TableViewLoadDataStoreImpl that can stall broker shutdown in production:

• All methods of TableViewLoadDataStoreImpl are synchronized, and shutdown()/close() performed a blocking tableView.close() while holding the store monitor (via ExtensibleLoadManagerImpl.disableBroker → stopLoadDataReportTasks).
• The table view reader's close future can only complete via a task on the Pulsar client's pinned internal executor thread (ConsumerImpl.cleanupAtClose → failPendingReceive).
• That same executor thread can concurrently be inside a ServiceUnitStateChannel StateChangeListener calling back into the store — BrokerLoadDataReporter.handleEvent → tombstone() → removeAsync(), also synchronized — where it blocks on the store monitor.

The closing thread holds the monitor and waits for the close future; the only thread that can complete that future waits for the monitor: a permanent deadlock. The trigger window is exactly what the test creates: a broker closing while Owned events from a previously closed broker's bundle handoff are still being processed. The captured logs confirm the mechanism: the reader's Closed consumer wire-level log appears immediately, and the blocked executor thread's Restarted producer ... restartReason=object is null fires at the exact moment the 300s interrupt releases the monitor.

The same lock/executor contention also explains the other #24628 signatures (bounded 5s startTableView/startProducer restart failures and cascade failures in later invocations).

Modifications

• TableViewLoadDataStoreImpl.closeTableView()/closeProducer(): close the table view and producer with non-blocking closeAsync() and log failures, instead of blocking close() under the store monitor. No client close is awaited while holding the monitor anymore, which removes the deadlock. PulsarService closes the shared internal client later during shutdown, so in-flight asynchronous closes cannot leak.
• Added a deterministic regression test LoadDataStoreTest.testShutdownDoesNotDeadlockWithConcurrentStoreAccess that reproduces the interleaving with a stubbed table view whose close future only completes after another thread has entered a synchronized store method. It deadlocks (30s test timeout) on the previous code and passes with the fix.
• Added a package-private @VisibleForTesting setTableView accessor used by the test.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• LoadDataStoreTest.testShutdownDoesNotDeadlockWithConcurrentStoreAccess deterministically reproduces the deadlock: it fails with a 30s timeout on the previous code (test thread parked in the blocking close holding the store monitor) and passes with the fix.
• Full LoadDataStoreTest passes (7/7).
• ExtensibleLoadManagerCloseTest passes locally with all 10 invocationCount runs of testCloseAfterLoadingBundles plus testLookup.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment