[improve][pip] PIP-441: Add Broker-Level Metrics for Skipped Non-Recoverable Data - #24716

Summary

This PIP proposes adding two broker-level metrics to provide essential visibility into non-recoverable data skipping when autoSkipNonRecoverableData is enabled.

New Metrics

• pulsar_broker_non_recoverable_ledgers_skipped_total - Count of entire ledgers skipped
• pulsar_broker_non_recoverable_entries_skipped_total - Count of individual entries skipped

Motivation

Currently, there is no visibility when Pulsar skips non-recoverable data during disaster recovery scenarios. This creates operational blind spots where:

• Operators cannot be alerted when data loss occurs
• No audit trail exists for compliance requirements
• Cannot distinguish between healthy systems and those silently losing data
• Unable to determine if issues are systematic (ledger-level) or localized (entry-level)

Implementation

• Adds counters to BrokerOperabilityMetrics class
• Integrates with existing skip methods:
  • ManagedLedgerImpl.skipNonRecoverableLedger() → ledger metric
  • ManagedCursorImpl.skipNonRecoverableEntries() → entry metric
• Supports both Prometheus and OpenTelemetry formats
• Broker-level approach avoids high-cardinality burden on metrics system

Operational Benefits

• Alerting: Get notified immediately when data loss occurs
• SLA Monitoring: Track data durability metrics over time
• Root Cause Analysis: Compare metrics to understand failure patterns
• Investigation Workflow: Use metrics for alerting, then check broker logs for specific topic details

Design Rationale

• Broker-level vs Topic-level: Avoids metrics system burden while maintaining essential visibility
• Two separate metrics: Provides granular insight into different types of data corruption
• Log-based investigation: Balances alerting capability with detailed forensics

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete