Default TCP sysctl values are incompatible with DRBD usage #13074

Problem

Talos default TCP sysctl values cause issues when running DRBD (used by LINSTOR/Piraeus for replicated storage). Specifically:

• net.ipv4.tcp_orphan_retries defaults to 0, which causes orphaned TCP connections to accumulate indefinitely. DRBD creates many peer connections between nodes, and under load (node reboots, resync storms) this leads to TCP port exhaustion.
• Default tcp_fin_timeout (60s) is too long for fast port recycling during DRBD reconnect storms.
• Default TCP keepalive settings (2h idle) are too slow to detect dead DRBD peers.

We (Cozystack team) have confirmed this on multiple production clusters. The following sysctl overrides resolve the issue:

machine:
  sysctls:
    net.ipv4.tcp_orphan_retries: "3"
    net.ipv4.tcp_fin_timeout: "30"
    net.core.netdev_max_backlog: "5000"
    net.core.netdev_budget: "600"
    net.core.netdev_budget_usecs: "8000"
    net.ipv4.tcp_keepalive_time: "600"
    net.ipv4.tcp_keepalive_intvl: "10"
    net.ipv4.tcp_keepalive_probes: "6"

Question about approach

We'd like to discuss the best way to address this. A few options:

1. Consider more reasonable defaults in Talos itself — some of these values (especially tcp_orphan_retries=0) seem aggressive for any workload with many TCP connections, not just DRBD.

2. Update DRBD extension documentation — document that these sysctls are highly recommended when using DRBD, so users know to set them in their machine config.

3. Apply sysctls automatically as part of the DRBD extension — though this raises the question of how to handle user-provided overrides in machineconfig. Extension-set sysctls that conflict with user-specified values could cause confusion.

What is the preferred approach from the Talos side?

Related: cozystack/cozystack#2332