cilium L2 not responding to ARP requests in VLAN iface... but sends gratuitous ARP...? #37959

iface vlan.1881
vlan id: 1881
Address: 1.0.0.52/29
Gateway: 1.0.0.49

# kubectl -n kube-system get lease | grep "cilium-l2announce"
cilium-l2announce-kong-prod-kong-gateway-proxy-2   bm2-metal                                                               22m

# kubectl -n kube-system exec pod/cilium-l9wwv -- cilium-dbg shell -- db/show l2-announce
IP             NetworkInterface
1.0.0.56   vlan.1881

# kubectl -n kube-system exec pod/cilium-l9wwv -- cilium-dbg shell -- db/show devices    
Name              Index   Selected   Type     MTU     HWAddr              Flags                       Addresses
lo                1       false      device   65536                       up|loopback                 127.0.0.1, ::1
ens16             2       false      device   1500    aa:77:34:5b:ec:f9   up|broadcast|multicast      fe80::a877:34ff:fe5b:ecf9
vlan.1881         3       true       vlan     1500    aa:77:34:5b:ec:f9   up|broadcast|multicast      1.0.0.52, fe80::a877:34ff:fe5b:ecf9
tailscale0        4       false      tuntap   1280                        up|pointtopoint|multicast   100.125.160.99, fd7a:115c:a1e0::1e01:a063, fe80::2ccf:e9f7:df00:6f93
cilium_net        5       false      veth     1500    9a:f7:07:ee:53:29   up|broadcast|multicast      fe80::98f7:7ff:feee:5329
cilium_host       6       false      veth     1500    52:2a:04:06:11:6a   up|broadcast|multicast      10.42.2.169, fe80::502a:4ff:fe06:116a
cilium_vxlan      7       false      vxlan    1500    12:c5:ad:74:1c:10   up|broadcast|multicast      fe80::10c5:adff:fe74:1c10
lxc8db2bffa7ef5   11      false      veth     1500    de:28:bb:d7:57:05   up|broadcast|multicast      fe80::dc28:bbff:fed7:5705
lxc84d84293d6ef   183     false      veth     1500    2a:19:48:0c:0f:cf   up|broadcast|multicast      fe80::2819:48ff:fe0c:fcf
lxc_health        371     false      veth     1500    a6:b8:43:2a:de:d4   up|broadcast|multicast      fe80::a4b8:43ff:fe2a:ded4
lxc2abfdded438c   373     false      veth     1500    d2:f2:54:99:77:b2   up|broadcast|multicast      fe80::d0f2:54ff:fe99:77b2

13:00:52.258145 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.0.0.56 tell 1.0.0.49, length 42
13:00:54.357723 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.0.0.56 tell 1.0.0.49, length 42
13:00:55.068209 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.0.0.56 tell 1.0.0.49, length 42
13:00:55.767885 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.0.0.56 tell 1.0.0.49, length 42
13:00:56.468304 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.0.0.56 tell 1.0.0.49, length 42
13:00:57.268271 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 1.0.0.56 tell 1.0.0.49, length 42
13:00:57.968840 ARP, Ethernet (len 6), IPv4 (len 4), Reply 1.0.0.56 is-at aa:77:34:5b:ec:f9, length 46 (gratuitous ARP)

# kubectl -n kube-system exec pod/cilium-l9wwv -- bpftool map dump pinned /sys/fs/bpf/tc/globals/cilium_l2_responder_v4
[{
        "key": {
            "ip4": 939625613,
            "ifindex": 3
        },
        "value": {
            "responses_sent": 0
        }
    }
]

apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "bm2-pool"
spec:
  blocks:
    - start: "1.0.0.56"
      stop: "1.0.0.58"
  serviceSelector:
    matchLabels:
      ipam: bm2-pool

apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: bm2-lbpolicy
  namespace: kube-system
spec:
  serviceSelector:
    matchExpressions:
      - {key: ipam, operator: In, values: [bm2-pool]}
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: DoesNotExist
    matchLabels:
      kubernetes.io/hostname: bm2-metal
  interfaces:
    - vlan.1881
  externalIPs: true
  loadBalancerIPs: true