Skip to content

bug(api): discovery endpoint timeout due to HTTP/2 request queuing on aggregated API server #2324

Open
Open
bug(api): discovery endpoint timeout due to HTTP/2 request queuing on aggregated API server#2324
Labels
area/apiIssues or PRs related to the cozystack-api aggregated API serverkind/bugCategorizes issue or PR as related to a buglifecycle/staleDenotes an issue or PR has remained open with no activity and has become stalepriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

Description

@kvaps
Andrei Kvapil (kvaps)
opened on Apr 2, 2026

Summary

The cozystack-api aggregated API server reports periodic Timeout or abort while handling GET /apis/apps.cozystack.io/v1alpha1 errors (~10-20 per minute). These timeouts cause the dashboard to intermittently return 500 errors and fail to load.

Root Cause

Discovery requests from system:kube-aggregator arrive at cozystack-api with already-expired deadlines. The requests are queued for ~30 seconds in the HTTP/2 transport layer before reaching the handler. Once delivered, cozystack-api processes them successfully in <1ms (HTTP 200), but the 30-second timeout has already fired.

Evidence from audit log

A single request showing both successful processing AND timeout:

RequestReceived:  2026-04-02T14:40:32.538478Z  user=system:kube-aggregator  src=10.80.4.150
ResponseComplete: 2026-04-02T14:40:32.539310Z  code=200  (processing time: 0.8ms)
Timeout fired:    2026-04-02T14:40:32Z          "Timeout or abort while handling"

The handler completed in 0.8ms with a 200 response, but the request's context deadline (set 30 seconds earlier by kube-apiserver) had already expired.

Why requests are delayed

The cozystack-api pod that receives dashboard watch streams also receives discovery probes from kube-apiserver. These share HTTP/2 connections from kube-apiserver's aggregation proxy. TCP socket analysis shows two connections from the same kube-apiserver to the affected pod:

  • fd=13: healthy (rtt=0.07ms, 514KB transferred) — carries successful discovery requests
  • fd=10: degraded (rtt=16ms, deviation=19ms, 30KB transferred) — likely carries delayed requests

The affected pod consistently has 27 active watch streams (one per ApplicationDefinition from dashboard BFF), while the unaffected pod has 0 watches. With 26 ApplicationDefinitions, each (*REST).Watch() creates a separate direct watch connection via client.WithWatch, generating significant HTTP/2 stream load through the kube-apiserver proxy.

Cascade to dashboard

  1. cozystack-api logs timeout errors (even though responses succeed)
  2. Some kube-apiserver discovery probes genuinely fail due to expired deadlines
  3. Dashboard BFF's swagger cache reload (2317 API paths) intermittently fails
  4. BFF becomes unresponsive → fails liveness probe (2s timeout) → killed → 500 for users
  5. Observed: 20 BFF container restarts over ~36 hours

Observations

  • The problem always affects the pod receiving watch streams (consistent across multiple pod restarts and nodes)
  • Direct access to the affected pod (curl from localhost) responds in 5ms — the pod itself is not slow
  • Disabling Priority and Fairness (--enable-priority-and-fairness=false) did not help
  • The forked k8s.io/apimachinery only changes scheme.go (GVK selection) and is unrelated
  • Each Watch() call in pkg/registry/apps/application/rest.go creates a new direct watch via r.w.Watch() instead of leveraging the shared controller-runtime informer cache (already watching HelmRelease in pkg/apiserver/apiserver.go:165). This creates 26+ individual watch streams per dashboard client session

Suggested Improvements

  1. Use shared informer cache for Watch() instead of individual client.WithWatch connections. The controller-runtime manager already watches HelmRelease — Watch() could filter events from the shared cache rather than creating N separate watch streams. This would reduce HTTP/2 stream pressure from N×clients to 1 shared watch.

  2. Increase dashboard BFF liveness probe timeout from 2s to a higher value to reduce unnecessary restarts during transient API slowness.

Environment

  • Cozystack v1.1.3
  • Kubernetes v1.34 (Talos)
  • 26 ApplicationDefinitions
  • 2 cozystack-api replicas
  • 3 control plane nodes

Activity

  1. dosubot
    added
    kind/bugCategorizes issue or PR as related to a bug
    on Apr 2, 2026

  2. kvaps commented on Apr 2, 2026

    @kvaps
    Andrei Kvapil (kvaps)
    on Apr 2, 2026
    MemberAuthor

    Additional findings: kube-controller-manager watch amplification

    Further investigation via kube-apiserver audit logs on the node (using talm) revealed that the watch stream count is significantly higher than initially estimated.

    Active watches through a single kube-apiserver (cp-150)

    From the audit log, 36 active watch streams pass through kube-apiserver on cp-150:

    • ~14 HelmRelease watches from system:serviceaccount:cozy-system:cozystack-api — these are the per-resource-type watches created by (*REST).Watch() going back into the same kube-apiserver
    • ~22 watches from system:kube-controller-manager — metadata informers for all apps.cozystack.io resource types (tenants, vpns, mongodbs, harbors, buckets, etc.)

    Watch amplification loop

    Each apps.cozystack.io watch creates a circular dependency through the same kube-apiserver:

    kube-controller-manager
  → kube-apiserver (aggregation proxy)
    → cozystack-api (REST.Watch)
      → kube-apiserver (HelmRelease watch)

    Every incoming watch for an apps.cozystack.io resource spawns a new outgoing HelmRelease watch back to kube-apiserver. With 26 ApplicationDefinitions and multiple clients (kube-controller-manager on each node + dashboard BFF), this creates 50-80+ concurrent HTTP/2 streams funneled through a single kube-apiserver.

    Impact on HTTP/2 transport

    The kube-apiserver audit log on cp-150 confirmed that all timeouts come from system:kube-aggregator (the availability controller). The requests arrive at cozystack-api with already-expired 30s deadlines — they were queued in the HTTP/2 transport for the full timeout period before delivery.

    This reinforces that using the shared informer cache instead of individual client.WithWatch connections would eliminate the amplification loop entirely.

  3. Aleksei Sviridkin (lexfrei)
    added
    area/apiIssues or PRs related to the cozystack-api aggregated API server
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    triage/acceptedIndicates an issue is ready to be actively worked on
    on Apr 27, 2026

  4. dosubot commented on Jul 27, 2026

    @dosubot
    dosubotbot
    on Jul 27, 2026 – with DosuBot

    Hi, Andrei Kvapil (@kvaps). I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.

    Recap

    • You identified that the intermittent cozystack-api discovery timeouts and dashboard 500s are not caused by slow handler execution.
    • The evidence points to kube-apiserver’s HTTP/2 aggregation proxy queuing discovery requests until their 30s deadlines expire before they reach the backend.
    • You also found significant watch amplification, including HelmRelease watches from cozystack-api and metadata informers from kube-controller-manager.
    • The resulting circular dependency can create 50–80+ concurrent HTTP/2 streams on a single apiserver, with timeouts attributed to system:kube-aggregator.
    • The likely fix is to replace per-resource client.WithWatch connections with shared informer cache usage to remove the amplification loop.

    Next Steps

    • Please let us know if this is still relevant on the latest version of the cozystack repository; if it is, you can keep the discussion open by commenting here.
    • Otherwise, this issue will be automatically closed in 7 days.

    Thanks for your understanding and contribution.

  5. dosubot
    added
    stale
    on Jul 27, 2026
  6. github-actions
    added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale
    and removed
    stale
    on Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiIssues or PRs related to the cozystack-api aggregated API serverkind/bugCategorizes issue or PR as related to a buglifecycle/staleDenotes an issue or PR has remained open with no activity and has become stalepriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions