Skip to content

Improve handling of CANCELLED errors in the dispatch layer #3300

Open

Description

@tstirrat15

CANCELLED at the server level should mean "the client cancelled the request and went away." Currently, with the way that our dispatching layer works, a dispatching gRPC client can return with a CANCELLED error at its layer and then that bubbles up to the caller, which obscures what's going on at the server level.

A couple things here:

  1. Wrap the dispatcher remote boundary to catch CANCELLED errors and turn them into UNAVAILABLEs if the caller's context isn't already cancelled. This happens in internal/dispatch/remote/cluster.go at lines 206, 780, 731, and 744. Make sure the context check is done on the dispatcher's inbound context rather than the hedging context.
  2. Make hedging cancels more visible with context.WithCancelCause in cluster.go at lines 552 and 557. We use this pattern at internal/graph/cursors.go:451, and we work around the lack of this pattern at internal/graph/lookupsubjects.go:354.

Activity

  1. breken-ai commented on Sep 9, 2026

    @breken-ai

    Fix up matching the spec - PR: #3301

    Someone filed this with an exact fix spec; we implemented it at the four named sites (DispatchCheck, DispatchExpand, both hedged-streaming returns), switched hedged contexts to context.WithCancelCause, and added unit + bufconn integration coverage. Full internal/dispatch/remote suite passes with -race. The non-hedged streaming setup paths are called out in the PR as deliberate follow-up scope.

    Built by breken, your AI support engineer - breken.ai finds the bug, proves the fix with unmatched RCA, answers customers in your voice, and keeps your code and support docs clean - this one's on us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions