Skip to content

Cloudflare RPC trace propagation changes method arguments for uninstrumented receivers #23233

Description

@matthew-gizmo

Environment

  • @sentry/cloudflare: 10.69.0
  • Cloudflare Workers service-binding RPC
  • Caller: wrapped with Sentry.withSentry()
  • Receiver: plain, uninstrumented WorkerEntrypoint

Description

With enableRpcTracePropagation: true, the Cloudflare SDK propagates trace context for RPC method calls by appending a trailing { __sentry_rpc_meta__: ... } argument.

A receiver wrapped by Sentry removes that argument before invoking application code. A plain receiver does not, so enabling Sentry on the caller changes the receiver's application-visible RPC arguments.

This can occur during gradual deployments, rollbacks, mixed SDK versions, or when an instrumented Worker calls a service that does not use Sentry.

Minimal reproduction

Define an uninstrumented receiver with an optional trailing parameter:

import { WorkerEntrypoint } from "cloudflare:workers";

export default class Receiver extends WorkerEntrypoint {
  inspect(first: string, second: string, optionalTimeoutMs?: number) {
    return {
      first,
      second,
      optionalArgumentType: typeof optionalTimeoutMs,
      receivedSentryMetadata:
        typeof optionalTimeoutMs === "object" &&
        optionalTimeoutMs !== null &&
        "__sentry_rpc_meta__" in optionalTimeoutMs,
    };
  }
}

Wrap only the caller and enable RPC trace propagation:

export default Sentry.withSentry(
  env => ({
    dsn: env.SENTRY_DSN,
    tracesSampleRate: 1,
    enableRpcTracePropagation: true,
  }),
  Caller,
);

From an active sampled trace, call:

await env.RECEIVER.inspect("first", "second");

Actual behavior

The receiver observes a third object argument containing __sentry_rpc_meta__. In this example, optionalArgumentType is "object" rather than "undefined".

This can silently alter behavior. For example, if an optional numeric argument is passed to setTimeout, the metadata object may be coerced into an immediate timeout.

Expected behavior

Enabling tracing on a caller should not change the receiver's application-visible RPC arguments. If mixed instrumentation cannot preserve RPC semantics, the SDK should fail safely or provide a clear runtime/configuration warning rather than forwarding internal metadata into application parameters.

Additional context

The SDK documentation says both sides should enable this option for full trace propagation. That explains why an instrumented receiver can extract the metadata, but it does not make mixed deployments safe: caller-only instrumentation currently changes the RPC method contract rather than merely losing trace continuity.

Possible directions include an out-of-band propagation mechanism, mixed-version-safe negotiation, or an explicit guard/warning when safe receiver extraction cannot be guaranteed.

Activity

  1. linear-code commented on Aug 10, 2026

    @linear-code
  2. moved this to Waiting for: Product Owner in GitHub Issues with 👀 3on Aug 10, 2026
  3. JPeer264 commented on Aug 10, 2026

    @JPeer264
    Member

    Hey thanks for raising this issue and using this opt-in.

    This is kinda intended, but I see the use case where it could lead to issues. It is actually more complicated and technically quite hard to achieve (but not impossible) - Internally we have no idea if another binding is instrumented or not, unless we go via our Vite plugin.

    I would assume your `Receiver is set as a Service Binding in your wrangler config, while it lives somewhere else?

    If so, then I see the following to make this work:

    1. We create an allow list which bindings should add RPC metadata (manual)
    2. And in our Vite plugin we could disable service bindings by default and make them opt-in in case other, because we know if bindings in the same main entrypoint are instrumented or not, but do not know about service bindings that live somewhere
  4. moved this from Waiting for: Product Owner to No status in GitHub Issues with 👀 3on Aug 10, 2026
  5. matthew-gizmo commented on Aug 11, 2026

    @matthew-gizmo
    Author

    Thanks for the reply. Yes, the receiver is a Service Binding configured in Wrangler and deployed independently from the caller.

    I agree the difficult part is that the caller needs to know whether the receiver is instrumented and capable of removing the metadata. That knowledge can become stale when the services are deployed independently: a gradual deployment, rollback, configuration change, or SDK-version mismatch can leave the caller appending metadata while the receiver no longer understands it. The Vite plugin can establish this safely for Workers it builds and controls together, but I don’t think it can guarantee compatibility with an independently deployed service as you say.

    I also want to stress that this is not only a theoretical compatibility concern: it caused a production incident for us. It was particularly difficult to diagnose because everything remained type-correct at compile time, while Sentry silently changed the runtime arguments crossing the RPC boundary. The resulting behaviour appeared inside the receiving Worker, far away from the instrumentation responsible for adding the argument, so discovering the root cause took significant investigation.

    I think the underlying platform limitation is that Cloudflare RPC does not expose a metadata channel equivalent to HTTP headers or gRPC metadata. Sentry therefore has nowhere transparent to carry tracing context. Appending a private argument is understandable as a workaround, but it changes the application-level RPC contract and bypasses the guarantees TypeScript gives us at that boundary. I can't see Cloudflare adding this anytime soon unfortunately.

    It would also be useful if the documentation made the compatibility requirement explicit: enabling propagation on the caller is only safe when the receiver is guaranteed to be instrumented with compatible extraction behaviour. Losing trace continuity during a mixed deployment would be acceptable; changing application-visible arguments is not.

    As a temporary workaround we have had to add linting to detect incompatible RPC method signatures, and comments to override to say that we are safely boundary checking. Engineers are pretty unhappy about this as a solution, but it's the best we can think of right now.

  6. moved this to Waiting for: Product Owner in GitHub Issues with 👀 3on Aug 11, 2026
  7. JPeer264 commented on Aug 12, 2026

    @JPeer264
    Member

    I also want to stress that this is not only a theoretical compatibility concern: it caused a production incident for us.

    Sorry that this happened. I'll try to make it clearer in the docs so it is clear what is actually happening behind the scenes.

    I think the underlying platform limitation is that Cloudflare RPC does not expose a metadata channel equivalent to HTTP headers or gRPC metadata.

    Exactly that is the issue, there is no other way to make it happen. It could work by pinging first on a Sentry owned function on the instrumented DO first, but this is an extra round trip which is IMO not good enough to add.


    I think the best way forward is adding an allow list with an exact list of which bindings should receive the RPC additions. However, when using Vite and our plugin, the allow list is automatically set for every binding within the same deployment, and needs an allow list for external bindings/services. Do you think this would solve this?

  8. moved this from Waiting for: Product Owner to No status in GitHub Issues with 👀 3on Aug 12, 2026
  9. added 2 commits that reference this issue on Aug 28, 2026
    3566403
    7917b0d
  10. JPeer264 commented on Aug 31, 2026

    @JPeer264
    Member

    @matthew-gizmo in 10.72.0 we released the safer option: rpcTracePropagationBindings, which is similar to our existing option tracePropagationTargets for HTTP requests. We deprecated enableRpcTracePropagation in favor of rpcTracePropagationBindings.

    By default rpcTracePropagationBindings doesn't do anything, as it is opt-in, when you use it with wrangler directly. So here you have to allow list each of your bindings via a string, or RegExp. If you'd use the Vite plugin this would be added automatically for you - as this would only add it for bindings within the same deployment, not external services.

    I'd suggest that you change your configs to allow list only the bindings which are within your deployment:

    // caller
    withSentry(
      () => ({
    -    enableRpcTracePropagation: true,
    +    rpcTracePropagationBindings: ['MY_BINDING', 'MY_OTHER_BINDING']
      }),
      MyApp,
    )
    
    // receiver
    // set it to an empty object or
    // you could even reuse the options from the caller, it wouldn't matter
    instrumentDurableObjectWithSentry(
      () => ({
    -    enableRpcTracePropagation: true,
    +    rpcTracePropagationBindings: []
      }),
      MyDO,
    )

    The docs will soon be out and are opened here.

  11. github-actions commented on Sep 23, 2026

    @github-actions
    Contributor

    A PR closing this issue has just been released 🚀

    This issue was referenced by PR #23486, which was included in the 11.0.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions