Developers will be able to control how distributed tracing information is propagated

@tarekgh

Description edited by @tarekgh to include the API proposal

Background and Motivation

When .NET applications need to send their distributed tracing state to another process, they currently rely on functionality baked into HttpClient to transmit this information via HTTP headers (as described in W3C tracecontext spec and baggage spec). Similarly, when ASP.NET application receive distributed tracing information, they rely on the ASP.NET libraries to populate distributed tracing state based on incoming headers.

Unfortunately, the current implementation requires both libraries (HttpClient and ASP.NET) have prior knowledge of the specification. This makes it hard to react to changes in the specification or support other specifications (e.g., B3, Jaeger).

Propagation is a crossing-cutting concern should be solved in System.Diagnostics.DiagnosticsSource rather than require every library to have special knowledge of the wire protocol involved. Solving this in a generic way will also allow other libraries in the .NET ecosystem to participate in distributed tracing without requiring knowledge of the wire protocols involved.

The propagators are part of OpenTelemetry specification too. Having them in .NET is a good step to support it for libraries not taking direct dependencies on the OpenTelemetry.

This proposal also addressing solving customer scenarios that is listed below in the Customer issues addressed section.

Proposed API

namespace System.Diagnostics
{
    public abstract class TextMapPropagator
    {
      public delegate void PropagatorGetterCallback(object carrier, string fieldName, out string? value, out IEnumerable<string>? values);
      public abstract IReadOnlyCollection<string> Fields { get; }
      public abstract void Inject(Activity activity, object carrier, Action<object, string, string> setter);
      public abstract void Extract(object carrier, PropagatorGetterCallback getter, out string? id, out string? state);
      public abstract void Extract(object carrier, PropagatorGetterCallback getter, out IEnumerable<KeyValuePair<string, string?>>? baggage);
      public static TextMapPropagator Current { get; set; }
      public static TextMapPropagator CreateLegacyPropagator() { throw null; }
      public static TextMapPropagator CreatePassThroughPropagator() { throw null; }
      public static TextMapPropagator CreateNoOutputPropagator() { throw null; }
    }
}

Usage Examples

TextMapPropagator propagator = TextMapPropagator.Current;

propagator.Inject(Activity.Current, httpRequest, (carrier1, name, value1) => Console.WriteLine($"{name}: {value1}"));

propagator.Extract(httpRequest, (carrier2, name, out value, out values) => { Console.WriteLine($"{name}"); value =  "Mapped Key Value"; values = null; } );

Customer issues addressed:

Pass-thru mode

@a-elsheikh, @rynowak

There are certain environments (service meshes, dapr) where you are trying to avoid creating a intermediate span. Creating intermediate spans that aren't exported result in the causal chain being broken. (dotnet/aspnetcore#30392)

Proposed solution:

Create Activity with predetermined SpanId (which is equal to the parent's span id). This will require a new API
Call ActivityListeners, i.e., no special heuristics to suppress Activity Created callbacks when in pass-thru mode. Though we expect the use case here is that there are no listeners in the process.
Open question: How does the library (ASP.NET) know that is in this special pass-thru mode?
- Is it a new static property on Activity?
- Or is it a library specific setting?
- Add a bit to ActivityContext to indicate we're in pass-thru mode
This scenario will be broken by uncooperative libraries who create child spans, but we will do nothing special to mitigate this

React to spec changes

@karelz, @MihaZupan, @Tratcher

As an example, the W3C baggage specification changed the name of the header from Correlation-Context to Baggage. We'd need a way to change to version and change propagation behavior without requiring every library to react to spec changes

-#45496
-dotnet/aspnetcore#28319

Turn off propagation

@karelz, @MihaZupan

There are instances where application want to turn off propagation the currently isn't an easy way to do without requiring every library to add library-specific settings. This could be addressed by adding a no-op propagator.

TODO: Add links to HttpClient issues where customers want to disable header propagation

Potential work items:

Add Propagator (abstract base class) to System.Diagnostics.DiagnosticSource
~~Add implementation for W3C propagator~~ W3C baggage specs still not finalized. we'll wait to expose this propagator till the specs are final.
Add implementation for no-op propagator
~~Add a static DefaultPropagator to Activity~~ the static property will be on the propagator abstract class (i.e. TextMapPropagator) instead of Activity class.
~~Add API to create Activity with predetermined SpanId~~ this will not be needed after we expose the propagator.
Result of the open question
Modify System.Net.Http.DiagnosticsHandler to use the Default Propagator
Modify Microsoft.AspNetCore.Hosting.HostingApplicationDiagnostics to use the Default Propagator

@cijothomas

cc @cijothomas @noahfalk @reyang

Add API to create Activity with predetermined SpanId

I think this one was also part of the Open Question? My recollection when we chatted is that we were leaning away from adding a new API that sets spanId and instead add a static property that causes the local child to mirror the remote parent spanId.

I think this one was also part of the Open Question? My recollection when we chatted is that we were leaning away from adding a new API that sets spanId and instead add a static property that causes the local child to mirror the remote parent spanId.

You're right, I believe the three options were:

Add a static property to Activity
Use the CreateActivity overload that already takes in ActivityContext. It can be inferred from the ActivityContext
Add a new API to CreateActivity with pre-determined SpanId

@shirhatti

@shirhatti and I just chatted about an alternative idea for how to do pass-through mode propagation. Rather than try to restrict which Activities start/make SpanIds mirror a remote parent, we could have a propagator that emits the original IDs regardless. To do this the propagator would start from Activity.Current, follow the Activity.Parent linked list up to the local root, then emit the root TraceId and ParentSpanId.

Runtime issues for reference:

Add an option to HttpClient's default DiagnosticHandler so that it does not create an Activity #41072: Not creating extra activities
HttpClient automatically adds Request-Id HTTP header #35337: On/off
Customized SocketsHttpHandler does not support diagnostics tracking #31261: Support for custom SocketsHttpHandlers

Based on your email I assume you meant to mark it as blocking, not blocked

@terrajobst

@terrajobst

Based on your email I assume you meant to mark it as blocking, not blocked

Yes, thanks for the catch. I hope we can use other tags to mark the issues ready for design review. It is easy to run into this mistake :-)

Since there's already a strong delegate type for the getter, let's use a strong delegate type for the setter to name the string parameters.
Extract(out id, out state) should use the "trace" prefix to help callers understand the alignment to (e.g.) Activity.TraceStateString
Instead of relying on the shape of the out parameters, the different Extract methods should use purpose names
We renamed CreateLegacyPropagator() to CreateDefaultPropagator()

namespace System.Diagnostics
{
    public abstract class TextMapPropagator
    {
      public delegate void PropagatorGetterCallback(object carrier, string fieldName, out string? fieldValue, out IEnumerable<string>? fieldValues);
      public delegate void PropagatorSetterCallback(object carrier, string fieldName, string fieldValue);

      public abstract IReadOnlyCollection<string> Fields { get; }
      public abstract void Inject(Activity activity, object carrier, PropagatorSetterCallback setter);
      public abstract void ExtractTraceIdAndState(object carrier, PropagatorGetterCallback getter, out string? traceId, out string? traceState);
      public abstract IEnumerable<KeyValuePair<string, string?>>? ExtractBaggage(object carrier, PropagatorGetterCallback getter);

      public static TextMapPropagator Current { get; set; }
      public static TextMapPropagator CreateDefaultPropagator() { throw null; }
      public static TextMapPropagator CreatePassThroughPropagator() { throw null; }
      public static TextMapPropagator CreateNoOutputPropagator() { throw null; }
    }
}

It has been decided to rename TextMapPropagator to DistributedContextPropagator. The rest of the approved proposal still the same.

#55539

namespace System.Diagnostics
{
    public abstract class DistributedContextPropagator
    {
      public delegate void PropagatorGetterCallback(object? carrier, string fieldName, out string? fieldValue, out IEnumerable<string>? fieldValues);
      public delegate void PropagatorSetterCallback(object? carrier, string fieldName, string fieldValue);

      public abstract IReadOnlyCollection<string> Fields { get; }
      public abstract void Inject(Activity? activity, object? carrier, PropagatorSetterCallback setter);
      public abstract void ExtractTraceIdAndState(object? carrier, PropagatorGetterCallback getter, out string? traceId, out string? traceState);
      public abstract IEnumerable<KeyValuePair<string, string?>>? ExtractBaggage(object? carrier, PropagatorGetterCallback getter);

      public static DistributedContextPropagator Current { get; set; }
      public static DistributedContextPropagator CreateDefaultPropagator() { throw null; }
      public static DistributedContextPropagator CreatePassThroughPropagator() { throw null; }
      public static DistributedContextPropagator CreateNoOutputPropagator() { throw null; }
    }
}

Navigation Menu

Repository navigation

Developers will be able to control how distributed tracing information is propagated #50658

Background and Motivation

Proposed API

Usage Examples

Customer issues addressed:

Pass-thru mode

React to spec changes

Turn off propagation

Potential work items:

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Developers will be able to control how distributed tracing information is propagated #50658

Description

Background and Motivation

Proposed API

Usage Examples

Customer issues addressed:

Pass-thru mode

React to spec changes

Turn off propagation

Potential work items:

Activity

shirhatti commented on Apr 2, 2021

noahfalk commented on Apr 3, 2021

shirhatti commented on Apr 3, 2021

noahfalk commented on Apr 7, 2021

MihaZupan commented on May 6, 2021

3 remaining items

terrajobst commented on Jun 30, 2021

tarekgh commented on Jun 30, 2021

bartonjs commented on Jul 6, 2021

tarekgh commented on Jul 13, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions