Zipkin Tracing spans

[Amit Saha]

Hey all,

This is what we are currently trying. On service 1, we have linkerd running with this config:

admin:

  port: 9990

telemetry:

- kind: io.l5d.zipkin

  host: <zipkin server DNS>

  port: 9410

  sampleRate: 1.0

namers:

- kind: io.l5d.consul

  host: <consul DNS>

routers:

- protocol: thrift

  label: service2

  thriftProtocol: binary

  dtab: |

    /svc  => /#/io.l5d.consul/.local/service2;

  client:

    thriftFramed: false

  servers:

  - port: 11500

    ip: 0.0.0.0

    thriftFramed: false

So, all requests from this service for port 11500 goes to the real service via linkerd. There is a linkerd instance running on service2 as will with a similar config, but that proxies requests on a different port to a different service ((via linked):

admin:

  port: 9990

telemetry:

- kind: io.l5d.zipkin

  host: <zipkin server DNS>

  port: 9410

  sampleRate: 1.0

namers:

- kind: io.l5d.consul

  host: <consul DNS>

routers:

- protocol: thrift

  label: service3

  thriftProtocol: binary

  dtab: |

    /svc  => /#/io.l5d.consul/.local/service3;

  client:

    thriftFramed: false

  servers:

  - port: 11501

    ip: 0.0.0.0

    thriftFramed: false

(The above are Python thrift services)

When I see the tracing data collected on the zipkin web UI, I see the data for the call to service2 and service 3 separately as separate traces. Now, of course this is something I expect, since there is no way linkerd can make the span from service2-service3 be in a nested span of the call from service1-service2?

If that is correct, how would I go about nesting a lower span in it's outer span via linkerd?

Thanks,

Amit.

[Kevin Lingerfelt]

Hi Amit,

I can confirm that the behavior that you're seeing is what I would expect to see for python thrift clients. Native thrift doesn't have the ability to propagate request context, which is what linkerd would need to link the two spans to the same trace. Unless the request from service2 to service3 passed along some information about the request from service1 to service2, there's no way for linkerd to link those 2 requests together.

For HTTP, we accomplish this by way of the l5d-ctx headers, which are described in more detail here:

https://buoyant.io/2017/03/14/a-service-mesh-for-kubernetes-part-vii-distributed-tracing-made-easy/#a-note-about-request-context

For thrift, it is possible to propagate request context, but I think you would need to use an upgraded protocol such as TTwitter, and I don't think that's supported in python, unfortunately. Maybe some other folks on the mailing list have other ideas that might work though.

Kevin

--
You received this message because you are subscribed to the Google Groups "linkerd-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to linkerd-users+unsubscribe@googlegroups.com.
To post to this group, send email to linker...@googlegroups.com.
Visit this group at https://groups.google.com/group/linkerd-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/linkerd-users/CANODV3m7rASrArH-nHpGiRtZudFK13DON-DfLAtEPZmqMXKomQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Amit Saha]

Thanks Kevin. That's a bummer. Will it be possible to point me to the source file(s) where the trace is being injected in linkerd and then looked up? I will look into Python thrift servers to see if we can do something.

I imagine linkerd + gRPC is a different story?

To unsubscribe from this group and stop receiving emails from it, send an email to linkerd-user...@googlegroups.com.

[Kevin Lingerfelt]

Hi Amit,

There's more info on this issue about how Linkerd serializes trace context:

https://github.com/linkerd/linkerd/issues/1428

And you're correct that gRPC requests will have a much better mechanism for context propagation. For example, here's a Go implementation of gRPC context propagation:

https://github.com/linkerd/linkerd-examples/blob/master/docker/helloworld/grpc/context.go

Hope that helps,

Kevin

To unsubscribe from this group and stop receiving emails from it, send an email to linkerd-users+unsubscribe@googlegroups.com.

[Amit Saha]

On Sat, Jul 15, 2017 at 10:58 AM Kevin Lingerfelt <k...@buoyant.io> wrote:

Hi Amit,

There's more info on this issue about how Linkerd serializes trace context:

https://github.com/linkerd/linkerd/issues/1428

Thanks again. I did a quick search with  calls to TraceId.serialize ("https://github.com/linkerd/linkerd/search?utf8=%E2%9C%93&q=TraceId.serialize&type=" ) but only saw two instances of it in h2 and http protocol packages. 

I was hoping to see how a trace ID would be set or one looked up and reused for Thrift protocol. Would it be possible to be pointed to that?

And you're correct that gRPC requests will have a much better mechanism for context propagation. For example, here's a Go implementation of gRPC context propagation:

https://github.com/linkerd/linkerd-examples/blob/master/docker/helloworld/grpc/context.go

Thanks, that looks pretty neat.

Hope that helps,

Kevin

To unsubscribe from this group and stop receiving emails from it, send an email to linkerd-user...@googlegroups.com.

[Kevin Lingerfelt]

Hi Amit,

I recommend looking at the TTwitterClientFilter, here:

https://github.com/linkerd/linkerd/blob/master/linkerd/protocol/thrift/src/main/scala/com/twitter/finagle/buoyant/linkerd/TTwitterClientFilter.scala

That filter applies the request context to outbound thrift request using the context fields that are part of the TTwitterThrift paylaods.

Kevin

Hope that helps,

Kevin

To unsubscribe from this group and stop receiving emails from it, send an email to linkerd-users+unsubscribe@googlegroups.com.