Duplicate SpanID is generated for trace spans

[Marco]

Hi Team, I have an inquiry related to trace spans generated by envoy sidecars:

Given that I have the following setup:

- A kubernetes cluster that has istio 0.8.0 as a service mesh with envoy sidecar injection enabled

- A service S1 deployed on that cluster

- An nginx-ingress-controller to access the service S1

After sending requests from nginx to the service S1, the generated trace spans are:

nginx (SpanID: 8640f64baf1a6a14)             [parent span]

  |

  |---> nginx  (SpanID: 3c2f583711047c5d)    [child span 1]

  |

  |---> S1     (SpanID: 3c2f583711047c5d)    [child span 2]

The issue is, the same SpanID 3c2f583711047c5d is generated for both nginx and S1 spans, which causing inconsistency in the tracing flow.

I dumped the request headers coming from nginx to the Service S1 below:

Accept: [*/*]

X-B3-Parentspanid: [8640f64baf1a6a14]

X-Envoy-Expected-Rq-Timeout-Ms: [15000]

Ssl-Client-Verify: [SUCCESS]

X-Forwarded-Host: [somehost:someport]

X-Forwarded-Proto: [https]

X-Scheme: [https]

X-Envoy-Internal: [true]

Ssl-Client-Issuer-Dn: [somevalues]

X-Forwarded-For: [someip]

X-Forwarded-Port: [443]

Content-Length: [153]

X-B3-Traceid: [751c7b5d680bc470]

Ssl-Client-Subject-Dn: [somevalues]

X-Request-Id: [2807a195f768fc7469ee1f68a946411f]

X-B3-Sampled: [1]

X-B3-Flags: [0]

X-Real-Ip: [someip]

X-Original-Uri: [someuri]

User-Agent: [curl/7.52.1]

Content-Type: [application/json]

X-B3-Spanid: [3c2f583711047c5d]

Is there anyway I can prevent generating the same SpanID for trace spans? Any directions or suggestions would be appreciated.

Kind regards,

Marco

[Marco]

Any recommendations about this issue?

Ideally, I would expect that nginx span would be the parent span of S1 span, and there is a new SpanID generated for S1 span. It should be something like this:

nginx                  (SpanID: 8640f64baf1a6a14)    [parent span]

  |

  |---> nginx          (SpanID: 3c2f583711047c5d)    [child span 1]

          |

          |---> S1     (SpanID: new-span-id)         [child span 2]

Thanks in advance

Marco

[Matt Klein]

+Gary/Ryan (tracing experts) that might have an idea.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/0323e3ef-4521-4964-89f4-4b7b73070e58%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Klein

Software Engineer

mkl...@lyft.com

https://calendly.com/mattklein123

[Ryan Burn]

It looks like this logic is getting invoked:

https://github.com/envoyproxy/envoy/blob/master/source/extensions/tracers/zipkin/tracer.cc#L85-L95

I think that’s for Zipkin’s shared client-server span style (https://github.com/opentracing/opentracing.io/issues/102).

But I don’t know the best way to disable it

[Gary Brown]

Hi Marco

Yes Ryan is correct, this is the way zipkin works.

It may be worth adding a configuration option to disable this behaviour and ensure the server span has an unique id. Could you create an issue for this?

Regards

Gary

On Wed, Aug 8, 2018 at 5:51 PM, Ryan Burn <ryan...@gmail.com> wrote:

It looks like this logic is getting invoked:

https://github.com/envoyproxy/envoy/blob/master/source/extensions/tracers/zipkin/tracer.cc#L85-L95

I think that’s for Zipkin’s shared client-server span style (https://github.com/opentracing/opentracing.io/issues/102).

But I don’t know the best way to disable it

On Aug 8, 2018, at 9:30 AM, Matt Klein <mkl...@lyft.com> wrote:

+Gary/Ryan (tracing experts) that might have an idea.

On Wed, Aug 8, 2018 at 1:31 AM Marco <marco.n...@gmail.com> wrote:

Any recommendations about this issue?

Ideally, I would expect that nginx span would be the parent span of S1 span, and there is a new SpanID generated for S1 span. It should be something like this:

nginx                  (SpanID: 8640f64baf1a6a14)    [parent span]

  |

  |---> nginx          (SpanID: 3c2f583711047c5d)    [child span 1]

          |

          |---> S1     (SpanID: new-span-id)         [child span 2]

Thanks in advance

Marco

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/0323e3ef-4521-4964-89f4-4b7b73070e58%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marco Bebway]

Thanks Gary and Ryan, for your responses.

I will create an issue for this.

Kind regards,

Marco

[Marco B]

Hi team,

Sorry for the very late follow-up.

I created an issue about this here https://github.com/envoyproxy/envoy/issues/4397.

Kindly take this forward.

Best regards,

Marco