Tracing span is not created by istio-proxy/envoy when the trace does not originate at the ingress

[Gaurav Abbi]

Hi Team, I am using Istio 0.7 as a service mesh in Kubernetes and thus also using Envoy. 

I have distributed tracing integrated but it is slightly different from the one we get in Istio OOB documentation.

I have an application that consumes events from an event-bus (nats streaming) and triggers webhooks. This is not a Kubernetes service, but a Kubernetes deployment with side-car injected.

nats-streaming ---> (push app + envoy sidecar)  -----> (service B + envoy sidecar)

In the trigger for the webhook, I am passing all the tracing context as headers. 

I expect the tracing span to be created and show up in the UI and link with the parent span. I see that not happening.

Here are the headers that I send to service B

map[x-b3-traceid:[3fa64900c340f7b0404f7f29654075af] x-b3-flags:[1] x-request-id:[11e1e9d1-5e2a-42b2-825d-c8a32116b7df] x-b3-spanid:[782d1dd76ecee838] x-b3-parentspanid:[7e9f8b0911a17f73] x-b3-sampled:[1]]

I can see the same headers in the service B where the span ID is not updated. This implies to me that the envoy is not creating a new span and linking with the parent span.

Is there a way I can debug this issue? Any suggestions or pointers would be helpful.

Note: I can see the original trace and spans in the tracing UI, but not the span when service B is called.

[Matt Klein]

+Gary who might be able to help with Zipkin debugging.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/8644b57e-cd36-4f80-8f45-1076876c8596%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Klein

Software Engineer

mkl...@lyft.com

https://calendly.com/mattklein123

[Gary Brown]

The x-request-id value indicates that the envoy proxy has been configured not to trace.

Are you only passing the X-B3 headers into the outbound proxy of your push app? If so, then for some reason that proxy is deciding not to trace your request - possibly you have configured a sampling policy? e.g. https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/runtime#config-http-conn-man-runtime-client-enabled

One option would be to set the https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/headers#x-client-trace-id header in your push app, as this should force tracing.

Post Istio version 0.7.1, the Envoy proxy will defer to the tracer when making sampling decisions, so as you are passing in the X-B3-Sampled header, Envoy will use this to determine whether to trace the traffic - only using its own sampling decision for new traces.

Hope that helps.

Regards

Gary

[Gaurav Abbi]

Hi Gary,

Thanks for the response.

I am passing the X-B3 headers to the outbound proxy of the push app. 

For me, what is perplexing is that when I do not pass the X-B3 headers, Envoy creates a trace for the push app ---> service B communication.
I tried setting the x-client-trace-id, but in this case also, it creates a separate trace when X-B3 headers are not passed and no trace when X-B3 Headers are passed.

Here is some more information about what I am trying to achieve.

envoy + request api (to publish messge) ---> messaging backend ----> push app + envoy ----> envoy + consumer service (service B).

Now there is a disconnect at the messaging backend level, but I am essentially trying to fill this disconnect by propagating the same trace context headers.

I can see separate traces:

1. envoy + request api (to publish messge) ---> messaging backend

2. push app + envoy ----> envoy + consumer service (service B)

  but not one holistic that shows the whole flow.

On Monday, April 16, 2018 at 11:23:21 AM UTC+2, Gary Brown wrote:

The x-request-id value indicates that the envoy proxy has been configured not to trace.

Are you only passing the X-B3 headers into the outbound proxy of your push app? If so, then for some reason that proxy is deciding not to trace your request - possibly you have configured a sampling policy? e.g. https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/runtime#config-http-conn-man-runtime-client-enabled

One option would be to set the https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/headers#x-client-trace-id header in your push app, as this should force tracing.

Post Istio version 0.7.1, the Envoy proxy will defer to the tracer when making sampling decisions, so as you are passing in the X-B3-Sampled header, Envoy will use this to determine whether to trace the traffic - only using its own sampling decision for new traces.

Hope that helps.

Regards

Gary

On Sat, Apr 14, 2018 at 9:07 PM, Matt Klein <mkl...@lyft.com> wrote:

+Gary who might be able to help with Zipkin debugging.

On Sat, Apr 14, 2018 at 8:08 AM, Gaurav Abbi <abbi....@gmail.com> wrote:

Hi Team, I am using Istio 0.7 as a service mesh in Kubernetes and thus also using Envoy. 

I have distributed tracing integrated but it is slightly different from the one we get in Istio OOB documentation.

I have an application that consumes events from an event-bus (nats streaming) and triggers webhooks. This is not a Kubernetes service, but a Kubernetes deployment with side-car injected.

nats-streaming ---> (push app + envoy sidecar)  -----> (service B + envoy sidecar)

In the trigger for the webhook, I am passing all the tracing context as headers. 

I expect the tracing span to be created and show up in the UI and link with the parent span. I see that not happening.

Here are the headers that I send to service B

map[x-b3-traceid:[3fa64900c340f7b0404f7f29654075af] x-b3-flags:[1] x-request-id:[11e1e9d1-5e2a-42b2-825d-c8a32116b7df] x-b3-spanid:[782d1dd76ecee838] x-b3-parentspanid:[7e9f8b0911a17f73] x-b3-sampled:[1]]

I can see the same headers in the service B where the span ID is not updated. This implies to me that the envoy is not creating a new span and linking with the parent span.

Is there a way I can debug this issue? Any suggestions or pointers would be helpful.

Note: I can see the original trace and spans in the tracing UI, but not the span when service B is called.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/8644b57e-cd36-4f80-8f45-1076876c8596%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gary Brown]

Hi

The problem is due to the 128bit trace id which is not currently supported in Envoy. This results in it just returning a null span - possibly it would be better to create a new trace if the inbound trace context appears invalid?

Although it is mentioned here , there is currently no issue specifically to add 128bit trace id support. Would you mind creating one?

Regards
Gary