[tracing] Knative serving and eventing tracing issues

73 views
Skip to first unread message

Andrey Stelmashenko

unread,
Jun 3, 2022, 8:33:38 AM6/3/22
to Knative Users
Hi Community,

I'm trying to reproduce this blog post https://knative.dev/blog/articles/distributed-tracing/ but using default knative capabilities in a bit other way.

My setup is simple, I have two services, broker and trigger:
Heartbit ksvc - is a service
EventDisplay ksvc - is another service

(ce) is cloudevent.
--(ce)--> - means sending cloud event

Heartbit ksvc --(ce)--> Broker --(ce)--> Trigger --(ce)--> EventDisplay ksvc

Heartbit ksvc is reusing incoming request headers, tracing headers injected by activator-service.

I configured tracing according to
Serving tracing - https://knative.dev/docs/serving/accessing-traces/
Eventing tracing - https://knative.dev/docs/eventing/accessing-traces/
For now I set debug: true for both configurations.

What I get is two separate traces:
Heartbit ksvc --(ce)--> Broker
Screenshot from 2022-06-03 15-31-50.png

and

Trigger --(ce)--> EventDisplay ksvc
Screenshot from 2022-06-03 15-31-24.png

If I look at the 2nd trace (starting from Trigger) then I see that there are no tracing headers and I see a warning (which says about it):
> invalid parent span IDs=8ae37449073c8462; skipping clock skew adjustment

My question is why tracing headers are not passed to Trigger?
Is it related to backend channel implementation which is used by default (it our case it is NatssChannel) used by broker and triggers?

vil...@us.ibm.com

unread,
Jun 3, 2022, 9:11:47 AM6/3/22
to Knative Users
It might indeed be related to NatssChannel not supporting tracing.

NatssChannel is unfortunately not a maintained backend implementation. If anyone is interested in contributing, please don't hesitate to contact us, either on Slack, email or DM me.

Lionel

Andrey Stelmashenko

unread,
Jun 7, 2022, 3:03:43 AM6/7/22
to Knative Users
Hey,
Do anybody know about other nats channels, nats and jetstream do those support tracing headers?

Andrey Stelmashenko

unread,
Jul 26, 2022, 4:34:59 AM7/26/22
to Knative Users
For those who may be interested, natss-channels based broker did not support tracing header propagation and exporting. This has been fixed by PR https://github.com/knative-sandbox/eventing-natss/pull/342 
It is not released yet (v1.4). The next release will include it.

Reply all
Reply to author
Forward
0 new messages