How to trace reconcilation activities with spans like in OpenTelemetry

[Erkan Erol]

Hi all,

Disclaimer: I am working for Red Hat and I am working on https://github.com/kubevirt. My question is about some challenges we have in the project but it is generic enough to discuss in a common email group.

I have an interesting question about k8s operators. You may have some answers or opinions about it. Feel free to state your experience/opinions.

Let's assume we have a product that consists of multiple k8s operators and they are working in a hierarchical way as below. 

When a user does change something in the top-level Custom Resource (A in the above), it propagates up to the bottom and leads to many changes in different layers. When you observe an activity (spec or status changes) in lower layers, it is not easy to understand where it comes from and what triggers those activities. Also, it is hard to measure performance of operators and monitor the lifecycle of changes.

Question: Do you know how to implement a tracing solution for this problem? Is there a way to track/visualize  reconciliation activities like spans in OpenTelemetry?

My opinion: Events can be useful but it is hard to correlate events without any request/operation id.  We can propagate a kind of request/operation id via spec+statuses but I am not sure about it.


Best,
Erkan

[da...@devigned.com]

Hi Erkan,

I think this proposal https://github.com/kubernetes/enhancements/pull/2312 attempts to solve this problem.  WDYT?

Cheers,

David

[Erkan Erol]

Hi,

@David thanks a lot for pointing this out.  Zvi Chana also shared some links with me.

I did some research and read the available resources. My findings are below. I am sharing them for documentation purposes.

My understanding is that some people have been thinking+discussing the problem and working on some possible solutions. However, there is no available solution yet in the open sources (I guess there are behind some corporate firewalls. )

Links:

Good blog post about the problem:

https://yue9944882.github.io/posts/how-to-monitor-kubernetes-operator-by-distributed-tracing/

One of the earliest KEPs about this problem. As far as I understand, the scope of the initial version was wider and contained some solutions for controller tracing but then its scope was narrowed. The good part is it is active.
https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/647-apiserver-tracing

This is the PR which adds the KEP above initially. If you want to understand this problem, I recommend you to read the comments there. Especially this one

https://github.com/kubernetes/enhancements/pull/650#issuecomment-461929747

As David pointed, this is a new KEP proposal which focuses on trace context propagation.  The PR is still open and it is not an approved KEP and there is no implemented solution.

https://github.com/kubernetes/enhancements/pull/2312

Best,

Erkan

--
You received this message because you are subscribed to a topic in the Google Groups "kubebuilder" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubebuilder/tNI6ZpQ2loM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubebuilder...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubebuilder/a234d66c-f052-4459-ac4d-0e49369a5273n%40googlegroups.com.

[Abby Bangser]

Hi all, just curious if there has been any more progression on this front? 

I reviewed these links and I can see the mentioned KEP (apiserver-tracing) has been delivered but it explicitly does not include tracing controllers (which is your use case for KubeVirt and mine for Kratix.io which is also built using Kubebuilder).

I am just starting my research so just wanted to touch base here to see if there is somewhere to collaborate!

Thanks,

Abby