Determining if a trace is incomplete

[Kamala Ramasubramanian]

Hi, 

I'm working on aggregate trace analysis for reasoning about system behavior and would like to eliminate incomplete traces from my corpus. Some of the traces drop spans some of the time and I want to be able to identify this as such when it happens. Before I try to do that, I am looking to understand how Zipkin decides that all spans corresponding to a given traceID are in. I looked at the docs a little bit but did not find any information. To repeat, my question is: 

How does Zipkin decide when all the spans corresponding to a trace are in and send it to the collector, which persists the traces?

Thanks,

Kamala

[Ben Plotnick]

Spans are immediately sent to the collector (modulo batching), which might do some preliminary processing and persist them to storage. You can somewhat tell if a trace is incomplete by collecting all spans given a trace id, and constructing a tree of spans linking spans with parent span id of X to the span with that has an id of X. If there are disjoint parts of that tree, then your trace is definitely incomplete. Otherwise, there’s no way to tell if a trace is complete or not. E.g if you have a delayed span that is a “leaf node” of the tree, you do not know about it. Same goes if there is an entire subtree that is missing. For root nodes, you can tell because it should have a null parent span id.

Your best bet is to wait some amount of time and/or do what the stream processing people do and update your results as new data comes in.

--

---
You received this message because you are subscribed to the Google Groups "zipkin-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipkin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zipkin-user/cda22163-9acd-4050-9273-2441416088c3%40googlegroups.com.

[Adrian Cole]

here's some code about the heuristics ben mentioned
https://github.com/adriancole/zipkin-voltdb/blob/master/storage/src/main/java/zipkin2/storage/voltdb/procedure/CompletePendingTraces.java#L127

I think Jorge maybe has some similar code somewhere, but what Ben said
is correct. You can get a good confidence a trace is complete,
heuristically, but you can't be sure late data isn't added. This means
anything you do must be reprocessable if any late data happens.

-A

> To view this discussion on the web visit https://groups.google.com/d/msgid/zipkin-user/CAAE4cEF1Zi-hDfWp%2BhRrJMW8topMcXUVp%2BOJDR7PgsDYetHkPA%40mail.gmail.com.