Unknown services after zipkin flush

[Tomasz Michnik]

io.zipkin.reporter2:zipkin-sender-okhttp3:2.7.13

io.zipkin.reporter2:zipkin-reporter:2.7.13

io.zipkin.finagle2:zipkin-finagle_2.12: 2.0.11

io.zipkin.finagle2:zipkin-finagle-http_2.12:2.0.11

While testing Zipkin we noticed that after some time, very strange spans occur in Zipkin server:

 {

    "traceId": "647a1056a8015c7a",

    "parentId": "2fc7a065ad3dc38a",

    "id": "01e3b03ac3a2a9b9",

    "name": "unknown",

    "timestamp": 1563454909360000,

    "localEndpoint": {

      "serviceName": "unknown",

      "ipv4": "127.0.0.1"

    },

    "annotations": [

      {

        "timestamp": 1563454909360000,

        "value": "finagle.flush"

      }

    ],

    "tags": {

      "clnt/response_payload_bytes": "3132"

    }

  }

It messed up completely  the UI as we can see mainly  "unknown" services.

Those spans are sent in zipkin2.finagle.SpanRecorder as they still exist in  spanMap after the deadline.

 void flush(Time deadline) {

        Iterator i = this.spanMap.values().iterator();

        while(i.hasNext()) {

            MutableSpan span = (MutableSpan)i.next();

            if (span.started().$less$eq(deadline)) {

                i.remove();

                span.addAnnotation(deadline, "finagle.flush");

                this.report(span);

            }

        }

    }

It turned up that those spans are just binary annotations generated by com.twitter.finagle.filter.PayloadSizeFilter.

The problem is that they are recorded right after other spans with a given traceId are sent to Zipkin server 

  if (span.isComplete()) {

            this.spanMap.remove(record.traceId(), span);

            this.report(span);

        }

and the map is cleared for that particular id.

In consequence finagle's binary annotations cannot be merged with other spans and wait in the map till the flush.

Any ideas how to tackle this issue?

[Brian Devins-Suresh]

Probably that filter needs to be moved earlier in the filter chain, if you care about what it is providing. Otherwise I would disable it completely.

--

---
You received this message because you are subscribed to the Google Groups "zipkin-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipkin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zipkin-user/53ab9acf-b234-427d-a8b8-0342e1778b91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adrian Cole]

The flush event causing "unknown" names can be handled with a
code/test. We have had similar issues in brave a while ago. This part
of the problem can be a github issue.

[Adrian Cole]

ps I agree with brian. even if we change to make flushed data more pretty it is still a config bug to create flushed data. the tracing filter should be the outer most

To view this discussion on the web visit https://groups.google.com/d/msgid/zipkin-user/CAO3-Ny4BZcOSM1--5Knz9Z_ejR-t_w9BJN31trxH82CeZhontA%40mail.gmail.com.

[Tomasz Michnik]

I am not sure if we can configure those filters.

As far understand the point is that  order of invocation of  PayloadSizeFilter and  ClientTracingFilter is wrong, right?

ClientTracingFilter - records information about  sent/received request and eventually markes spans as completed (which sends spans to the server)

They are probably added inside finagle ( com.twitter.finagle.client.StackClient, com.twitter.finagle.Http.Client )

It looks like it is  the internal configuration of finagle and we cannot influence this.

To unsubscribe from this group and stop receiving emails from it, send an email to zipki...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zipkin-user/53ab9acf-b234-427d-a8b8-0342e1778b91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "zipkin-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zipki...@googlegroups.com.

[Adrian Cole]

can you verify with finagle slack or gitter or email list? most things are configurable and if not may be an oversight

To unsubscribe from this group and stop receiving emails from it, send an email to zipkin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zipkin-user/e4a12c7a-7056-46ea-86e3-f3ba22b88a5d%40googlegroups.com.

[Tomasz Michnik]

Ok, I will try to do that.

[Adrian Cole]

zipkin-finagle 2.1.0 is out with the the couple in-flight changes. thanks for your patience, Tomasz!

[Tomasz Michnik]

I tackled the "Unknown services" with Finagle client configuration. 

Thanks for the info anyway;)