microprofile.io opentracing spec - reboot

[Steve Fontes]

Hi All,

I really think that supporting opentracing.io compliant Tracers in microprofile.io will be a big benefit for developers and particularly for operators in microservice environments.

Moving forward to agree on a specification for opentracing in microprofile.io, we need to resolve some issues.

The current spec proposes the following API:
    @Trace
    @Notrace

    The ability to inject an io.opentracing.Tracer.

The current spec also proposes some non-API requirements, which are at least as important. See https://github.com/eclipse/microprofile-opentracing/blob/master/spec/src/main/asciidoc/microprofile-opentracing-spec.asciidoc
   
Since last time we discussed this here, there have been some developments.

At https://github.com/opentracing-contrib/java-jaxrs:
    I believe the API implemented is:

        @Traced(operationName=<override of operation name>) -- only used to change span name from default, all server entry points will be traced

        Access to the io.opentracing.Tracer is through utility io.opentracing.util.GlobalTracer.get().
        You can inject an io.opentracing.contrib.jaxrs2.server.TracingContext through @BeanParam TracingContext
        TracingContext allows access to a SpanContext stored in an incoming request attribute.
       
        Container and Client request/response filters are provided that will automatically trace incoming and outgoing requests if configured correctly.
        Tracing can be disabled for outgoing requests by adding .property(TracingProperties.TRACING_DISABLED, true) when the request is made.
        I don't see a way to disable tracing of incoming requests.
       
        Default Span name for incoming request is: requestContext.getMethod()
        Default Span name for outgoing request is: requestContext.getMethod()
        Tags on all incoming requests are at least: (Tags.SPAN_KIND.getKey(), Tags.SPAN_KIND_SERVER)
        Tags on all outgoing requests are at least: (Tags.SPAN_KIND.getKey(), Tags.SPAN_KIND_CLIENT)

        Requires developers to explicitly add some code to their applications.
       
At https://cwiki.apache.org/confluence/display/CXF20DOC/Using+OpenTracing
    I believe the API implemented is:
   
        No annotations that I could see.
       
        Access to the io.opentracing.Tracer is through utility io.opentracing.util.GlobalTracer.get()
        You can inject an org.apache.cxf.tracing.TracerContext through @Context annotation
        TracerContext allows access to Span operations
       
        Container and Client request/response filters are provided that will automatically trace incoming and outgoing requests if configured correctly.
        I don't see a way to disable tracing of incoming requests or outgoing requests.
       
        Span name for incoming request is: requestContext.getMethod() requestContext.getUriInfo().getRequestUri()
        Span name for outgoing request is: requestContext.getMethod() requestContext.getUriInfo().getRequestUri()
        Tags on all incoming requests are at least: (Tags.HTTP_METHOD.getKey(), method),(Tags.HTTP_URL.getKey(), uri.toString()),(Tags.HTTP_STATUS.getKey(), responseStatus)
        Tags on all outgoing requests are at least: (Tags.HTTP_METHOD.getKey(), method),(Tags.HTTP_URL.getKey(), uri.toString()),(Tags.HTTP_STATUS.getKey(), responseStatus)
       
        Support for JAX-WS is also provided.
        Requires developers to explicitly add some code to their applications.
       
I think the microprofile.io spec should be basically as follows:
    Access to the io.opentracing.Tracer is through @Inject io.opentracing.Tracer
    @Trace becomes @Traced -- might as well make it match the io.opentracing JAX-RS project.
        @Traced(value="true", operationName=<override of operation name>)
   
    I think we need to be able to turn off tracing for some methods that would be traced by default.
    I propose @Trace(false) instead of @Notrace. I think I saw this option in something by Pavol Loffay, but I can't find the reference now.
    Need to specify a way to disable tracing for outgoing requests. Only way I have seen so far is by adding .property(TracingProperties.TRACING_DISABLED, true). I can't think of a way to accomplish this with an annotation.
   
    I propose specifying default Span names and tags as:
        Default Span name for incoming request is: requestContext.getMethod() requestContext.getUriInfo().getRequestUri()
        Default Span name for outgoing request is: requestContext.getMethod() requestContext.getUriInfo().getRequestUri()
        Tags on all incoming requests are at least: (Tags.SPAN_KIND, Tags.HTTP_METHOD, Tags.HTTP_URL, Tags.HTTP_STATUS, Tags.ERROR if true)
        Tags on all outgoing requests are at least: (Tags.SPAN_KIND, Tags.HTTP_METHOD, Tags.HTTP_URL, Tags.HTTP_STATUS, Tags.ERROR if true)
   
    Also, the spec will make it clear that developers will not be required to add any code to their applications to get basic distributed tracing support of inbound and outbound requests.

    I'll make the changes to the spec and API's and we can continue to move this proposal along.

Steve

[Mike Croft]

HI Steve, great to see this getting attention again! I've taken the liberty of creating a room in Gitter to help discuss things more. I know there are some guys in Payara interested in OpenTracing, so I'll point them in this direction!

Here's the chat room:

https://gitter.im/eclipse/microprofile-opentracing

[Gary Brown]

Hi Steve

Thanks for rebooting this thread!

One comment about span name, as mentioned in the opentracing spec (https://github.com/opentracing/specification/blob/master/specification.md):

• An operation name, a human-readable string which concisely represents the work done by the Span (for example, an RPC method name, a function name, or the name of a subtask or stage within a larger computation). The operation name should be the most general string that identifies a (statistically) interesting class of Span instances. That is, "get_user" is better than "get_user/314159".

That is why the default operation/span name that tends to be used in tracing instrumentation is the HTTP method. Not very descriptive, but general.

Regards

Gary

--
You received this message because you are subscribed to the Google Groups "Eclipse MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/cb4e65c9-02be-40ba-b034-33da64c3fb34%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Steve Fontes]

Hi Gary,

But in a typical UI for displaying Spans wouldn't you just see"
GET
   GET
   GET
      GET

or something like that, which seems not very useful to me.
I'm not sure if knowing all of the Spans that were caused by a GET is statistically interesting. What could you do with that knowledge? Whereas knowing all of the Spans to a particular URL would be interesting, for comparison amongst all URLs for example.

So, a couple of questions:
1. Should the microprofile.io spec for opentracing define the default Span name for Spans? I say Yes.
2. If Yes for 1, should the microprofile.io spec for opentracing use Method or Method+URL for the default Span name? I say Method+URL, as Method alone is statistically not interesting.

Steve

On Monday, October 2, 2017 at 6:11:02 AM UTC-4, Gary Brown wrote:

...

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

[Gary Brown]

Hi Steve

I totally agree method on its own is not interesting - but at the same time, if we have URLs that include path parameters, they are also not useful.

As indicated in the spec - the ideal would be some meaningful names - and in terms of URLs with path parameters, the URL template would be perfect - the problem is how to access it.

Regards

Gary

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/f2192607-87a3-4a5d-8fea-f2d0c0c9fcba%40googlegroups.com.

[Steve Fontes]

OK, I'm coming around now.
Over in https://gitter.im/eclipse/microprofile-opentracing, Pavol Loffay suggested this:
https://github.com/opentracing-contrib/java-jaxrs/blob/master/opentracing-jaxrs2/src/main/java/io/opentracing/contrib/jaxrs2/server/ServerSpanDecorator.java#L61

Which I now see is what you are suggesting. It makes a lot of sense to me.

I'd like to ask kind of a meta-question. Should default Span names be part of the specification, or is that something that should be left to each implementation?

Steve

[Gary Brown]

Hi Steve

If the aim is to leverage the opentracing-contrib framework instrumentations, which I think would be good to avoid unnecessary duplication, then it is probably better not to define a default in this spec - and instead where possible work with the OT community to improve any names that are too generic.

Regards

Gary

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/ba991b6f-e1f7-46c3-ac3a-a5caa5becc87%40googlegroups.com.

[Heiko Rupp]

Define in the Spec what is possible and sensible. Even if it sort of repeats the OT spec.

We had in Java(2)EE too many of those voids, where things are left to the vendor and no two vendors were able to remotely do the same.

[Steve Fontes]

I've created a pull request with changes to the spec. -- https://github.com/eclipse/microprofile-opentracing/pull/5 --
We have to get some agreement on that, then have to get agreement how to write a tck.

So, another meta question.
    This spec not only defines an API for developers to use, the spec also defines some behaviors that a microprofile.io implementation needs to support.
    That is, automatic Span creation and propagation with no changes to application code.
    Is that appropriate for the microprofile.io specification process?
    Is this process just intended to define API's, or is it ok to specify requirements outside of the development realm? Things that make ops easier.

For the tck, we'll have to test that the APIs are implemented as expected, but also that the automatic behaviors are implemented as expected.

Any comments on a good way to do that? Each implementation will probably have a different way of specifying the Tracer implementation to use.
The Tracer implementation needs to be made available even for apps that have no opentracing code added.

I was thinking of providing a TestTracer implementation as part of the tck, then an implementation would need to do some implementation specific
configuration to make that TestTracer available to an app that has no opentracing code added.

There would be some capability to query the TestTracer to make sure the implementation cause the expected Spans to be produced.

So, ready for comments if anyone has any ideas.

Steve

[John D. Ament]

I see that Open Tracing already have a Java API - https://github.com/opentracing/opentracing-java/tree/master/opentracing-api

I'm not sure how widely used it is, but does it make sense that we're leveraging that and saying that implementations support that API?

John

[Gary Brown]

Hi Steve

On Fri, Oct 6, 2017 at 6:50 PM, Steve Fontes <steve.m...@gmail.com> wrote:

I was thinking of providing a TestTracer implementation as part of the tck, then an implementation would need to do some implementation specific
configuration to make that TestTracer available to an app that has no opentracing code added.

There would be some capability to query the TestTracer to make sure the implementation cause the expected Spans to be produced.

There is already a MockTracer that can be used in tests.

Regards

Gary

[Heiko Rupp]

Am Freitag, 6. Oktober 2017 19:50:23 UTC+2 schrieb Steve Fontes:

So, another meta question.
    This spec not only defines an API for developers to use, the spec also defines some behaviors that a microprofile.io implementation needs to support.
    That is, automatic Span creation and propagation with no changes to application code.
    Is that appropriate for the microprofile.io specification process?
    Is this process just intended to define API's, or is it ok to specify requirements outside of the development realm? Things that make ops easier.

I am totally +1 on defining requirements for ops things. Leaving (too much) stuff as  vendor defined is bad, as this will defrag the MicroProfile implementation space.

If the only thing that the MP-OT spec defines is "look at the OT spec", then no MP-OT spec is needed at all.

The beauty of OT is that it is generic. But that is also the danger. 

[Steve Fontes]

The microprofile,io spec for opentracing is slightly different than the java API defined in https://github.com/opentracing/opentracing-java/tree/master/opentracing-api. The opentracing/opentracing-java API is an API where everything is included as part of the application itself. So, when you define a Tracer in your application, you can use something like GlobalTracer.get() to access the Tracer wherever you need it. But in the microprofile.io spec, The application can take advantage of Tracer function, without actually having to provide the Tracer implementation, or the JAX-RS filters that manage Spans, as part of the application. The microprofile.io implementation provides a mechanism to define a Tracer for the application, and the microprofile.io implementation provides a mechanism for creating and propagaing Spans for network requests. The opentracing/opentracing-java spec can't be dropped wholesale into a microprofile.io implementation to provide support.

Steve

[John D. Ament]

Agreed, and this is where I think we need to draw the line:

- Yes, it may use the opentracing APIs (rather than define its own)

- What API needs to be supplemented in MP.

Is that a reasonable approach?  For instance, you're using a tracer example.  I would imagine you're just specifying the filters etc.

John

[Steve Fontes]

Hey John,

Sorry, I misunderstood your original comment. When I read "Open Tracing already have a Java API" I read it as "Open Tracing already have a JAX-RS API", even though you provide the link. I've been too immersed in JAX-RS lately.

So, Microprofile.io WILL support the Opentracing Java API defined at https://github.com/opentracing/opentracing-java/tree/master/opentracing-api. The support will be that an application can get access to an io.opentracing.Tracer object,
and use the opentracing defined Java API's on that Tracer Object.

One part of the microprofile.io spec that is not defined in the opentracing Java spec is that in microprofile.io implementations, the Tracer will be available through "@Inject io.opentracing.Tracer myTracer".

A second part of the microprofile.io spec that somewhat matches the opentracing JAX-RS spec, is that an @Traced annotation can be used to indicate a method should have a Span created for it.

The rest of the microprofile.io spec defines microprofile.io implementation behaviors that are not related to developer code. This part of the spec defines implementation behavior for JAX-RS applications, when the application has no opentracing code added by developers.

Most likely, the implementation of the behaviors for JAX-RS applications will be very similar to the implementation at https://github.com/opentracing-contrib/java-jaxrs, but the microprofile.io specification does not require an exact match with what is available in the opentracing JAX-RS implementation.

Steve

Steve

[John D. Ament]

Perfect, sounds like a reasonable solution.

BTW, do you plan to setup a reoccurring call on OpenTracing? May be useful to help corral the community together on this.

John

[Steve Fontes]

Hi Gary,

I'm trying to get a handle on how to write a tck that tests the microprofile.io implementation, where the Tracer is included in the implementation, not in the application.
(one of the requirements of the spec is to allow tracing of apps where no opentracing code has been added to the app itself)
I think this means we can't include the Tracer in the shrinkwrap war file. So, I don't think we can access the Tracer methods directly from the tck.

My idea - which may be kind of screwy - is to wrap the MockTracer with a TestTracer that overrides the toString() method to return the value of the finishedSpans() method as JSON.
Then the tck test can call a URL in the app to get the state of the Tracer when needed, because the app can access the Tracer with @Inject Tracer.

Steve

[Steve Fontes]

Hi John,
I'm going to be on vacation for two weeks starting Saturday, so I'll start a call when I get back.
Steve

[Emily Jiang]

Steve,
You can create a doodle page this week to gather people's availability and then schedule the recurring call, starting in two weeks' time.

Emily

[Steve Fontes]

Poll is at:

https://doodle.com/poll/a2s9i672myiyupy7

to have first call at the start of November.
Steve

[Cesar Saavedra]

I joined the call today and there was nobody on the bridge.  When will the calls start for open tracing?

[Steve Fontes]

Sorry Cesar,
I thought the meetings had been scheduled for every other week, so I missed calling in last week. I'm trying to get tomorrow's call rescheduled to Monday also, so be aware of that. After the US Thanksgiving holiday, I hope to have a more stable pattern to the meetings.
Steve Fontes