Micrometer, Observation API and OpenTelemetry

[Bruno Baptista]

Hi all,

In order to better align Quarkus with Observability concepts that require analysis of many signals (metrics, traces, ...) and origins, I've created a PoC at SmallRye OpenTelemetry to evaluate the integration of Micrometer, the Micrometer Observation API and OpenTelemetry.

The idea is to have Micrometer metrics and observations be transparently integrated with the OpenTelemetry SDK allowing:

- Create parent spans with the Observation API, create child spans with OpenTelemetry and vice-versa.

- Generate metrics with both Micrometer and OpenTelemetry independently.

- Generate metrics from Observed methods. 

- All outputs would be handled by the OpenTelemetry exporters using the OTLP protocol.

Micrometer is widely used in the Quarkus ecosystem and in the Java world but the OpenTelemetry Metrics API is fairly new and less intuitive. 

This way we could have the best of 2 worlds. Ease of use and wide support both in the instrumentation and in the exporter sides.

This is not a solution using a custom Micrometer OpenTelemetry registry, this is actually using the Micrometer API to instrument and create OpenTelemetry metrics, inside the SDK.

Using the Observation API has additional advantages:

- Automatically generated documentation, including the used semantic attributes.

- Instrument the code once with the Observation API and if a different instrumentation framework comes up in the future, we only need to add a new handler. 

- Instrumentation simplicity using synthetic sugar. There are many examples and probably a blog post would be worthwhile to compare the APIs.

To achieve this I created 3 PRs:

• Add support for OTel Metrics 
  
• Micrometer to Otel bridge 
  
• Observation to OpenTelemetry bridge 
  

Please mind, this is not production ready, there are many TODO in there and the project is more of a testbed. The important point is that all the pieces actually fit together. 

These PRs mimic the work that needs to happen in Quarkus, less additional requirements for context and scope propagation.

This will also allow the creation of a single Observability extension where you can instrument with your framework of choice and have a unified OTLP output for all signals. 

For this work I've had the help of the Micrometer community, especially from Marcin Grzejszczak. It's a joy to work with them. 

Please let me know what you think of this approach and if it's something you'd like to see in Quarkus.

Cheers!

Bruno Baptista

https://twitter.com/brunobat_

https://redhat.com

[Erin Schnabel]

Fantastic work, Bruno!

I think this is the right direction.

Thanks,
Erin
 
------
Erin Schnabel <ebul...@redhat.com>
Distinguished Engineer

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/7a822aca-c3f7-4125-b360-77c3ffcc7b28n%40googlegroups.com.

[Marco Bungart]

Sounds nice!

Do we already have plans how logs fit into all of this? As far as I know, Micrometer does not (yet?) deal with logs. Furthermore, there is this draft PR from Loic Mathieu [1]. So it seems that logs are handled separately. I had a discussion with Roberto Cortez a while back. The power of OTel is the common context between all parts. This means we need the same (metadata) context for all parts. Do we have ideas for a common metadata context?

[1]: https://github.com/quarkusio/quarkus/pull/38239

[Loïc MATHIEU]

Hi,

For logs I will rework my PR to integrate directly inside OTel as was discussed with Bruno.
Logs are really small regarding integration surface just a bridge to create between Quarkus logging backend and the OTel log.

Regards,

Loïc

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/0e96ff32-d4f5-4f83-b12e-80f7408a5422%40googlemail.com.

[Bruno Baptista]

Yes, Logs are planned and the existing PR sounds like a good platform to integrate it.

With Loic's permission I also plan to contribute to it.

Logs are a different problem from metrics as OpenTelemetry Logging in Java is not a new logging framework, just a carrier for logs produced by existing frameworks.

This should simplify everything (famous last words) because it should be part of the existing extension. Once implemented, it will be active by default. 

At a different level, @Ales Justin is working on this PR https://github.com/quarkusio/quarkus/pull/38448 with a new Grafana dev service that will be able to receive and view traces, metrics and logs. 

This will greatly simplify development and visualisation of the produced data. 

Cheers.

Bruno Baptista

https://twitter.com/brunobat_

https://redhat.com

You received this message because you are subscribed to a topic in the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quarkus-dev/uVg9BmUTC4o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVHOykVcqy2JrZ%2BuuH-_7AEH6EuQn8d9n-FvvvEGr-p8ww%40mail.gmail.com.

[Marco Bungart]

Does this integration also integrat the contexts between logging and OpenTelementry? And will this work with "legacy" code, i.e. when something is set through MDC, will it "automagically" available in the OTel context?

Cheers,
Marco

[Bruno Baptista]

That's the idea Marco. 

But it will not need that kind of tight integration. Once you place data in the MDC, the logging framework will pick it up when the log line is generated, Otel Logging will just get that log line with all it holds and potentially add more attributes on top of it. 

Bruno Baptista

https://twitter.com/brunobat_

https://redhat.com

You received this message because you are subscribed to a topic in the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quarkus-dev/uVg9BmUTC4o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/4c68b7bf-eb54-404a-b2d2-62f55fee1d54%40googlemail.com.