Micrometer, Observation API and OpenTelemetry roadmap

[Bruno Baptista]

Hi all,

This is an overview of the observability work planned for Quarkus and also a follow up to the previous email with the subject "[quarkus-dev] Micrometer, Observation API and OpenTelemetry".

In the previous email I sketched the vision and a set of validation PoCs implemented under the SmallRye OpenTelemetry library. It turns out that it was possible to implement a bridge from Micrometer to OpenTelemetry Metrics and also a bridge between the newer ObservationAPI and OpenTelemetry Metrics and Tracing.

This reinforces a vision where you can use any of the main metrics APIs in Java and handle the data with the OpenTelemetry SDK allowing export from there, in a standard way with the OTLP protocol. 

—

We want to support all the OpenTelemetry signals and unify the output of Tracing, Metrics and Logging, regardless of what monitoring/observability libraries you are using.

—

There are additional advantages with this strategy. 

Right now only green field projects use OpenTelemetry Metrics and everyone will have their own feeling of what APIs are more convenient. The applications of our user base and Quarkus own automatic instrumentation are implemented with Micrometer. We aim to prevent the need for everyone to reimplement this functionality. Therefore, this bridge will enable us and current users to reuse existing code while integrating with the OpenTelemetry Metrics API as desired.

Also to be aware…

We will not perfectly align Quarkus automatic metrics with the OpenTelemetry semantic conventions, for the time being. There are many changes happening on OpenTelemetry right now and we want to shield us from too many changes.

We plan to move the micrometer-registry-prometheus extension to Quarkuiverse and mark the current one as deprecated.

We are seizing the fact that Micrometer 1.13 has breaking changes, related with the Prometheus Java client 0.x to 1.x upgrade, to serve as the 1st version to place on Quarkiverse. The core extension will remain with Micrometer 1.12.x until it's removed.

We will promote the Micrometer OTLP registry to core, for projects not using the quarkus-openelemetry extension and implement the mentioned Micrometer to OpenTelemetry bridge to serve projects using both extensions. 

The quarkus-opentelemetry extension has a supported status and the new metrics and logging features will be experimental.

To conciliate these 2 states, I'm planning to set metrics and logging as disabled by default.

The other option would be to create a new quarkus-open-telemetry-experimental that would contain that code until it's promoted and moved to the main extension.

Please let us know what you prefer.

We've been holding the upgrade of OpenTelemetry because of breaking changes with the HTTP semantic conventions, only affecting tracing, in the current version.

We will upgrade soon and end the current transition period. Please check this previous email.

In order to fulfill this vision in Quarkus we plan to proceed with the tasks in the following order:

Component	Description	Status
Grafana devservice	Add a way to easily see traces, metrics and logs while developing	Merged. Available on Quarkus 3.11. https://quarkus.io/version/main/guides/observability-devservices-lgtm
OpenTelemetry Metrics	Allow the users to create metrics with OpenTelemetry. Disabled by default.	PR available. https://github.com/quarkusio/quarkus/pull/39032
OpenTelemetry upgrade	We've been on OpenTelemetry 1.32.x for a while now because of semantic conventions breaking changes. We will move to 1.38.x.	No Started. See explanation here.
OpenTelemetry Logging	Allow to forward existing logs to be forwarded with OpenTelemetry. Disabled by default.	Draft PR available. https://github.com/quarkusio/quarkus/pull/38239 Parked for now because we need the senders that will become available after OTel Metrics are merged.
micrometer-registry-prometheus	Create a new extension on Quarkiverse based on Micrometer 1.13+. Deprecate current core extension and later remove it.	Not Started.
Micrometer metrics to OpenTelemetry metrics bridge	Use Micrometer produced metrics in the OpenTelemetry SDK.	Not Started. PoC available here.
Micrometer Observation API to OpenTelemetry bridge	Use metrics and traces produced by the Observation API on the OpenTelemetry SDK	Not Started. Poc available here.

Please let us know if you agree or disagree.

We are also open for contributions :)

Thanks for reading until the end.

Cheers!

Bruno Baptista

https://twitter.com/brunobat_

https://redhat.com

[Max Rydahl Andersen]

We plan to move the micrometer-registry-prometheus extension to
Quarkuiverse and mark the current one as deprecated.

Do you mean mark deprecated and copy to quarkiverse or just move to quarkiverse and then relocate the existing GAV?

The quarkus-opentelemetry extension has a supported status and the new
metrics and logging features will be experimental.

To conciliate these 2 states, I'm planning to set metrics and logging as
disabled by default.

The other option would be to create a new
quarkus-open-telemetry-experimental that would contain that code until it's
promoted and moved to the main extension.

My first hunch is having it disabled by default seems simpler.

If we do the -experiemental extension then that artifact jar will have duplicate packages (I assume?)
and have to stay around for easy updates.

Please let us know what you prefer.

We've been holding the upgrade of OpenTelemetry because of breaking changes
with the HTTP semantic conventions, only affecting tracing, in the current
version.

What are those breaking changes? inside opentelemetry?

/max

We will upgrade soon and end the current transition period. Please check
this previous email

<https://groups.google.com/g/quarkus-dev/c/MsU_KWwKgoo/m/klx-yLwDAQAJ> .

In order to fulfill this vision in Quarkus we plan to proceed with the
tasks in the following order:

Component

Description

Status

Grafana devservice

Add a way to easily see traces, metrics and logs while developing

Merged. Available on Quarkus 3.11.

https://quarkus.io/version/main/guides/observability-devservices-lgtm

OpenTelemetry Metrics

Allow the users to create metrics with OpenTelemetry. Disabled by default.

PR available.

https://github.com/quarkusio/quarkus/pull/39032

OpenTelemetry upgrade

We've been on OpenTelemetry 1.32.x for a while now because of semantic
conventions breaking changes. We will move to 1.38.x.

No Started.

See explanation here
<https://groups.google.com/g/quarkus-dev/c/MsU_KWwKgoo/m/klx-yLwDAQAJ> .

OpenTelemetry Logging

Allow to forward existing logs to be forwarded with OpenTelemetry. Disabled
by default.

Draft PR available.

https://github.com/quarkusio/quarkus/pull/38239

Parked for now because we need the senders that will become available after
OTel Metrics are merged.

micrometer-registry-prometheus

Create a new extension on Quarkiverse based on Micrometer 1.13+. Deprecate
current core extension and later remove it.

Not Started.

Micrometer metrics to OpenTelemetry metrics bridge

Use Micrometer produced metrics in the OpenTelemetry SDK.

Not Started.
PoC available here

<https://github.com/smallrye/smallrye-opentelemetry/tree/main/implementation/micrometer-otel-bridge>
.

Micrometer Observation API to OpenTelemetry bridge

Use metrics and traces produced by the Observation API on the OpenTelemetry
SDK

Not Started.
Poc available here.

<https://github.com/smallrye/smallrye-opentelemetry/tree/main/implementation/observation-otel-bridge>

Please let us know if you agree or disagree.

We are also open for contributions :)

Thanks for reading until the end.

Cheers!

Bruno Baptista

https://twitter.com/brunobat_

https://redhat.com

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAPC_VG%3DsFyFijyeScMWaZKzr-a5d1WmXvF%2BGfnWbOnebUd%2B%3DBw%40mail.gmail.com.

[Bruno Baptista]

We plan to move the micrometer-registry-prometheus extension to
Quarkuiverse and mark the current one as deprecated.

Do you mean mark deprecated and copy to quarkiverse or just move to quarkiverse and then relocate the existing GAV?

Mark it as deprecated and leave it be in core, using Micrometer 1.12.x until it is deleted.

At the same time implement on Quarkiverse the new version of micrometer-registry-prometheus using Micromenter 1.13.x... The one that has breaking changes.

It just feels like a nice pointcut to do this. 

 

The quarkus-opentelemetry extension has a supported status and the new
metrics and logging features will be experimental.

To conciliate these 2 states, I'm planning to set metrics and logging as
disabled by default.

The other option would be to create a new
quarkus-open-telemetry-experimental that would contain that code until it's
promoted and moved to the main extension.

My first hunch is having it disabled by default seems simpler.

If we do the -experiemental extension then that artifact jar will have duplicate packages (I assume?)
and have to stay around for easy updates.

We can minimize the packages overlap, but it will be hard to eliminate them completely.

Please let us know what you prefer.

We've been holding the upgrade of OpenTelemetry because of breaking changes
with the HTTP semantic conventions, only affecting tracing, in the current
version.

What are those breaking changes? inside opentelemetry?

It's about semantic conventions. HTTP semantinc conventions have changed, next will be messaging and JDBC.

This affects the conventions for the attribute names and what they should contain, for all signals. 

There are other API breaking changes and general shuffle of packages between libraries... But those we general absorb them and the users don't feel it, however, semantic conventions are a pain because they will affect the dashboards.

[Max Rydahl Andersen]

>> We plan to move the micrometer-registry-prometheus extension to
>> Quarkuiverse and mark the current one as deprecated.
>>
>> Do you mean mark deprecated and copy to quarkiverse or just move to
>> quarkiverse and then relocate the existing GAV?
>>
> Mark it as deprecated and leave it be in core, using Micrometer 1.12.x
> until it is deleted.
> At the same time implement on Quarkiverse the new version of
> micrometer-registry-prometheus using Micromenter 1.13.x... The one that has
> breaking changes.
> It just feels like a nice pointcut to do this.

mkay.

but if micrometer 1.12.x is still in core the quarkus platform bom will lock it to 1.12.x
and platform will need to include quarkiverse variant and bump to 1.13.x.

Won't that be problematic?

[Bruno Baptista]

>> We plan to move the micrometer-registry-prometheus extension to
>> Quarkuiverse and mark the current one as deprecated.
>>
>> Do you mean mark deprecated and copy to quarkiverse or just move to
>> quarkiverse and then relocate the existing GAV?
>>
> Mark it as deprecated and leave it be in core, using Micrometer 1.12.x
> until it is deleted.
> At the same time implement on Quarkiverse the new version of
> micrometer-registry-prometheus using Micromenter 1.13.x... The one that has
> breaking changes.
> It just feels like a nice pointcut to do this.

mkay.

but if micrometer 1.12.x is still in core the quarkus platform bom will lock it to 1.12.x
and platform will need to include quarkiverse variant and bump to 1.13.x.

Won't that be problematic?

The idea is moving away from micrometer-registry-prometheus altogether. 

When I say deprecated on core, it means to move it out of the platform as well and be replaced by the otlp registry. 

The newer versions of the prometheus DB support services pushing data with the OTLP protocol. See: https://prometheus.io/docs/prometheus/latest/feature_flags/#otlp-receiver

From our perspective, micrometer-registry-prometheus will become a legacy solution and therefore, moved to Quarkiverse. 

[George Gastaldi]

Maybe we should have a quarkus-deprecated organization for extensions that are no longer maintained. Moving them to Quarkiverse doesn't feel right as it is supposed to be community-engaged.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAPC_VGkYUtWB-eYe_LGgv6abkptBBZ48iG82VMhFqDZ3mVK0mA%40mail.gmail.com.

[Bruno Baptista]

I think, like the other Micrometer registries, we want to keep supporting the Prometheus one, just not as a platform extension. 

It will not be deprecated and forgotten. 

Bruno Baptista

https://twitter.com/brunobat_

https://redhat.com

[Max Rydahl Andersen]

It's completely natural for things to be deprecated.

 having such be in a place where Community engagement can happen so those who have interest and ability can participate seems to be fitting. Community engagement isn't just for all new shiny things :)

/max

https://about.me/maxandersen

[Erin Schnabel]

We can move up to 1.13 with the prometheus registry in core: we have to change our internal dependency to *-simpleclient to avoid the breaking API changes and continue moving forward with fixes etc.

It should be noted that the micrometer team did this because of where prometheus is going re: supported APIs. 

1) prometheus meter registry in core —> micrometer meter registry prometheus simpleclient 1.13… (avoids breaking changes)

2) quarkiverse prometheus registry -> micrometer prometheus registry (proper) 1.13.x

Users have to opt-into the breaking API change if they use prometheus APIs directly (moving to quarkiverse)

Given how often we’re being asked about Otel doing all the things, I think this shift is inevitable.. 

Thanks,
Erin
 
------
Erin Schnabel <ebul...@redhat.com>
Distinguished Engineer

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAPC_VGkYUtWB-eYe_LGgv6abkptBBZ48iG82VMhFqDZ3mVK0mA%40mail.gmail.com.

[Bruno Baptista]

Hi,

Just a quick update on the Observability work.

OpenTelelemetry Metrics will be available with experimental status stating on next Quarkus 3.13.

The OpenTelemetry semantic conventions have also been updated and include breaking changes. 

Also, the Observability Dev Service is getting a Dev UI Card, soon: https://github.com/quarkusio/quarkus/pull/41281

For more details, please check the Quarkus migration guide for the upcoming Quarkus 3.13: https://github.com/quarkusio/quarkus/wiki/Migration-Guide-3.13 

Updated table: 

Component

Description

Status

Grafana devservice

Add a way to easily see traces, metrics and logs while developing

Merged. Available on Quarkus 3.11.

https://quarkus.io/version/main/guides/observability-devservices-lgtm 

OpenTelemetry Metrics 

Allow the users to create metrics with OpenTelemetry. Disabled by default.

Merged.

Available on Quarkus 3.13.

https://github.com/quarkusio/quarkus/pull/39032

OpenTelemetry upgrade

We've been on OpenTelemetry 1.32.x for a while now because of semantic conventions breaking changes. We will move to 1.38.x.

Merged.

Available on Quarkus 3.13.

See explanation here.

https://github.com/quarkusio/quarkus/pull/41521

OpenTelemetry Logging

Allow to forward existing logs to be forwarded with OpenTelemetry. Disabled by default.

Draft PR available.

https://github.com/quarkusio/quarkus/pull/38239

Parked for now because we need the senders that will become available after OTel Metrics are merged.

micrometer-registry-prometheus

Create a new extension on Quarkiverse based on Micrometer 1.13+. Deprecate current core extension and later remove it.

Not Started.

Micrometer metrics to OpenTelemetry metrics bridge

Use Micrometer produced metrics in the OpenTelemetry SDK. 

Not Started.
PoC available here.

Micrometer Observation API to OpenTelemetry bridge

Use metrics and traces produced by the Observation API on the OpenTelemetry SDK

Not Started.
Poc available here.

 

Bruno Baptista

https://twitter.com/brunobat_

https://redhat.com

[Bruno Baptista]

Hi

Updates on the Observability roadmap.

OpenTelemetry Logging is now merged as a preview and will be available on upcoming Quarkus 3.16.

There is also news about the Grafana LGTM dev service. It now includes dashboards that will display Micrometer metrics data sent by Prometheus or OTLP registries. 

As an attachment, you can see the status of the roadmap. For some reason, google groups doesn't like my tables. 

Regards

Bruno Baptista

[Georgios Andrianakis]

Congrats to all the folks that worked on these features!

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/790c4162-19a3-4575-934f-ca8dab342557n%40googlegroups.com.

--

Georgios Andrianakis

Independent Contractor

[Bruno Baptista]

Absolutely, 

Special thanks to Loïc Mathieu for opening and contributing to the OpenTelemetry Logging PR.

Also to Melloware for the Grafana LGTM dashboards PR.

Cheers!

Bruno Baptista

https://twitter.com/brunobat_

https://redhat.com

You received this message because you are subscribed to a topic in the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/quarkus-dev/y5-ojIVsa_M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-moOPCE4zuSJgQTXeBMh4Sb2MJN2iYPy9KbFAVeRqO2%3Dg%40mail.gmail.com.