Java Spring App and Prometheus Exemplar Support

[Jack Neely]

Friends,

I wrote about this issue some months ago when I was trying to put together a company wide demo and running out of time. My apologies for not following through.

I'm attempting to write a simple Java Spring framework app that serves as a working example of the metric and logging practices we'd like to follow as our standards. I've been trying for a while now to get exemplar support working to demo how to pivot from metrics to traces, but I've never been successful in getting exemplars to appear.

I've copied the example code to GitHub.

https://github.com/jjneely/java-exemplar-example

This is a super simple Gradle / Spring Boot app and running "./gradlew bootrun" should get it up and running. Log output will indicate if exemplar support is enabled and it should be as it runs with the OpenTelemetry Agent 1.9.1. However, using curl against the metric endpoint does not show exemplars and having Prometheus scrape the endpoint does not show exemplars.

$ curl -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' http://localhost:8081/

I've manually added exemplar values in the code, yet no luck. I feel like I'm missing something simple here. I'm hoping this issue is reproducible and someone can help me figure out what I've missed here.

I've also sent a note in the CNCF Slack:

https://cloud-native.slack.com/archives/C167KFM6C/p1641494335010700

Thanks for the help! Now that I'm not under a time crunch I promise to get back to folks.

Jack Neely

[Jack Neely]

Hey Fabian,

Thanks for the reply! I did check out the it_exemplars_otel_agent and followed the instructions here

https://github.com/prometheus/client_java/blob/master/OTEL_EXEMPLARS.md

for getting that up and working. I used Otel 1.9.1 to match my version and was able to see exemplars with this test code.

$ java -Dotel.traces.exporter=logging -Dotel.metrics.exporter=none -javaagent:./opentelemetry-javaagent.jar -jar ./exa
mple-spring-boot-app.jar
$ curl -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' http://localhost:8080/metrics | grep sp
an_id
requests_total{path="/hello"} 3.0 # {span_id="4f836ba5e1b35cc2",trace_id="0d02651c1faf5b6aeb9a2cabdcdc0009"} 1.0 1641568523.154
requests_total{path="/god-of-fire"} 3.0 # {span_id="d00084d7529b0881",trace_id="0d02651c1faf5b6aeb9a2cabdcdc0009"} 1.0 1641568523.278

So the test code definitely works. Now I'm trying to understand what's different in my code in GitHub so that I can reproduce these results. The only real things that stood out to me are:

* The hotspot default metrics were missing in my code -- I've added these, but this didn't fix exemplars
* I'm manually instrumenting exemplars in all but one Counter
* I'm using the Scheduler for repeated tasks rather than a REST-like app.

But I'm out of ideas about what to try next here and I'm looking for some assistance as to what the issue is with my example code. Again, that lives here: https://github.com/jjneely/java-exemplar-example

Jack Neely

On Thu, Jan 6, 2022, at 16:13, Fabian Stäber wrote:
> Hi Jack,
>
> You can find a working example in the it_exemplars_otel_agent
> integration test. The test runs the ExampleSpringBootApp in a Docker
> container with the OpenTelemetry agent attached.
> ExemplarsOpenTelemetryAgentIT verifies that Counter and Histogram
> metrics have Exemplars if they were observed in the context of a
> distributed trace.
>
> Fabian

>> --
>> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com <mailto:prometheus-developers%2Bunsu...@googlegroups.com>.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/73deecb0-da81-4970-ae20-66cc3543f7c1%40www.fastmail.com.

[Jack Neely]

I wanted to post some updates to this issue in case any one else runs into this.

Fabian found that Spring Boot implicitly includes "io.prometheus.:simpleclient_common:0.10.0" and I had not explicitly listed that implementation dependency in my Gradle build file. (Although I had listed several other simpleclient dependencies at 0.14.1.). The 0.10.0 version does not have Prometheus Exemplar support. But adding the following line to upgrade that dependency to 0.14.1 fixed the problem.

implementation 'io.prometheus:simpleclient_common:0.14.1'

From this point on I was able to see both the manually instrumented exemplars as well as OpenTelemetry autoinstrumented span_id and trace_id added by having the Agent loaded with:

curl -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' ...

Also, they appeared in my test Prometheus running on my laptop.

Thanks!
Jack Neely

> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to prometheus-devel...@googlegroups.com.