Exemplars in client_java not working

[jjn...@linuxczar.net]

I am building a demo application to show off Prometheus Exemplars and OpenTelemetry Tracing abilities with Java / Spring applications.  However, I cannot get exemplars to appear in the metrics HTTP endpoint.  I'm using the following curl command and I get metrics in the OpenMetrics format, but no exemplars.

    $ curl -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' http://localhost:8081/metrics

I've got the Prometheus client_java 0.12.0 loaded and I'm using that to directly instrument.  In build.gradle:

dependencies {
    ...
    implementation 'io.prometheus:simpleclient:0.12.0'
    implementation 'io.prometheus:simpleclient_hotspot:0.12.0'
    implementation 'io.prometheus:simpleclient_httpserver:0.12.0'
    implementation 'io.prometheus:simpleclient_tracer_otel:0.12.0'
    implementation 'io.prometheus:simpleclient_tracer_otel_agent:0.12.0'
    implementation('io.opentelemetry:opentelemetry-api:1.0.0')
    implementation('io.opentelemetry:opentelemetry-extension-annotations:1.0.0')
}

I setup my Prometheus metrics in the class constructor:

    prometheusErrors = io.prometheus.client.Counter.build()
      .namespace("custommetricsdemo")
      .name("errors")
      .help("Test Prometheus Client Library errors counter")
      .withExemplars()
      .register();
    prometheusTimer = io.prometheus.client.Summary.build()
      .namespace("custommetricsdemo")
      .name("latency_timer")
      .help("Test Prometheus Client Library latency summary")
      .quantile(0.5, 0.01)
      .quantile(0.95, 0.01)
      .register();
    prometheusHistogram = io.prometheus.client.Histogram.build()
      .namespace("custommetricsdemo")
      .name("histogram")
      .help("Test Prometheus Client Library latency histogram")
      .withExemplars()
      .register();
​
    try {
      io.prometheus.client.exporter.HTTPServer server
        = new io.prometheus.client.exporter.HTTPServer(8081);
    } catch (Exception e) {
      logger.error("Failed to setup Prometheus HTTP server", e);
    }

This code runs a ScheduledTask at a random interval which sleeps for a random time.  Thanks to running with the OpenTelemetry Agent, I can see trace_id and span_id in the logback logs.  That part works well.  I can also query the span IDs in code:

logger.info("{}, {}", span.getSpanContext().getSpanId(), span.getSpanContext().getTraceId());

I could not get the auto-exemplars to work, so I began to observe them in code:

prometheusErrors.incWithExemplar("span_id", span.getSpanContext().getSpanId(), "trace_id", span.getSpanContext().getTraceId());

I also tested with different exemplar tag names:

prometheusHistogram.observeWithExemplar(sw.getTotalTimeSeconds(), "span_foo", span.getSpanContext().getSpanId(), "trace_bar", span.getSpanContext().getTraceId());

However, no exemplars ever appear in the output.

$ curl -H 'Accept: application/openmetrics-text; version=1.0.0; charset=utf-8' http://localhost:8081/metrics
# TYPE custommetricsdemo_histogram histogram
# HELP custommetricsdemo_histogram Test Prometheus Client Library latency histogram
custommetricsdemo_histogram_bucket{le="0.005"} 3.0
custommetricsdemo_histogram_bucket{le="0.01"} 3.0
custommetricsdemo_histogram_bucket{le="0.025"} 3.0
custommetricsdemo_histogram_bucket{le="0.05"} 4.0
custommetricsdemo_histogram_bucket{le="0.075"} 4.0
custommetricsdemo_histogram_bucket{le="0.1"} 4.0
custommetricsdemo_histogram_bucket{le="0.25"} 6.0
custommetricsdemo_histogram_bucket{le="0.5"} 9.0
custommetricsdemo_histogram_bucket{le="0.75"} 11.0
custommetricsdemo_histogram_bucket{le="1.0"} 11.0
custommetricsdemo_histogram_bucket{le="2.5"} 11.0
custommetricsdemo_histogram_bucket{le="5.0"} 11.0
custommetricsdemo_histogram_bucket{le="7.5"} 11.0
custommetricsdemo_histogram_bucket{le="10.0"} 11.0
custommetricsdemo_histogram_bucket{le="+Inf"} 11.0
custommetricsdemo_histogram_count 11.0
custommetricsdemo_histogram_sum 2.732210076
custommetricsdemo_histogram_created 1.632231018649E9

What am I missing here?  Is something interfering with the Spring Boot libraries perhaps?

Jack Neely

[jjn...@linuxczar.net]

I'm still mystified by this problem.  Using "curl -v" I can tell that my Java app is responding in the OpenMetrics Version 1.0.0 format.  Contacts at Grafana have suggested removing with ".withExemplars()" on the metric object builders, but I get the same results with or without it.  I've also attempted to remove the Micrometer Prometheus support in this Spring Boot app in case that was conflicting with the native client_java.  Yet, no difference in output, exemplars are never present.

I can definitely provide full code for this demo, there's nothing really special here.  But I was hoping that I've just made a simple mistake, but I fear this may be more complex than that.  Any advice with how to get this demo app working and producing exemplars is appreciated.

Jack Neely

[Fabian Stäber]

Hi Jack,

I don't see any obvious mistakes, but if you provide some code to reproduce the error I'll have a look.

There's a unit test for `incWithExemplar()` here, can you see the difference between your code and the test? https://github.com/prometheus/client_java/blob/master/simpleclient/src/test/java/io/prometheus/client/CounterTest.java#L110-L144

If you have a way to reproduce this, ideally open an issue on https://github.com/prometheus/client_java then I'll get back to it.

Fabian

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/e3c3c6c6-ffda-4720-8bb6-a8b427b8c005n%40googlegroups.com.