OpenTelemetry tracing is no longer sent from WildFly?

[Edgar Vonk]

Hi,

We used to have OpenTelemetry up and running from our WildFly 35.0.1 (bootable JAR) application where we sent OTLP tracing from WildFly to the OpenTelemetry (OTEL) Collector and then on to Tempo (and then visualise this data in Grafana). However this no longer works for our application. It seems that WildFly no longer sends any OpenTelemetry tracing whatsoever. We did not change any of our related WildFly configuration.

I even tried to use the same configuration as is documented here: https://github.com/wildfly/quickstart/tree/main/opentelemetry-tracing . E.g. in our WildFly configuration we set:

# CLI script to enable OpenTelemetry for the quickstart application in the application server
if (outcome != success) of /extension=org.wildfly.extension.opentelemetry:read-resource
    /extension=org.wildfly.extension.opentelemetry:add()
    /subsystem=opentelemetry:add()
end-if

/subsystem=opentelemetry:write-attribute(name=endpoint,value=http://${OTEL_COLLECTOR_HOST:localhost}:4317)
/subsystem=opentelemetry:write-attribute(name=exporter-type,value=otlp)
/subsystem=opentelemetry:write-attribute(name=sampler-type,value=on)
/subsystem=opentelemetry:write-attribute(name=max-export-batch-size,value=512)

(taken from: https://github.com/wildfly/quickstart/blob/main/opentelemetry-tracing/configure-opentelemetry.cli)

But also with this configuration no traces are sent to the OTEL Collector it seems (we have debugging on in the OTEL Collector).

Unfortunately I don't know when this started happening but my guess is that it broke with a WildFly upgrade in the past months or so. We upgrade WildFly regularly and try to be on the most recent version while we can. We are currently on WildFly 35.0.1. 

Any ideas? Could this be an issue in WildFly? Or is maybe additional configuration required for it to work again?

[Jason Lee]

Are there any log messages that might give hints? Have you added MicroProfile Telemetry to the deployment? Per the spec, OpenTelemetry is disabled by default when MP is available. That's the opposite of the default behavior of the main OpenTelemetry extension, but that's what the spec says, so that's what the server does. :) See this page for details: https://download.eclipse.org/microprofile/microprofile-telemetry-2.0/microprofile-telemetry-spec-2.0.html#sec:enabling-opentelemetry-support

Jason Lee

Principal Software Engineer

Red Hat JBoss EAP

Java Champion

--
You received this message because you are subscribed to the Google Groups "WildFly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/wildfly/de079dd0-bc9d-4e66-a4f1-1117579e20c7n%40googlegroups.com.

[Edgar Vonk]

Thanks a lot! It's working again. :-)

Still need to fine-tune our configuration. Am I right that in using the MicroProfile Telemetry extension we can / should now do all OpenTelemetry using this extension and no longer the 'main OpenTelemetry extension'?

[Jason Lee]

The answer is a very helpful "it depends". :) The only thing that MP Telemetry offers over the OpenTelemetry extension is the ability to configure otel via MicroProfile Config. In fact, all of the actual functionality for MicroProfile Telemetry in WildFly is handled by the OpenTelemetry extension, so all you gain by adding MP Telemetry is MP Config support. That said, you don't really lose anything by adding it (once you realize you have to re-enable otel support). If you need other MicroProfile libraries, I would just add the appropriate property and move on to more important tasks. However, if you don't need any MP support, you can remove the extension and not lose anything.

That's a long-winded way of saying that it depends on the needs of your application, but that's the answer. :)

Glad the fix was simple and you got it working, though! I like easy problems. :P

Jason Lee

Principal Software Engineer

Red Hat JBoss EAP

Java Champion

To view this discussion visit https://groups.google.com/d/msgid/wildfly/3bda1a7e-ae7a-405a-bff7-ea5ccdb917d5n%40googlegroups.com.

[Edgar Vonk]

Very clear. Thanks a lot! We did switch to the MP Telemetry extension even if it is does not offer anything new. At least it seems to be just a little bit more of an open standard than the OpenTelemetry extension and we do like open standards. ;-)

[Edgar Vonk]

The only thing is, there should be a law against negative variables.

I mean:

otel.sdk.disabled

really..? Why on earth not name this variable:

otel.sdk.enabled

so much easier to read..

[Jason Lee]

Ha! Agreed. Personally, I wouldn't have added the property at all and just assumed that if the library is present tracing (now metrics and logs) are desired, but that ship has sailed, sadly. :)

Jason Lee

Principal Software Engineer

Red Hat JBoss EAP

Java Champion

To view this discussion visit https://groups.google.com/d/msgid/wildfly/042e6167-5665-4820-a91b-c053d85ebfc8n%40googlegroups.com.