Unable to export metrics when WildFly is initializing

[Ehsan Zaery Moghaddam]

Hi

We have enabled the open tracing system and we noticed that when the WildFly is in the startup process, if the Prometheus server calls the /metrics endpoint, WildFly prints a stacktrace for each metrics that is has detected (i.e. more than thousands) with the following message:

WFLYCTL0379: System boot is in process; execution of remote management operations is not currently available

Is there any configurations/tricks to avoid having these stacktraces during initialization?

Here is an example for wildfly_jpa_second_level_cache_size_in_memory metric:

2020-03-02 09:38:06,739 WARN                          [management I/O-2] [metrics] Unable to export metric wildfly_jpa_second_level_cache_size_in_memory: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute second-level-cache-size-in-memory on [

    ("deployment" => "application.ear"),

    ("subdeployment" => "application-persistence.jar"),

    ("subsystem" => "jpa"),

    ("hibernate-persistence-unit" => "application.ear/application-persistence.jar#app"),

    ("entity-cache" => "com.app.persistence.entity.Country")

]: "WFLYCTL0379: System boot is in process; execution of remote management operations is not currently available".

        at org.wildfly.extension.microprofile.metrics.MetricCollector.readAttributeValue(MetricCollector.java:303)

        at org.wildfly.extension.microprofile.metrics.MetricCollector.access$200(MetricCollector.java:70)

        at org.wildfly.extension.microprofile.metrics.MetricCollector$2.getValue(MetricCollector.java:174)

        at org.wildfly.extension.microprofile.metrics.MetricCollector$2.getValue(MetricCollector.java:171)

        at io.smallrye.metrics.exporters.OpenMetricsExporter.createSimpleValueLine(OpenMetricsExporter.java:445)

        at io.smallrye.metrics.exporters.OpenMetricsExporter.exposeEntries(OpenMetricsExporter.java:178)

        at io.smallrye.metrics.exporters.OpenMetricsExporter.getEntriesForScope(OpenMetricsExporter.java:150)

        at io.smallrye.metrics.exporters.OpenMetricsExporter.exportAllScopes(OpenMetricsExporter.java:101)

        at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:116)

        at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:73)

        at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:81)

        at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)

        at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)

        at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)

        at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)

        at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)

        at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)

        at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)

        at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)

        at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)

        at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:666)

        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:376)

        at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)

        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)

        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)

        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)

        at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)

        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)

        at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

Regards

Ehsan

[Brian Stansberry]

Hi Ehsan,

Can you file a JIRA for this at https://issues.redhat.com/projects/WFLY ? MP Metrics component. If the WildFly version you experienced this with wasn't 18 or one of the 19 betas, if you could try with 18.0.1.Final or 19.0.0.Beta2 that would be nice, just to confirm it isn't already fixed. But looking at a quick scan of changes in that component I'm not seeing something that looks like a fix for this.

Best regards,

Brian

[Ehsan Zaery Moghaddam]

Thanks Brian

I tested that in 18.0.1.Final and 19.0.0.Beta2 and it happens in both. So as per your suggestion I filed the following issue in JIRA for that:

https://issues.redhat.com/browse/WFLY-13188

Regards

Ehsan