Some Metrics are always zero

[1h0m5s]

Hello Everyone,

We are using WildFly 31.0.1.Final and are trying to setup some monitoring. Therefore, we are using the Micrometer subsystem to export the metrics. Here is my standalone.xml configuration:

<subsystem xmlns="urn:wildfly:micrometer:1.1" exposed-subsystems="*">
  <otlp-registry endpoint="${env.MICROMETER_ENDPOINT:http://localhost:4318/v1/metrics}" step="${env.MICROMETER_STEP:15}"/>

</subsystem>

During my test I observed that the following metrics are always reporting the value 0.

cpu_system_load_average
cpu_available_processors
thread_count
thread_daemon_count
thread_max_count
classloader_loaded_classes_count
classloader_loaded_classes_total
classloader_unloaded_classes_total

I configured already the "statistics-enabled" flag for datasources, transactions, and undertow.

How can I get these values?

Best regards,

Thomas

[Roland Spindelbalker-Davila]

This seems to be caused by a faulty extraction of the related MBean values, I opened a ticket for the bug: https://issues.redhat.com/browse/WFLY-19151

BR,

Roland Spindelbalker-Davila

[Jason Lee]

Thanks for raising the issue and the detailed report. You did half the work for me. :) I have PR opened here: https://github.com/wildfly/wildfly/pull/17759

Jason Lee

Principal Software Engineer

Red Hat JBoss EAP

--
You received this message because you are subscribed to the Google Groups "WildFly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/00b2436c-29ea-4b62-abca-a04e6e7044d0n%40googlegroups.com.

[Rudi Araújo]

Hi folks,

I'm having the same issue but with the undertow metrics - they are reported, but consistently as 0, despite making several HTTP requests. I'm using WildFly 29.0.1.

I made sure that the undertow subsystem has the statistics-enabled attribute turned on:

[standalone@localhost:9990 /] /subsystem=undertow:read-resource
{
    "outcome" => "success",
    "result" => {
        "default-security-domain" => "other",
        "default-server" => "default-server",
        "default-servlet-container" => "default",
        "default-virtual-host" => "default-host",
        "instance-id" => expression "${jboss.node.name}",
        "obfuscate-session-route" => false,
        "statistics-enabled" => true,
        "application-security-domain" => {"other" => undefined},
        "buffer-cache" => {"default" => undefined},
        "byte-buffer-pool" => {"default" => undefined},
        "configuration" => {
            "filter" => undefined,
            "handler" => undefined
        },
        "server" => {"default-server" => undefined},
        "servlet-container" => {"default" => undefined}
    }
}

I made sure that the micrometer subsystem is exposing all subsystems (even tried with ["undertow"] only):

[standalone@localhost:9990 /] /subsystem=micrometer:read-resource

{
    "outcome" => "success",
    "result" => {
        "exposed-subsystems" => ["*"],
        "endpoint" => expression "${OTEL_COLLECTOR_URL}",
        "step" => 1L
    }
}

I made sure the wildfly.statistics-enabled and wildfly.undertow.statistics-enabled system props are set to true.

I tried with and without the metrics subsystem enabled - same result. One thing I did notice is that when the metrics subsystem is enabled, it doesn't report any undertow metrics at all (not even at 0). I suspect these issues might be related.

If it helps, here is the config I'm using, slightly modified to remove the management HTTPS port. And the CLI commands I'm using to enable micrometer:

if (outcome != success) of /extension=org.wildfly.extension.micrometer:read-resource
    /extension=org.wildfly.extension.micrometer:add
    /subsystem=micrometer:add()
end-if

/subsystem=micrometer:write-attribute(name=endpoint, value="${OTEL_COLLECTOR_URL}")
/subsystem=micrometer:write-attribute(name=step, value=1)

/subsystem=undertow:write-attribute(name=statistics-enabled, value=true)

And of course I'm reloading the server after this (and also tried shutdown --restart=true)

I have run out of ideas to pursue - I have tried going through the WildFly code but couldn't find any reason for the behaviour. Some help or guidance would be much appreciated.

Thanks for your time!

Kind regards,

Rudi Araújo.

[Jason Lee]

Which metrics specifically are you interested in? I'll try to recreate the failure locally...

Jason Lee

Principal Software Engineer

Red Hat JBoss EAP

Java Champion

To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/dcbbc36b-2409-4137-add0-df978829993an%40googlegroups.com.

[Rudi Araújo]

Hi Jason,

Thanks for the reply.

All of the Undertow metrics are interesting to me, but here are the ones I would really like to see working well if possible:

• undertow_active_sessions
• undertow_expired_sessions_total
• undertow_rejected_sessions_total
• undertow_sessions_created_total
• undertow_bytes_received_total
• undertow_bytes_sent_total
• undertow_error_count_total
• undertow_processing_time_seconds_total
• undertow_request_count_total
• undertow_request_time_seconds_total

Note that Undertow is not the only subsystem whose metrics aren't working properly: for example, the EE and EJB3 subsystems also seem to be suffering from the same issue.

Thanks again for looking into this, and please let me know if you need any additional info.

Kind regards,

Rudi.

[Jason Lee]

FYI: I think I have a fix for the Undertow issue here: https://github.com/wildfly/wildfly/pull/18103

Jason Lee

Principal Software Engineer

Red Hat JBoss EAP

Java Champion

On Thu, Jul 25, 2024 at 2:52 PM Jason Lee <jaso...@redhat.com> wrote:

Sorry, no, there has not been. There is a JIRA file, though (https://issues.redhat.com/browse/WFLY-19583), and as soon as I can finish up an unrelated -- if somewhat adjacent -- testing issue, I'll get right on this.

Jason Lee

Principal Software Engineer

Red Hat JBoss EAP

Java Champion

On Tue, Jul 23, 2024 at 3:40 AM Rudi Araújo <rudi....@gmail.com> wrote:

Hi Jason,

I just thought I'd check in to see if there's been any progress on the Undertow metrics reported as 0.

Thanks for your time.

Kind regards,

Rudi Araujo.

On Thu, 11 Jul 2024 at 14:40, Rudi Araújo <rudi....@gmail.com> wrote:

Hi Jason,

Thanks for your time on this! On my last email on the other thread I added a link to a small repository containing an example application where this issue can be observed as well. https://github.com/autoluminescent/wildfly-percentiles-example 

Although the percentiles issue seem to be fixed on versions >=31, the Undertow metrics are still reported as 0.

Please let me know if you have any questions.

Kind regards,

Rudi.

On Wed, 10 Jul 2024, 18:03 Jason Lee, <jaso...@redhat.com> wrote:

I haven't forgotten about you on this one, but I just haven't had a chance to dig in yet. Hopefully soon. :)

To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/CAL6Mz9cvmvw78H%3DvmCadNW91mNVY5j5CZ%3DmCV-LKGbFA14Rtpg%40mail.gmail.com.

[Jason Lee]

The fix for this has been merged into main (which will be WF 34) and the 33.x branch (for 33.0.1). If you have time, trying your app with a SNAPSHOT of either version to verify the fix would be super helpful. :)

Thanks!

Jason Lee

Principal Software Engineer

Red Hat JBoss EAP

Java Champion

[Rudi Araújo]

Hi Jason,

Thanks a lot for this! I have been off for a couple of weeks, and now I'll be focusing on something else, but I'll let you know once I try the fix.

Kind regards,

Rudi.

[Rudi Araújo]

Hi Jason,

Since the application we're building on has upgraded to the latest WildFly stable version (33.0.2), I finally got around to testing the metrics again, and unfortunately I still see the same issue.

I've updated this small reproducer repo: https://github.com/autoluminescent/wildfly-percentiles-example

Is there anything I'm missing in this example? Some configuration, perhaps?

Much appreciated for your time.

Thanks,

Rudi.

[Rudi Araújo]

Hi again,

Please ignore my previous email, I found the issue - I forgot to set statistics-enabled to true. All is good now!

Thanks again for your help,

Rudi.