CPU usage 100% with wildfly 26.

[Arjun Lodhe]

Hello everyone,

We are observing high CPU utilization with security scanner (e.g. Qualys).

So we have a JAVA application deployed on wildfly which works fine normally. But as soon as we run the Qualys scanner the utilization goes high (upto 100%) and it stays like this until we reboot the server or restart the wildfly services. 

Wildfly version: WFLYSRV0050: WildFly Full 26.1.3.Final (WildFly Core 18.1.2.Final).

Also, we dont see this with wildfly version: WFLYSRV0049: WildFly Full 19.0.0.Final (WildFly Core 11.0.0.Final)

Regards,

Arjun

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful.  The views and opinions included in this email belong to their author and do not necessarily mirror the views and opinions of the company.  If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. 

[Pratik Gupta]

Hi,

We are also facing the same issue where CPU utilization going very high at the time of security scan. Any solution or workaround for this? 

- Pratik

[Kumar Dawani]

Hello All,

Lot of of our customers are facing the similar issue, please suggest a quick fix or workaround for this issue.

If anyone has any clue to the fix or root cause of the issue please post it would be really helpful.

Regards,

Kumar

[Darran Lofthouse]

I would suggest seeing if you can reproduce this in a non production environment so that you can undertake some investigation as to what is happening.

A couple of things that may be useful could be some GC logging and a few thread dumps taken during the high CPU utilisation to try and see what is happening at the time.

[Arjun Lodhe]

Thank you for the response Darran Lofthouse. 

Unfortunately, when the CPU  utilization is high, we are not able to generate heapdump or threaddump. However I managed to generate process stack using gdb and almost all the processes are stuck here:

Thread 246 (Thread 0x7fa93ceb5700 (LWP 8409)):
#0  0x00007fa965568a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa964432873 in os::PlatformEvent::park() () from /usr/lib/jvm/java-11-openjdk-11.0.18.0.10-1.el7_9.x86_64/lib/server/libjvm.so
#2  0x00007fa9643dc1a2 in Monitor::IWait(Thread*, long) () from /usr/lib/jvm/java-11-openjdk-11.0.18.0.10-1.el7_9.x86_64/lib/server/libjvm.so
#3  0x00007fa9643dcd66 in Monitor::wait(bool, long, bool) () from /usr/lib/jvm/java-11-openjdk-11.0.18.0.10-1.el7_9.x86_64/lib/server/libjvm.so
#4  0x00007fa96414a08e in JVM_WaitForReferencePendingList () from /usr/lib/jvm/java-11-openjdk-11.0.18.0.10-1.el7_9.x86_64/lib/server/libjvm.so
#5  0x00007fa94ab50a30 in ?? ()
#6  0x00007fa93ceb48f0 in ?? ()

Regards,

Arjun

--
You received this message because you are subscribed to the Google Groups "WildFly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/55400351-49f6-4580-920f-742773cd9bd0n%40googlegroups.com.

[Mike Lothian]

Hi

The issue is in Infinispan, I see it here too, if you delete the infinispan directories in data and tmp it should get you back up and running

I've tried backporting some fixes to the infinispan 13.0.21 & 13.0.22 releases and built wildfly 26.1.3 with those releases but the issue still persists

Does anyone else have any ideas on how to debug this issue?

Cheers

Mike

[Mike Lothian]

That should have said wildfly 25.0.1

On Tue, 16 Jul 2024 at 17:19, Mike Lothian <mi...@fireburn.co.uk> wrote:
>
> So backporting the fixes in
> https://github.com/infinispan/infinispan/pull/11806 is a bit beyond
> me, as it trying to backport
> https://github.com/wildfly/wildfly/pull/16076
>
> What I have done is built a local copy of wildfly-core 17.0.3 with the
> following patch:
>
> --- a/pom.xml
> +++ b/pom.xml
> @@ -161,7 +161,7 @@
> <version.com.google.guava.failureaccess>1.0.1</version.com.google.guava.failureaccess>
> <version.com.jcraft.jsch>0.1.55</version.com.jcraft.jsch>
> <version.commons-lang>2.6</version.commons-lang>
> - <version.io.undertow>2.2.12.Final</version.io.undertow>
> + <version.io.undertow>2.2.33.Final</version.io.undertow>
> <version.jakarta.inject.jakarta.inject-api>1.0.3</version.jakarta.inject.jakarta.inject-api>
> <version.jakarta.json.jakarta-json-api>1.1.6</version.jakarta.json.jakarta-json-api>
> <version.javax.xml.bind.jaxb-api>2.4.0-b180830.0359</version.javax.xml.bind.jaxb-api>
> @@ -195,7 +195,7 @@
> <version.org.jboss.invocation>1.7.0.Final</version.org.jboss.invocation>
> <version.org.jboss.jandex>2.4.1.Final</version.org.jboss.jandex>
> <version.org.jboss.jboss-dmr>1.6.1.Final</version.org.jboss.jboss-dmr>
> - <version.org.jboss.jboss-vfs>3.2.15.Final</version.org.jboss.jboss-vfs>
> + <version.org.jboss.jboss-vfs>3.2.17.Final</version.org.jboss.jboss-vfs>
> <version.org.jboss.logging.commons-logging-jboss-logging>1.0.0.Final</version.org.jboss.logging.commons-logging-jboss-logging>
> <version.org.jboss.logging.jboss-logging>3.4.2.Final</version.org.jboss.logging.jboss-logging>
> <version.org.jboss.logging.jboss-logging-tools>2.2.1.Final</version.org.jboss.logging.jboss-logging-tools>
>
> The undertow change is to fix some known vulnerabilites, but will
> probably need to move to 2.2.34 when released
>
> The jboss-vfs change is to work around an issue where zip files aren't
> unpacked properly - very annoying
>
> I then built wildfly 15.0.1 which picks the above up, it uses
> infinispan 12 which I'm hoping isn't affected by the 100% cpu bug
>
> I'll keep you posted

> > To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/9ceb12b8-160a-4412-9dc6-536cd450af72n%40googlegroups.com.

[Mike Lothian]

> To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/9ceb12b8-160a-4412-9dc6-536cd450af72n%40googlegroups.com.

[Kumar Dawani]

It seems like issue is coming when openjdk is upgraded from 11.0.17 +, issue is not coming on 11.0.17 openjdk version.

You received this message because you are subscribed to a topic in the Google Groups "WildFly" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wildfly/Zv62kiDZebc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wildfly+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/CAHbf0-GHf780HKHYKEu37-y6EUg3PU1E7S%3DxCYjSnR6TZ%3D%2Bn%3DA%40mail.gmail.com.

[Arjun Lodhe]

Hey Mike,

Were you able to resolve the high cpu issue by the custom wildfly-core? If yes, what exactly are you changing here? We use standalone-full.xml based configuration.

Also, we are observing this issue only with openjdk11.0.18 and beyond. If we downgrade the openjdk version to 11.0.17, we don't see high cpu issue anymore. Could you please confirm this also in your system.

Regards,

Arjun

To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/CAHbf0-FQd0eartzqLCnT7Tc-dX4mt%2B2W0%2BBah-8gobk08gMdxw%40mail.gmail.com.

[serg.k...@gmail.com]

Hey guys! There is many  possible reasons and really hard to catch root cause without any debug information. I don't think heap dump can help here but thread dump or fly radar does.
First, you can try to send "kill -3 <PID>" signal to make thread dump to console and post it here for future analize.

Some time ago, i had a height CPU utilization issue which was related to the ActiveMQ deadlock. Just in case it is your case - here is details and workaround: https://kostenko.org/blog/2019/06/wildfly-activemq-deadlock.html

пʼятниця, 19 липня 2024 р. о 08:30:56 UTC+1 Arjun Lodhe пише:

[Scott Marlow]

Good tip to capture a thread dump!  

For me, I like to capture a few thread dumps and reading them is important also. :)

If a cpu is at 100 util, you are looking for the thread that is executing Java code that isn't blocking on things that typically block (like file or database io). 

While thread dumps do not contain program state (e.g. variables), they do contain the current call stack of every thread.

I recommend that you spend time reading the thread dumps before sharing them and then you can summarize what you saw when sending them. Or just send the summary.

I'm the early Java days I used thread dumps a lot lot to improve Java performance of a Java server and it was very helpful!

Scott

To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/c3f2566d-0460-4389-9ab1-83668b52eae3n%40googlegroups.com.

[Arjun Lodhe]

We are able to solve the issue by downgrading the openjdk version to 11.0.17. Can anyone else also confirm this ?

Unfortunately, when the CPU  utilization is high, we are not able to generate heapdump or threaddump. However I managed to generate process stack using gdb and almost all the processes are stuck here:

Thread 246 (Thread 0x7fa93ceb5700 (LWP 8409)):
#0  0x00007fa965568a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa964432873 in os::PlatformEvent::park() () from /usr/lib/jvm/java-11-openjdk-11.0.18.0.10-1.el7_9.x86_64/lib/server/libjvm.so
#2  0x00007fa9643dc1a2 in Monitor::IWait(Thread*, long) () from /usr/lib/jvm/java-11-openjdk-11.0.18.0.10-1.el7_9.x86_64/lib/server/libjvm.so
#3  0x00007fa9643dcd66 in Monitor::wait(bool, long, bool) () from /usr/lib/jvm/java-11-openjdk-11.0.18.0.10-1.el7_9.x86_64/lib/server/libjvm.so
#4  0x00007fa96414a08e in JVM_WaitForReferencePendingList () from /usr/lib/jvm/java-11-openjdk-11.0.18.0.10-1.el7_9.x86_64/lib/server/libjvm.so
#5  0x00007fa94ab50a30 in ?? ()
#6  0x00007fa93ceb48f0 in ?? ()

Regards,

Arjun

To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/c3f2566d-0460-4389-9ab1-83668b52eae3n%40googlegroups.com.

[serg.k...@gmail.com]

Well, unfortunately above is not enough as from this stack trace we can see that  thread  is currently waiting and not actively using CPU resources. 
From other hand JVM_WaitForReferencePendingList - is seems to related to the garbage collector (which for sure can affect CPU)

So i still recommend to find a way to do thread dump and enable gc logging by  (-verbose:gc) , or ideally use  flight recorder  or other profiler (JProfiler etc...)

BR, kostenko

понеділок, 22 липня 2024 р. о 13:57:35 UTC+1 Arjun Lodhe пише:

[John Saccoccio]

Any SSLConduit references in the stack trace?  An Undertow bug in JDK 11 change killed us.  Seems IBM is locking down support, can't access.

https://access.redhat.com/solutions/6997164

• After upgrading our JDK to the latest, we are seeing high CPU and unresponsiveness on JBoss EAP 7 when any Qualys security scan is run

https://bugzilla.redhat.com/show_bug.cgi?id=2174246

Downgrade to JDK earlier than jdk 11.0.18 (we're using 11.0.15)