WildFly not shutting down

[M K]

Hi,

we are running a standalone WildFly 39.0.1 instance with the provided Windows service tooling, and every now and then WildFly is not shutting down correctly. It's not a new issue, it has occurred in previous WildFly versions as well.

I have attached the the log from one of these hanging WildFly instances. As you can see, the server begins shutting down but never completes the process (the last log entry is eight minutes after the start of shutdown).

I suspect the cause cannot be determined by simply analyzing this log. If that is indeed the case, what additional information or diagnostics would help identify the root cause? For example, would a thread dump be useful?

Thank you very much in advance!

Best regards

Manuel K

[Aaron Ogburn]

You're correct the hang can't be diagnosed through the log silence as hung threads by nature aren't really producing log messages for any info as well.  There is a timeout feature for start/stops and other management tasks that defaults to 300 seconds and should induce an automatic thread dump to server logs for diagnostics, but that wasn't reached.  Have you increased the jboss.as.management.blocking.timeout property perchance anywhere?  But otherwise, the best diagnostic for such a hang (at any point whether shutdown, start up, or runtime under load) would be thread dumps to see all current thread stacks and where any shutdown operations are currently pending.  That can be produced with your JDK's jstack command:

jstack -l JAVA_PID > thread-dump.out

If you run JBoss as a windows service, then that can require elevated privileges so you'd typically then need to run that through psexec:

C:\PSTools>psexec -s c:\Java\jdkX.Y.Z_W\bin\jstack.exe -l 4180 >dump.txt

And if you want to go more verbose, it can sometimes be helpful to have a full heap dump through jmap or jcmd as well.  That takes some more effort and knowledge to really dig through beyond just reading a text file like in the thread dump, but a full heap dump can include the thread stacks but then also capture all in heap objects to fully inspect their states that might further answer any questions about why some operations are incomplete:

C:\..\PSTools>psexec -s jmap -dump:format=b,file=%USERPROFILE%\Desktop\jmap_heap.hprof JBOSS_EAP_PID

[M K]

Thank you for your helpful reply!

We have indeed increased the jboss.as.management.blocking.timeout value to many hours. This was necessary to accommodate sometimes long deployment times of our application (for example, Hibernate Search indexing, etc.).

I'll report back once I have more data. Thanks!

Best regards,

Manuel K