Wildfly : Metaspace is not getting cleared even after GC

382 views
Skip to first unread message

Rahul Kumar

unread,
Mar 27, 2023, 10:28:24 AM3/27/23
to WildFly

We're using

  1. OpenJDK 11
  2. Wildfly-17.0.1.Final
  3. Spring + Hibernate Application (Deployed as WAR in Wildfly)

With each deployment, the metaspace keeps on increasing and not getting garbage collected 

NOTE: We've set -XX:MaxMetaspaceSize parameter so that GC is triggered once the metaspace reaches the threshold

We are able to receive java.lang.OutOfMemoryError: Metaspace error upon reaching the MaxMetaspaceSize, but after that Full GC runs and doesn't clear any metaspace, below is the snippet of GC Log when it triggers Full GC due to threshold limit reached

------------------------------------------------------------------

[2023-03-21T05:54:15.335+0000] GC(1321) Pause Full (Metadata GC Threshold)
[2023-03-21T05:54:15.422+0000] GC(1317) Concurrent Sweep 159.546ms
[2023-03-21T05:54:15.422+0000] GC(1317) User=0.32s Sys=0.00s Real=0.16s
[2023-03-21T05:54:15.422+0000] GC(1321) Phase 1: Mark live objects
[2023-03-21T05:54:15.981+0000] GC(1321) Phase 1: Mark live objects 558.134ms
[2023-03-21T05:54:15.981+0000] GC(1321) Phase 2: Compute new object addresses
[2023-03-21T05:54:16.197+0000] GC(1321) Phase 2: Compute new object addresses 216.252ms
[2023-03-21T05:54:16.197+0000] GC(1321) Phase 3: Adjust pointers
[2023-03-21T05:54:16.832+0000] GC(1321) Phase 3: Adjust pointers 635.469ms
[2023-03-21T05:54:16.832+0000] GC(1321) Phase 4: Move objects
[2023-03-21T05:54:17.060+0000] GC(1321) Phase 4: Move objects 227.785ms
[2023-03-21T05:54:17.062+0000] GC(1321) Pause Full (Metadata GC Threshold) 443M->437M(907M) 1727.534ms
[2023-03-21T05:54:17.062+0000] GC(1320) ParNew: 5797K->0K(58944K)
[2023-03-21T05:54:17.062+0000] GC(1320) CMS: 452714K->448254K(869848K)
[2023-03-21T05:54:17.062+0000] GC(1320) Metaspace: 509715K->509715K(974848K)
[2023-03-21T05:54:17.063+0000] GC(1320) Pause Young (Metadata GC Threshold) 447M->437M(1232M) 1733.687ms
[2023-03-21T05:54:17.063+0000] GC(1320) User=1.67s Sys=0.05s Real=1.74s
[2023-03-21T05:54:17.063+0000] GC(1317) Old: 443369K->448254K(869848K)
[2023-03-21T05:54:17.063+0000] GC(1322) Pause Young (Metadata GC Threshold)
[2023-03-21T05:54:17.063+0000] GC(1322) Using 8 workers of 8 for evacuation
[2023-03-21T05:54:17.065+0000] GC(1323) Pause Full (Metadata GC Threshold)
[2023-03-21T05:54:17.066+0000] GC(1323) Phase 1: Mark live objects
[2023-03-21T05:54:17.631+0000] GC(1323) Phase 1: Mark live objects 565.499ms
[2023-03-21T05:54:17.631+0000] GC(1323) Phase 2: Compute new object addresses
[2023-03-21T05:54:17.899+0000] GC(1323) Phase 2: Compute new object addresses 267.410ms
[2023-03-21T05:54:17.899+0000] GC(1323) Phase 3: Adjust pointers
[2023-03-21T05:54:18.549+0000] GC(1323) Phase 3: Adjust pointers 649.816ms
[2023-03-21T05:54:18.549+0000] GC(1323) Phase 4: Move objects
[2023-03-21T05:54:18.631+0000] GC(1323) Phase 4: Move objects 82.086ms
[2023-03-21T05:54:18.633+0000] GC(1323) Pause Full (Metadata GC Threshold) 438M->437M(1232M) 1568.185ms
[2023-03-21T05:54:18.633+0000] GC(1322) ParNew: 10K->0K(391744K)
[2023-03-21T05:54:18.633+0000] GC(1322) CMS: 448254K->448254K(869848K)
[2023-03-21T05:54:18.633+0000] GC(1322) Metaspace: 509715K->509715K(974848K)
[2023-03-21T05:54:18.633+0000] GC(1322) Pause Young (Metadata GC Threshold) 437M->437M(1232M) 1570.109ms
[2023-03-21T05:54:18.633+0000] GC(1322) User=1.55s Sys=0.01s Real=1.57s
[2023-03-21T05:54:18.633+0000] GC(1324) Pause Young (Metadata GC Clear Soft References)
[2023-03-21T05:54:18.633+0000] GC(1324) Using 8 workers of 8 for evacuation
[2023-03-21T05:54:18.635+0000] GC(1325) Pause Full (Metadata GC Clear Soft References)
[2023-03-21T05:54:18.636+0000] GC(1325) Phase 1: Mark live objects
[2023-03-21T05:54:19.244+0000] GC(1325) Phase 1: Mark live objects 607.533ms
[2023-03-21T05:54:19.244+0000] GC(1325) Phase 2: Compute new object addresses
[2023-03-21T05:54:19.425+0000] GC(1325) Phase 2: Compute new object addresses 180.976ms
[2023-03-21T05:54:19.425+0000] GC(1325) Phase 3: Adjust pointers
[2023-03-21T05:54:19.931+0000] GC(1325) Phase 3: Adjust pointers 505.607ms
[2023-03-21T05:54:19.931+0000] GC(1325) Phase 4: Move objects
[2023-03-21T05:54:20.061+0000] GC(1325) Phase 4: Move objects 129.839ms
[2023-03-21T05:54:20.062+0000] GC(1325) Pause Full (Metadata GC Clear Soft References) 437M->385M(1232M) 1427.034ms
[2023-03-21T05:54:20.062+0000] GC(1324) ParNew: 0K->0K(391744K)
[2023-03-21T05:54:20.062+0000] GC(1324) CMS: 448254K->394889K(869848K)
[2023-03-21T05:54:20.062+0000] GC(1324) Metaspace: 509715K->509715K(974848K)
[2023-03-21T05:54:20.066+0000] GC(1324) Pause Young (Metadata GC Clear Soft References) 437M->385M(1232M) 1432.729ms
[2023-03-21T05:54:20.066+0000] GC(1324) User=1.43s Sys=0.01s Real=1.43s
[2023-03-21T05:54:20.066+0000] Metaspace (data) allocation failed for size 11
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] Usage:
[2023-03-21T05:54:20.066+0000]   Non-class:    442.51 MB capacity,   435.47 MB ( 98%) used,     6.21 MB (  1%) free+waste,   856.81 KB ( <1%) overhead.
[2023-03-21T05:54:20.066+0000]       Class:     63.13 MB capacity,    59.73 MB ( 95%) used,     3.07 MB (  5%) free+waste,   342.06 KB ( <1%) overhead.
[2023-03-21T05:54:20.066+0000]        Both:    505.65 MB capacity,   495.19 MB ( 98%) used,     9.28 MB (  2%) free+waste,     1.17 MB ( <1%) overhead.
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] Virtual space:
[2023-03-21T05:54:20.066+0000]   Non-class space:      448.00 MB reserved,     447.88 MB (>99%) committed
[2023-03-21T05:54:20.066+0000]       Class space:      504.00 MB reserved,      64.12 MB ( 13%) committed
[2023-03-21T05:54:20.066+0000]              Both:      952.00 MB reserved,     512.00 MB ( 54%) committed
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] Chunk freelists:
[2023-03-21T05:54:20.066+0000]    Non-Class:  5.31 MB
[2023-03-21T05:54:20.066+0000]        Class:  1015.00 KB
[2023-03-21T05:54:20.066+0000]         Both:  6.30 MB
[2023-03-21T05:54:20.066+0000]
[2023-03-21T05:54:20.066+0000] MaxMetaspaceSize: 512.00 MB
[2023-03-21T05:54:20.066+0000] CompressedClassSpaceSize: 504.00 MB
[2023-03-21T05:54:20.066+0000]

------------------------------------------------------------------

Below are the configurations in standalone.conf -Xms64m -Xmx2048m -XX:NativeMemoryTracking=detail -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m

We've taken Heap Dumps and further analysed using MAT and it seems org.jboss.modules.ModuleClassLoader is not getting cleared after subsequent deployments.

Same behaviour is observed even with undeployments.

mat_overview.png

mat_dominator_tree.png

We've tried below combinations, but the behaviour is same, and metaspace is not getting cleared

  1. openjdk11 + wildfly-17.0.1.Final

  2. openjdk11 + wildfly-18.0.1.Final

  3. openjdk16 + wildfly-17.0.1.Final

We are trying to find the root cause of this, does it have something related to internal implementation of JBoss Modules that it keeps the classes in memory even after undeployments or redeployments or are we missing something else here ?

James Perkins

unread,
Mar 27, 2023, 8:16:56 PM3/27/23
to WildFly
Something has to be holding onto resources. Have you tracked the path to GC to see what it might be?

Richard Opalka

unread,
Mar 28, 2023, 8:39:22 AM3/28/23
to James Perkins, WildFly
Hello Rahul,

   I recommend you to upgrade to the latest WildFly 26.1.3 version. There are many issues fixed and one of them might be yours.

Rio

--
You received this message because you are subscribed to the Google Groups "WildFly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/f4123501-1e8e-4ca7-8e6e-c67af44e75ecn%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages