High CPU utilisation in App server after deploying remote infinispan

[Apurva Save]

We are using wildfly version 20.0.1 with remote infinispan (version 10.1.8) for caching

We observed our App server machine CPU utilisation reaches 50% and stays there till we restart the app server. This behaviour is started after we have introduced remote infinispan with app server. 

Why this CPU showing this behaviour and how to optimise this?

By analysing thread dump we found one thread is consuming CPU.

Below is stack trace for that thread 

java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked <0x00000007a7787978> (a java.lang.IllegalStateException)
at java.lang.Throwable.<init>(Throwable.java:287)
at java.lang.Exception.<init>(Exception.java:84)
at java.lang.RuntimeException.<init>(RuntimeException.java:80)
at java.lang.IllegalStateException.<init>(IllegalStateException.java:75)
at java.lang.Throwable.initCause(Throwable.java:457)
- locked <0x00000006c7c21d98> (a java.lang.ClassCastException)
at org.jboss.marshalling.TraceInformation.getOrAddTraceInformation(TraceInformation.java:66)
at org.jboss.marshalling.TraceInformation.addIncompleteObjectInformation(TraceInformation.java:160)
at org.jboss.marshalling.TraceInformation.addIncompleteObjectInformation(TraceInformation.java:150)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1554)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:283)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:231)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1864)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1778)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1726)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1726)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1406)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:283)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:231)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1864)
at org.jboss.marshalling.river.RiverObjectInputStream.defaultReadObject(RiverObjectInputStream.java:81)
at org.apache.xerces.dom.ParentNode.readObject(ParentNode.java:1009)
at sun.reflect.GeneratedMethodAccessor1326.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.callReadObject(JDKSpecific.java:186)
at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:214)

[Paul Ferraro]

I suspect that you might be running into the issue described here:

https://markmail.org/message/x5e5nreyj7eruyly

which was fixed by:

http://svn.apache.org/viewvc?view=revision&revision=1829683

If so, upgrading xerces to 2.12.2 should solve your problem.

[Apurva Save]

We have upgraded xercesImpl from 2.9.1 to 2.12.2 and dependent jar xml-apis 1.3.04 to 1.4.01.

But still same issue persist.