swap in use

[Karol]

hi,

I have a lot of memory allocated for dspace 32,64,128 Gb - it doesn't matter,  swap is in use anyway. I found such errors in tomcat:

The web application [server] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@4e12dfd8]) and a value of type [org.springframework.util.ConcurrentLruCache] (value [org.springframework.util.ConcurrentLruCache@4cf0f8fd]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak

And swap is used by the process:
/usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
 
Greetings,

Karol

[Bram Luyten]

Hi Karol,

On which version of DSpace do you see this happening?

Cheers,

Bram

Bram Luyten 1335 Jefferson Road # 23282, Rochester, NY 14692 Gaston Geenslaan 14, 3001 Leuven, Belgium DSpace Express Hosting - Open Repository Hosting - Custom DSpace Services

--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/ac1e66bb-21ef-40e9-88b1-421f1e313211n%40googlegroups.com.

[Karol]

Hi Bram,

it's happening on DSpace 7.5, angular 7.5. My environment:

-  Debian 11

-  Java version: 11.0.20 (java-11-openjdk)

-  Apache Tomcat/9.0.43

-  pm2 version 5.3.0 started with cluster mode

Thanks,

Karol

[DSpace Technical Support]

Hi Karol,

Based on the details you've shared, it's really difficult to tell what may be happening.  It sounds like Tomcat is using your swap, so that'd imply that it is overwhelmed.  This could be that the DSpace backend is overwhelmed at that time, or maybe something in Tomcat isn't "tuned" or configured properly?

Have you followed the performance tuning guide (especially the parts for Tomcat)? 

https://wiki.lyrasis.org/display/DSDOC7x/Performance+Tuning+DSpace

Are there any other clues in your logs or the Tomcat logs which might provide more information (the logs you shared aren't very informative...they just show that swap is likely occurring, but not clear why)?   Any errors you see in those logs?  Here's our troubleshooting guide for help on locating errors: 

https://wiki.lyrasis.org/display/DSPACE/Troubleshoot+an+error#Troubleshootanerror-DSpace7.x(orabove)

Basically, it's unclear to me what could be going on.  So, it'd be very helpful if you could search for more clues & double check that you've "tuned" Tomcat via CATALINA_OPTS or similar.

Tim

[Karol]

Hi Tim,

yes, I used https://wiki.lyrasis.org/display/DSDOC7x/Performance+Tuning+DSpace for increase performance. I found something like this:

[2023-09-01 10:03:55] [warning] Failed to clear soft references from ObjectStreamClass$Caches for web application [ROOT]
[2023-09-01 10:03:55] [warning] java.lang.ClassCastException: class java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearCache(WebappClassLoaderBase.java:2282)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches(WebappClassLoaderBase.java:2257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearReferences(WebappClassLoaderBase.java:1626)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.stop(WebappClassLoaderBase.java:1554)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:461)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5469)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
[2023-09-01 10:03:55] [warning]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-09-01 10:03:55] [warning]     at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
[2023-09-01 10:03:55] [warning]     at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
[2023-09-01 10:03:55] [warning]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-09-01 10:03:55] [warning]     at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
[2023-09-01 10:03:55] [warning]     at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:498)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:982)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.startup.Catalina.stop(Catalina.java:849)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:996)
[2023-09-01 10:03:56] [warning] java.lang.ClassCastException: class java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearCache(WebappClassLoaderBase.java:2282)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches(WebappClassLoaderBase.java:2257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearReferences(WebappClassLoaderBase.java:1626)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.stop(WebappClassLoaderBase.java:1554)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:461)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5469)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
[2023-09-01 10:03:56] [warning]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-09-01 10:03:56] [warning]     at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
[2023-09-01 10:03:56] [warning]     at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
[2023-09-01 10:03:56] [warning]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-09-01 10:03:56] [warning]     at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
[2023-09-01 10:03:56] [warning]     at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:498)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:982)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.startup.Catalina.stop(Catalina.java:849)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:996)
[2023-09-01 10:03:56] [warning] The web application [server] appears to have started a thread named [Thread-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
[2023-09-01 10:03:56] [info]  java...@11.0.20/java.lang.Object.wait(Native Method)
[2023-09-01 10:03:56] [info]  org.dspace.service.impl.HttpConnectionPoolService$IdleConnectionMonitorThread.run(HttpConnectionPoolService.java:176)
[2023-09-01 10:03:56] [warning] Failed to clear soft references from ObjectStreamClass$Caches for web application [server]
[2023-09-01 10:03:56] [warning] java.lang.ClassCastException: class java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')

Best,

Karol

[DSpace Technical Support]

Hi Karol,

The errors you are sharing still aren't making sense to me.  They all seem to reference Tomcat having issues... and none of them reference DSpace.  Are you certain you are using a supported version of Tomcat? https://wiki.lyrasis.org/display/DSDOC7x/Installing+DSpace#InstallingDSpace-ServletEngine(ApacheTomcat9,Jetty,CauchoResinorequivalent)

Are you running anything else in Tomcat besides DSpace? 

Have you tried restarting Tomcat to see if the behavior continues (or see if errors are shown in the Tomcat logs during the restart)?

What are your CATALINA_OPTS settings that you configured via https://wiki.lyrasis.org/display/DSDOC7x/Performance+Tuning+DSpace ?

Basically, based on what you've shared, it all seems to imply there's something wrong with your Tomcat installation.  I'm not sure what could be wrong, but you may want to look more closely there.  You also could try installing Tomcat over again to see if something went wrong with the Tomcat install (even small things like permissions issues can sometimes cause problems in Tomcat). 

I wish I had better ideas, but I've never seen this issue before.

Tim