swap in use

214 views
Skip to first unread message

Karol

unread,
Aug 26, 2023, 9:03:49 AM8/26/23
to DSpace Technical Support
hi,

I have a lot of memory allocated for dspace 32,64,128 Gb - it doesn't matter,  swap is in use anyway. I found such errors in tomcat:

The web application [server] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@4e12dfd8]) and a value of type [org.springframework.util.ConcurrentLruCache] (value [org.springframework.util.ConcurrentLruCache@4cf0f8fd]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak

And swap is used by the process:
/usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
 
Greetings,

Karol

Bram Luyten

unread,
Aug 27, 2023, 10:16:50 AM8/27/23
to DSpace Technical Support
Hi Karol,

On which version of DSpace do you see this happening?

Cheers,

Bram

logoBram Luyten
1335 Jefferson Road # 23282, Rochester, NY 14692
Gaston Geenslaan 14, 3001 Leuven, Belgium
DSpace Express Hosting - Open Repository Hosting - Custom DSpace Services


--
All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
---
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/ac1e66bb-21ef-40e9-88b1-421f1e313211n%40googlegroups.com.

Karol

unread,
Aug 28, 2023, 3:21:27 AM8/28/23
to DSpace Technical Support
Hi Bram,

it's happening on DSpace 7.5, angular 7.5. My environment:

-  Debian 11
-  Java version: 11.0.20 (java-11-openjdk)
-  Apache Tomcat/9.0.43
-  pm2 version 5.3.0 started with cluster mode

Thanks,

Karol

DSpace Technical Support

unread,
Aug 31, 2023, 1:09:49 PM8/31/23
to DSpace Technical Support
Hi Karol,

Based on the details you've shared, it's really difficult to tell what may be happening.  It sounds like Tomcat is using your swap, so that'd imply that it is overwhelmed.  This could be that the DSpace backend is overwhelmed at that time, or maybe something in Tomcat isn't "tuned" or configured properly?

Have you followed the performance tuning guide (especially the parts for Tomcat)? 

Are there any other clues in your logs or the Tomcat logs which might provide more information (the logs you shared aren't very informative...they just show that swap is likely occurring, but not clear why)?   Any errors you see in those logs?  Here's our troubleshooting guide for help on locating errors: 

Basically, it's unclear to me what could be going on.  So, it'd be very helpful if you could search for more clues & double check that you've "tuned" Tomcat via CATALINA_OPTS or similar.

Tim

Karol

unread,
Sep 1, 2023, 5:46:18 AM9/1/23
to DSpace Technical Support
Hi Tim,

yes, I used https://wiki.lyrasis.org/display/DSDOC7x/Performance+Tuning+DSpace for increase performance. I found something like this:

[2023-09-01 10:03:55] [warning] Failed to clear soft references from ObjectStreamClass$Caches for web application [ROOT]
[2023-09-01 10:03:55] [warning] java.lang.ClassCastException: class java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearCache(WebappClassLoaderBase.java:2282)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches(WebappClassLoaderBase.java:2257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearReferences(WebappClassLoaderBase.java:1626)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.stop(WebappClassLoaderBase.java:1554)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:461)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5469)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
[2023-09-01 10:03:55] [warning]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-09-01 10:03:55] [warning]     at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
[2023-09-01 10:03:55] [warning]     at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
[2023-09-01 10:03:55] [warning]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-09-01 10:03:55] [warning]     at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
[2023-09-01 10:03:55] [warning]     at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:498)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:982)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.startup.Catalina.stop(Catalina.java:849)
[2023-09-01 10:03:55] [warning]     at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:996)
[2023-09-01 10:03:56] [warning] java.lang.ClassCastException: class java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearCache(WebappClassLoaderBase.java:2282)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesObjectStreamClassCaches(WebappClassLoaderBase.java:2257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.clearReferences(WebappClassLoaderBase.java:1626)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappClassLoaderBase.stop(WebappClassLoaderBase.java:1554)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:461)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5469)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
[2023-09-01 10:03:56] [warning]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-09-01 10:03:56] [warning]     at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
[2023-09-01 10:03:56] [warning]     at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1400)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1389)
[2023-09-01 10:03:56] [warning]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-09-01 10:03:56] [warning]     at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
[2023-09-01 10:03:56] [warning]     at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:976)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:498)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:982)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.startup.Catalina.stop(Catalina.java:849)
[2023-09-01 10:03:56] [warning]     at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:996)
[2023-09-01 10:03:56] [warning] The web application [server] appears to have started a thread named [Thread-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
[2023-09-01 10:03:56] [info]  java...@11.0.20/java.lang.Object.wait(Native Method)
[2023-09-01 10:03:56] [info]  org.dspace.service.impl.HttpConnectionPoolService$IdleConnectionMonitorThread.run(HttpConnectionPoolService.java:176)
[2023-09-01 10:03:56] [warning] Failed to clear soft references from ObjectStreamClass$Caches for web application [server]
[2023-09-01 10:03:56] [warning] java.lang.ClassCastException: class java.io.ObjectStreamClass$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectStreamClass$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')

Best,

Karol

DSpace Technical Support

unread,
Sep 1, 2023, 10:56:31 AM9/1/23
to DSpace Technical Support
Hi Karol,

The errors you are sharing still aren't making sense to me.  They all seem to reference Tomcat having issues... and none of them reference DSpace.  Are you certain you are using a supported version of Tomcat? https://wiki.lyrasis.org/display/DSDOC7x/Installing+DSpace#InstallingDSpace-ServletEngine(ApacheTomcat9,Jetty,CauchoResinorequivalent)

Are you running anything else in Tomcat besides DSpace? 

Have you tried restarting Tomcat to see if the behavior continues (or see if errors are shown in the Tomcat logs during the restart)?

What are your CATALINA_OPTS settings that you configured via https://wiki.lyrasis.org/display/DSDOC7x/Performance+Tuning+DSpace ?

Basically, based on what you've shared, it all seems to imply there's something wrong with your Tomcat installation.  I'm not sure what could be wrong, but you may want to look more closely there.  You also could try installing Tomcat over again to see if something went wrong with the Tomcat install (even small things like permissions issues can sometimes cause problems in Tomcat). 

I wish I had better ideas, but I've never seen this issue before.

Tim
Reply all
Reply to author
Forward
0 new messages