High CPU and Crash

Ludwig Possie

unread,

Jun 5, 2025, 4:26:00 PM6/5/25

to Archivesspace_Users_Group

I'm out of ideas on how to troubleshoot this issue with AS. When the docker image is started it starts fine, but after a few days the App service will start running at 350%-400% when viewed with ctop. It becomes unresponsive and I have to stop all services and restart them again. I do see an error in the log files, but I'm not familiar with ruby to understand the error or even begin to troubleshoot. Here are my log files. Any suggestions would be appreciated. Thx.

[2025-06-05T20:09:50.821547 #84] ERROR -- : Thread-3841868: Unhandled exception!
E, [2025-06-05T20:09:50.821734 #84] ERROR -- :
timeout: 5.0, elapsed: 5.0917977259960026
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/connection_pool/threaded.rb:280:in `raise_pool_timeout'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/connection_pool/threaded.rb:153:in `acquire'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/connection_pool/threaded.rb:91:in `hold'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/database/connecting.rb:283:in `synchronize'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/adapters/jdbc.rb:260:in `execute'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/dataset/actions.rb:1189:in `execute'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/adapters/jdbc.rb:757:in `fetch_rows'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/dataset/actions.rb:164:in `each'
org/jruby/RubyEnumerable.java:818:in `map'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/dataset/actions.rb:466:in `map'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/model/user.rb:137:in `derived_permissions'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/model/user.rb:154:in `can?'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/rest.rb:239:in `block in permissions'
org/jruby/RubyBasicObject.java:2532:in `instance_eval'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/rest.rb:359:in `block in returns'
org/jruby/RubyArray.java:4910:in `all?'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/rest.rb:359:in `block in returns'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/request_context.rb:24:in `open'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/rest.rb:348:in `block in returns'
org/jruby/RubyMethod.java:118:in `call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1775:in `block in compile!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1065:in `block in route!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1083:in `route_eval'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1065:in `block in route!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1114:in `block in process_route'
org/jruby/RubyKernel.java:1423:in `catch'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1112:in `process_route'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1063:in `block in route!'
org/jruby/RubyArray.java:1981:in `each'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1060:in `route!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1184:in `block in dispatch!'
org/jruby/RubyKernel.java:1423:in `catch'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1155:in `invoke'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1179:in `dispatch!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:995:in `block in call!'
org/jruby/RubyKernel.java:1423:in `catch'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1155:in `invoke'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:995:in `call!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:984:in `call'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/main.rb:329:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/xss_header.rb:20:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/path_traversal.rb:18:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/json_csrf.rb:28:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/base.rb:53:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/base.rb:53:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/frame_options.rb:33:in `call'
uri:classloader:/vendor/rack-3.0.11/rack/null_logger.rb:13:in `call'
uri:classloader:/vendor/rack-3.0.11/rack/head.rb:15:in `call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:219:in `call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:2086:in `call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1645:in `block in call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1861:in `synchronize'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1645:in `call'
uri:classloader:/vendor/rack-3.0.11/rack/urlmap.rb:76:in `block in call'
org/jruby/RubyArray.java:1981:in `each'
uri:classloader:/vendor/rack-3.0.11/rack/urlmap.rb:60:in `call'
org/jruby/rack/ext/Servlet.java:83:in `call'
E, [2025-06-05T20:09:52.965593 #84] ERROR -- : Thread-3841788: Unhandled exception!
E, [2025-06-05T20:09:52.965812 #84] ERROR -- :
timeout: 5.0, elapsed: 7.2243107908871025

--

Ludwig Possié

Systems Admin

Stewart Library

Weber State University

801-626-8093

Blake Carver

unread,

Jun 5, 2025, 4:59:08 PM6/5/25

to Archivesspace_Users_Group

It looks like ArchivesSpace can't talk to MySQL and is timing out.

From: 'Ludwig Possie' via Archivesspace_Users_Group <Archivesspac...@lyrasislists.org>
Sent: Thursday, June 5, 2025 4:25 PM
To: Archivesspace_Users_Group <archivesspac...@lyrasislists.org>
Subject: [ArchivesSpace Users Group] High CPU and Crash

--
You received this message because you are subscribed to the Google Groups "Archivesspace_Users_Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Archivesspace_User...@lyrasislists.org.
To view this discussion visit https://groups.google.com/a/lyrasislists.org/d/msgid/Archivesspace_Users_Group/CA%2BOpDmaLQ1YQGf%2BePjdvfFXVeGwacRyuE7yE0Sx_%2BTJcH5LjHQ%40mail.gmail.com.

Jimmy Chen

unread,

Jun 5, 2025, 10:21:43 PM6/5/25

to Blake Carver, Archivesspace_Users_Group

You can increase connection pool size in the configure file ( config/config.rb ) like

AppConfig[:db_max_connections] = 50

To view this discussion visit https://groups.google.com/a/lyrasislists.org/d/msgid/Archivesspace_Users_Group/DM6PR22MB2309869D46CD73D435E91C3C9F6FA%40DM6PR22MB2309.namprd22.prod.outlook.com.

--

Zhimin (Jimmy) Chen

Software Engineer

NCAR Library & Archives

zhiminchen@ucar.edu

240-205-3492

Andy Bussey

unread,

Jun 6, 2025, 4:24:15 AM6/6/25

to Ludwig Possie, Archivesspace_Users_Group

Dear Ludwig

If the issue has only been happening recently, check if you are being crawled. That happened to us very recently and put a huge load on our PUI server. The recourse in that case is to block certain incoming traffic - talk to your IT security folks if you have them. If it is not that and just regular traffic, then increasing the db connections is likely to help

Cheers

Andy

--

You received this message because you are subscribed to the Google Groups "Archivesspace_Users_Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Archivesspace_User...@lyrasislists.org.
To view this discussion visit https://groups.google.com/a/lyrasislists.org/d/msgid/Archivesspace_Users_Group/CA%2BOpDmaLQ1YQGf%2BePjdvfFXVeGwacRyuE7yE0Sx_%2BTJcH5LjHQ%40mail.gmail.com.

--

Andy Bussey

Head of Library Digital Services

University of Sheffield Library

a.bu...@sheffield.ac.uk

+44 (0)7787 152149

Ludwig Possie

unread,

Jun 6, 2025, 12:18:11 PM6/6/25

to Andy Bussey, Archivesspace_Users_Group

Great advice everyone! I'll increase the db connections and work with our campus IT team to monitor the traffic going to the application. Thank you!!

L

Joshua D. Shaw

unread,

Jun 9, 2025, 6:52:46 AM6/9/25

to Andy Bussey, Ludwig Possie, Archivesspace_Users_Group

Also. If you are running version 4.0 and you are running plugins, you may be experiencing a bug fixed in 4.1.0 related to plugin template files (https://github.com/archivesspace/archivesspace/issues/3508)

Joshua

From: 'Ludwig Possie' via Archivesspace_Users_Group <Archivesspac...@lyrasislists.org>
Sent: Friday, June 6, 2025 12:17 PM
To: Andy Bussey <a.bu...@sheffield.ac.uk>
Cc: Archivesspace_Users_Group <archivesspac...@lyrasislists.org>
Subject: Re: [ArchivesSpace Users Group] High CPU and Crash

To view this discussion visit https://groups.google.com/a/lyrasislists.org/d/msgid/Archivesspace_Users_Group/CA%2BOpDmbJT4_Osfu4ctwuT0m23NDNPCpv0MeokyN9bQ7vsPfkWQ%40mail.gmail.com.

Reply all

Reply to author

Forward