High CPU and Crash
242 views
Skip to first unread message
Ludwig Possie
Jun 5, 2025, 4:26:00 PM6/5/25
to Archivesspace_Users_Group
I'm out of ideas on how to troubleshoot this issue with AS. When the docker image is started it starts fine, but after a few days the App service will start running at 350%-400% when viewed with ctop. It becomes unresponsive and I have to stop all services and restart them again. I do see an error in the log files, but I'm not familiar with ruby to understand the error or even begin to troubleshoot. Here are my log files. Any suggestions would be appreciated. Thx.
[2025-06-05T20:09:50.821547 #84] ERROR -- : Thread-3841868: Unhandled exception!
E, [2025-06-05T20:09:50.821734 #84] ERROR -- :
timeout: 5.0, elapsed: 5.0917977259960026
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/connection_pool/threaded.rb:280:in `raise_pool_timeout'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/connection_pool/threaded.rb:153:in `acquire'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/connection_pool/threaded.rb:91:in `hold'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/database/connecting.rb:283:in `synchronize'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/adapters/jdbc.rb:260:in `execute'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/dataset/actions.rb:1189:in `execute'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/adapters/jdbc.rb:757:in `fetch_rows'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/dataset/actions.rb:164:in `each'
org/jruby/RubyEnumerable.java:818:in `map'
/archivesspace/gems/gems/sequel-5.84.0/lib/sequel/dataset/actions.rb:466:in `map'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/model/user.rb:137:in `derived_permissions'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/model/user.rb:154:in `can?'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/rest.rb:239:in `block in permissions'
org/jruby/RubyBasicObject.java:2532:in `instance_eval'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/rest.rb:359:in `block in returns'
org/jruby/RubyArray.java:4910:in `all?'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/rest.rb:359:in `block in returns'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/request_context.rb:24:in `open'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/lib/rest.rb:348:in `block in returns'
org/jruby/RubyMethod.java:118:in `call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1775:in `block in compile!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1065:in `block in route!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1083:in `route_eval'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1065:in `block in route!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1114:in `block in process_route'
org/jruby/RubyKernel.java:1423:in `catch'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1112:in `process_route'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1063:in `block in route!'
org/jruby/RubyArray.java:1981:in `each'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1060:in `route!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1184:in `block in dispatch!'
org/jruby/RubyKernel.java:1423:in `catch'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1155:in `invoke'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1179:in `dispatch!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:995:in `block in call!'
org/jruby/RubyKernel.java:1423:in `catch'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1155:in `invoke'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:995:in `call!'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:984:in `call'
/archivesspace/data/tmp/jetty-0_0_0_0-8089-backend_war-_-any-5670738201293833883/webapp/WEB-INF/app/main.rb:329:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/xss_header.rb:20:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/path_traversal.rb:18:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/json_csrf.rb:28:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/base.rb:53:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/base.rb:53:in `call'
/archivesspace/gems/gems/rack-protection-3.2.0/lib/rack/protection/frame_options.rb:33:in `call'
uri:classloader:/vendor/rack-3.0.11/rack/null_logger.rb:13:in `call'
uri:classloader:/vendor/rack-3.0.11/rack/head.rb:15:in `call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:219:in `call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:2086:in `call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1645:in `block in call'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1861:in `synchronize'
/archivesspace/gems/gems/sinatra-3.2.0/lib/sinatra/base.rb:1645:in `call'
uri:classloader:/vendor/rack-3.0.11/rack/urlmap.rb:76:in `block in call'
org/jruby/RubyArray.java:1981:in `each'
uri:classloader:/vendor/rack-3.0.11/rack/urlmap.rb:60:in `call'
org/jruby/rack/ext/Servlet.java:83:in `call'
E, [2025-06-05T20:09:52.965593 #84] ERROR -- : Thread-3841788: Unhandled exception!
E, [2025-06-05T20:09:52.965812 #84] ERROR -- :
timeout: 5.0, elapsed: 7.2243107908871025
Blake Carver
Jun 5, 2025, 4:59:08 PM6/5/25
to Archivesspace_Users_Group
It looks like ArchivesSpace can't talk to MySQL and is timing out.
From: 'Ludwig Possie' via Archivesspace_Users_Group <Archivesspac...@lyrasislists.org>
Sent: Thursday, June 5, 2025 4:25 PM
To: Archivesspace_Users_Group <archivesspac...@lyrasislists.org>
Subject: [ArchivesSpace Users Group] High CPU and Crash
Sent: Thursday, June 5, 2025 4:25 PM
To: Archivesspace_Users_Group <archivesspac...@lyrasislists.org>
Subject: [ArchivesSpace Users Group] High CPU and Crash
--
You received this message because you are subscribed to the Google Groups "Archivesspace_Users_Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Archivesspace_User...@lyrasislists.org.
To view this discussion visit https://groups.google.com/a/lyrasislists.org/d/msgid/Archivesspace_Users_Group/CA%2BOpDmaLQ1YQGf%2BePjdvfFXVeGwacRyuE7yE0Sx_%2BTJcH5LjHQ%40mail.gmail.com.
You received this message because you are subscribed to the Google Groups "Archivesspace_Users_Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Archivesspace_User...@lyrasislists.org.
To view this discussion visit https://groups.google.com/a/lyrasislists.org/d/msgid/Archivesspace_Users_Group/CA%2BOpDmaLQ1YQGf%2BePjdvfFXVeGwacRyuE7yE0Sx_%2BTJcH5LjHQ%40mail.gmail.com.
Jimmy Chen
Jun 5, 2025, 10:21:43 PM6/5/25
to Blake Carver, Archivesspace_Users_Group
You can increase connection pool size in the configure file ( config/config.rb ) like
AppConfig[:db_max_connections] = 50
Andy Bussey
Jun 6, 2025, 4:24:15 AM6/6/25
to Ludwig Possie, Archivesspace_Users_Group
Dear Ludwig
If the issue has only been happening recently, check if you are being crawled. That happened to us very recently and put a huge load on our PUI server. The recourse in that case is to block certain incoming traffic - talk to your IT security folks if you have them. If it is not that and just regular traffic, then increasing the db connections is likely to help
Cheers
Andy
--
You received this message because you are subscribed to the Google Groups "Archivesspace_Users_Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Archivesspace_User...@lyrasislists.org.
To view this discussion visit https://groups.google.com/a/lyrasislists.org/d/msgid/Archivesspace_Users_Group/CA%2BOpDmaLQ1YQGf%2BePjdvfFXVeGwacRyuE7yE0Sx_%2BTJcH5LjHQ%40mail.gmail.com.
Andy Bussey
Head of Library Digital Services
University of Sheffield Library
Ludwig Possie
Jun 6, 2025, 12:18:11 PM6/6/25
to Andy Bussey, Archivesspace_Users_Group
Great advice everyone! I'll increase the db connections and work with our campus IT team to monitor the traffic going to the application. Thank you!!
L
Joshua D. Shaw
Jun 9, 2025, 6:52:46 AM6/9/25
to Andy Bussey, Ludwig Possie, Archivesspace_Users_Group
Also. If you are running version 4.0 and you are running plugins, you may be experiencing a bug fixed in 4.1.0 related to plugin template files (https://github.com/archivesspace/archivesspace/issues/3508)
Joshua
From: 'Ludwig Possie' via Archivesspace_Users_Group <Archivesspac...@lyrasislists.org>
Sent: Friday, June 6, 2025 12:17 PM
To: Andy Bussey <a.bu...@sheffield.ac.uk>
Cc: Archivesspace_Users_Group <archivesspac...@lyrasislists.org>
Subject: Re: [ArchivesSpace Users Group] High CPU and Crash
Sent: Friday, June 6, 2025 12:17 PM
To: Andy Bussey <a.bu...@sheffield.ac.uk>
Cc: Archivesspace_Users_Group <archivesspac...@lyrasislists.org>
Subject: Re: [ArchivesSpace Users Group] High CPU and Crash
Reply all
Reply to author
Forward
0 new messages