cas 7.1.1 Deadlock caused by changes in Commit 999c8ce
UD Heavy
Deadlock Caused by Changes in Commit 999c8ce (05.09.2024)
Issue:
Following the implementation of changes introduced in commit 999c8ce (dated 05.09.2024), a deadlock issue has been identified under high system load. Load testing of CAS revealed that the application hangs when processing 16 messages per second. The issue occurs on machines with the following configuration:
- 4 GB of RAM
- 4 CPU cores
Steps to Reproduce:
- Launch the system with the specified configuration.
- Apply a load of 16 messages per second.
- The system hangs before the application fully starts.
Luís Costa
Just a brief opinion about deadlocks on CAS (in my case CAS 6.6.14).
At some point, when adding new modules (like SAML or a cache ticket registry), our cas server frezzed/hung on startup.
After intensive analysis with tools like VisualVM and jstack, we found out the hang was related to log4j2 asynchronous configuration.
Our solution was to turn off log4j2 asynchronous (like described in https://apereo.github.io/cas/development/logging/Logging.html#log4j2-properties) and also to use a more recent version of log4j2 (to avoid other problems I don't remember).
I don't know if you're having a related issue, but I leave here our experience.
Kind regards,
Luís Costa
UD Heavy
Update on Deadlock Issue
After further investigation, the deadlock issue under high system load has been traced to a problem with virtual threads pinning. Specifically, the issue arises because a new virtual thread is created for each incoming request. However, these virtual threads fail to release the native threads they are pinned to, leading to thread exhaustion and causing the application to hang.
The issue can be temporarily resolved by disabling virtual threads in the configuration using the following property:
spring.threads.virtual.enabled=false.