CAS 3.5.x CPU utilization problem

124 views
Skip to first unread message

abdelrahman halawa

unread,
Jul 28, 2019, 7:51:49 AM7/28/19
to CAS Community
Hello all,

I noticed that the CPU utilization of my CAS server reaches above 90% with only 75 concurrent users or maybe less.

Has this happened with you as well? Any suggestions to overcome this?

I use embedded Tomcat with default settings.

Thanks in advance. 

David Curry

unread,
Jul 28, 2019, 10:44:12 AM7/28/19
to cas-...@apereo.org
How many CPUs?
How much memory?
What operating system?
What else is running on the server (nothing, hopefully)?
Is the server paging or swapping (you don't want it to be)?
If you're running on Linux VMs, do you have an entropy source for the crypto (you should)? 


--

DAVID A. CURRY, CISSP
DIRECTOR • INFORMATION SECURITY & PRIVACY
THE NEW SCHOOL • INFORMATION TECHNOLOGY

71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003
+1 646 909-4728 • david...@newschool.edu



--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/fe73e941-cf70-4d77-a747-1f1c4d76a527%40apereo.org.

abdelrahman halawa

unread,
Jul 28, 2019, 11:00:34 AM7/28/19
to cas-...@apereo.org
How many CPUs?
     4X Intel(R) Xeon(R) @ 2.40GHz, 2400 Mhz, 2 Core(s)
How much memory?
     16G
What operating system?
    Windows Server 2012
What else is running on the server (nothing, hopefully)?
    Nothing
Is the server paging or swapping (you don't want it to be)?
    The default of Windows Server



To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CA%2Bd9XAPPO%3D_pb%3Dg7_f5T28aUWLWEkYpyTNn_p27cq5tZQqLcOw%40mail.gmail.com.


--
Best regards,
------------------------------------
​
photo 
Abdelrahman Halawa
Teacher Assistant, Computer and Systems Department, Al-Azhar University
+2 01008131693 | abdelrah...@gmail.com | Skype: abdelrahmanhalawa | Maadi, Cairo, Egypt

David Curry

unread,
Jul 28, 2019, 11:47:55 AM7/28/19
to cas-...@apereo.org
Well, I'm not too knowledgeable about Windows Server, but that seems reasonable. Is the server in general at 90% CPU, or is it the actual Java process that's at 90%? 

I don't run the embedded server so don't know its defaults -- What's the Java heap size limited to? Should be less than the physical memory. What garbage collector is it using? On our 4-core 16GB Linux servers we run this:

jsvc.exec -Dcatalina.home=/opt/tomcat/latest -Dcatalina.base=/opt/tomcat/latest -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/opt/tomcat/latest/conf/logging.properties -cp /opt/tomcat/latest/bin/commons-daemon.jar:/opt/tomcat/latest/bin/bootstrap.jar:/opt/tomcat/latest/bin/tomcat-juli.jar -pidfile /var/run/tomcat.pid -java-home /usr/lib/jvm/java-openjdk -user tomcat -Xms512M -Xmx8192M -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:MaxGCPauseMillis=500 -server org.apache.catalina.startup.Bootstrap

which is enough for CAS with Hazelcast ticket registry and the management webapp (all in the same Tomcat process). We're also running a small mongod (2gb cache size) on each server which handles the services registry. Most of the time, the servers are idle or very close to it.

Could you have something that takes a (relatively) long time that causing the server to block? A slow LDAP, or slow database query, or something? We had problems like that when we were using MongoDB as our ticket registry; under heavy activity Mongo was too slow and the CAS threads would block waiting on it to complete. It wasn't pretty.

--Dave


--

DAVID A. CURRY, CISSP
DIRECTOR • INFORMATION SECURITY & PRIVACY
THE NEW SCHOOL • INFORMATION TECHNOLOGY

71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003
+1 646 909-4728 • david...@newschool.edu


To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAMKQGEb5aaQ6-vNkqsC_XfZt8Y0HN5H%3D6iFwPC%3DeHsa9hPu2oA%40mail.gmail.com.

abdelrahman halawa

unread,
Jul 28, 2019, 12:36:19 PM7/28/19
to cas-...@apereo.org
Nice, I will try your JVM settings.
By the way, the CPU reaches about 100% only with Java (CAS).
Here my Tomcat configuration:

server.max-http-header-size=2097152
server.use-forward-headers=true
server.connection-timeout=20000
server.error.include-stacktrace=NEVER
server.compression.enabled=true
server.compression.mime-types=application/javascript,application/json,application/xml,text/html,text/xml,text/plain
server.tomcat.max-http-post-size=2097152
server.tomcat.min-spare-threads=20
server.tomcat.max-threads=200
server.tomcat.accept-count=1024
server.tomcat.max-connections=10000
server.tomcat.port-header=X-Forwarded-Port
server.tomcat.protocol-header=X-Forwarded-Proto
server.tomcat.protocol-header-https-value=https
server.tomcat.remote-ip-header=X-FORWARDED-FOR
server.tomcat.uri-encoding=UTF-8

Could you share your Tomcat configuration with me?
I want to check if the CPU issue related to Tomcat misconfiguration or not


To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CA%2Bd9XAO2bPNFfUdz6BCgyjA8tqrYWrgy__JmKd2H0ocAySkVYQ%40mail.gmail.com.

abdelrahman halawa

unread,
Jul 29, 2019, 7:33:51 AM7/29/19
to cas-...@apereo.org
Hi david.curry,

How are you

Could you tell me the Tomcat version you use?

Also, If you don't mind, reply to the previous email

David Curry

unread,
Jul 29, 2019, 8:32:19 AM7/29/19
to cas-...@apereo.org
At the moment we're using Tomcat 8.5.x and Java 1.8.x (OpenJDK).

I don't have Tomcat settings like the ones you show (property file style), because I use an external Tomcat and it's configured with XML files. The settings are the same as what's documented here:


The only performance-related changes there would be enabling resource caching, enabling asynchronous requests (I don't believe CAS will work without this), and making sure that asynchronous logging is enabled.

The SSL configuration does enable a better-performing SSL library, but I can't imagine that your problem lies there. (My reasons for enabling it were less for performance reasons and more for better support of newer ciphers, etc.)


--

DAVID A. CURRY, CISSP
DIRECTOR • INFORMATION SECURITY & PRIVACY
THE NEW SCHOOL • INFORMATION TECHNOLOGY

71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003
+1 646 909-4728 • david...@newschool.edu


To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAMKQGEYHXukSCO6pSrPRhKAT4Tnx6tY9PhP2abp1k_o_snR2wg%40mail.gmail.com.

abdelrahman halawa

unread,
Jul 29, 2019, 10:39:10 AM7/29/19
to cas-...@apereo.org
OK. Thanks for helping.

Is the number of transactions in the picture is high, if yes, Does this explains the CPU utilization?
image.png


To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CA%2Bd9XAMqgtgwWQ_Awa%2B8L3K3kQpveMM_CLieicYK97YwPmv04Q%40mail.gmail.com.

David Curry

unread,
Jul 29, 2019, 2:47:26 PM7/29/19
to cas-...@apereo.org
Did you say 75 concurrent users? That does seem kind of high to me unless they're all logging in at exactly the same time. You'll see several back-and-forths with the client each time someone logs in, but by "several" I mean 5 or 10 (depending on different conditions), not tens or hundreds.


--

DAVID A. CURRY, CISSP
DIRECTOR • INFORMATION SECURITY & PRIVACY
THE NEW SCHOOL • INFORMATION TECHNOLOGY

71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003
+1 646 909-4728 • david...@newschool.edu


To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAMKQGEbWcuqmJkmmk_aoWw4u1UKz-ibXoLyWeHRfv-oEBViU6Q%40mail.gmail.com.

abdelrahman halawa

unread,
Jul 30, 2019, 4:36:33 AM7/30/19
to cas-...@apereo.org
Sorry, there is something I have to clarify it. The picture above is taken from load testing by Jmeter tool at 200 concurrent users for 5 minutes.
My question was about, Is the high number of transactions/s explains why the CPU utilization is so high?   


To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CA%2Bd9XAMyHUu1H%2B%3DTqJuQEXe%3DYp5tRmBKXsRzbKa%3D6905p%3DR%3D-g%40mail.gmail.com.

abdelrahman halawa

unread,
Jul 30, 2019, 6:44:35 AM7/30/19
to cas-...@apereo.org
Hi David,

Thank you for cooperation.
Kindly, Could you tell me what is the maximum concurrent users can your own CAS server (one CAS server) service them without any kind of errors?

David Curry

unread,
Jul 30, 2019, 7:59:35 AM7/30/19
to cas-...@apereo.org
Well, sure, that could explain it. But I wouldn't say it's positively the reason.

--

DAVID A. CURRY, CISSP
DIRECTOR • INFORMATION SECURITY & PRIVACY
THE NEW SCHOOL • INFORMATION TECHNOLOGY

71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003
+1 646 909-4728 • david...@newschool.edu


To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAMKQGEa_CgFo1VcdBxDtmgDzFLy4%3DxXPRXz5A9h778oNao0Bng%40mail.gmail.com.

Lucas Francisco Delgado Duarte

unread,
Jul 30, 2019, 11:23:06 AM7/30/19
to cas-...@apereo.org
My production server uses CAS 5.3 with over 30k users and no high CPU usage.
Apache (load balancer) with 3 JBoss instances, ldap auth and hazelcast for web and ticket registry cache.

I recommend you to start your jvm with CPU profiling tools (i like to use JProfiler). With it, you can evaluate which methods are consuming your CPU time and try to resolve your problem.

[]'s,
kanedb

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CA%2Bd9XAOjjScfyKFfLCPZ25xNA64%3DHbXfhcHs1mTUYhW-dshOdg%40mail.gmail.com.

abdelrahman halawa

unread,
Jul 30, 2019, 4:03:46 PM7/30/19
to CAS Community
Hi Kanedb,
OK, I'm going to do, thank you for your help.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.


--
Best regards,
------------------------------------
​
photo 
Abdelrahman Halawa
Teacher Assistant, Computer and Systems Department, Al-Azhar University
+2 01008131693 | abdelrahmanhalawa...@gmail.com | Skype: abdelrahmanhalawa | Maadi, Cairo, Egypt

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.


--
Best regards,
------------------------------------
​
photo 
Abdelrahman Halawa
Teacher Assistant, Computer and Systems Department, Al-Azhar University
+2 01008131693 | abdelrahmanhalawa...@gmail.com | Skype: abdelrahmanhalawa | Maadi, Cairo, Egypt


--
Best regards,
------------------------------------
​
photo 
Abdelrahman Halawa
Teacher Assistant, Computer and Systems Department, Al-Azhar University
+2 01008131693 | abdelrahmanhalawa...@gmail.com | Skype: abdelrahmanhalawa | Maadi, Cairo, Egypt

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.


--
Best regards,
------------------------------------
​
photo 
Abdelrahman Halawa
Teacher Assistant, Computer and Systems Department, Al-Azhar University
+2 01008131693 | abdelrahmanhalawa...@gmail.com | Skype: abdelrahmanhalawa | Maadi, Cairo, Egypt

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.


--
Best regards,
------------------------------------
​
photo 
Abdelrahman Halawa
Teacher Assistant, Computer and Systems Department, Al-Azhar University
+2 01008131693 | abdelrahmanhalawa...@gmail.com | Skype: abdelrahmanhalawa | Maadi, Cairo, Egypt


--
Best regards,
------------------------------------
​
photo 
Abdelrahman Halawa
Teacher Assistant, Computer and Systems Department, Al-Azhar University
+2 01008131693 | abdelrahmanhalawa...@gmail.com | Skype: abdelrahmanhalawa | Maadi, Cairo, Egypt

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-...@apereo.org.

Alfonso Veraluz

unread,
Aug 2, 2019, 6:54:43 AM8/2/19
to CAS Community
You can try to raise the server.tomcat.max-threads from 200 to 800 and check if Tomcat performs better. 
Reply all
Reply to author
Forward
0 new messages