Recommendations for CATALINA_OPTS for cas 5.x with tomcat 8.5.x

[William E.]

Does anyone have any recommendations for CATALINA_OPTS for cas 5.x on tomcat 8?

I am finding that our setup steadily eats up memory to the point that it eventually crashes from out of memory and has to be restarted.

Current settings:

CATALINA_OPTS="-Djava.awt.headless=true -Dfile.encoding=UTF-8 -server -Xms1g -Xmx6g -XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC -XX:-UseCompressedOops"

JAVA_OPTS=$CATALINA_OPTS

Thanks,

William

[Martin Bohun]

What is your:

1. operation system

2. how much RAM do you have

3. how much swap do you have

if you are on  Linux you can do:

1.    uname -a

2-3. free -m

and post the output here

regards,

martin

[William E.]

RHEL 7, 8GB ram, swap is 4GB.  It's a VM in our vSphere cluster+SAN.  I actually have three, two PROD nodes behind a load balancer and one test node.  All have same specs and all show the issue.  Steadily chews up memory until eventual crash, 1-6 hours depending on load.

The asme servers were running cas 3.6 . + shibboleth 3.3.x for quite a while without memory issues.  Upgraded and tried to consolidate to just cas 5, using it's saml2 capabilities to replace the shibboleth component.  But, it's not going as well as I had hoped.

Been working with Unicon Support on it, but it appears to be a memory leak in cas 5.2, based on heap analysis.  So I am kinda of stuck.

Thanks for your help!

[Jeffrey Ramsay]

I have been using this setup within our VMware environment without any problems.

$ cat bin/setenv.sh
export JAVA_HOME=/u01/app/badm/apps/java/jdk1.8.0_152
export CATALINA_HOME=/u01/app/badm/apps/dev/apache-tomcat-8-auth
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/apr/lib"
export JAVA_OPTS="$JAVA_OPTS -Dcatalina.home=$CATALINA_HOME"
JAVA_OPTS="$JAVA_OPTS -server -d64 -Xms2048m -Xmx2048m"
JAVA_OPTS="$JAVA_OPTS -XX:NewSize=768m -XX:MaxNewSize=768m"
JAVA_OPTS="$JAVA_OPTS -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=1024m"
JAVA_OPTS="$JAVA_OPTS -XX:SurvivorRatio=12 -XX:MaxTenuringThreshold=0"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC"
JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC -XX:+UseTLAB"
JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
JAVA_OPTS="$JAVA_OPTS -Djava.security.egd=file:/dev/./urandom"
JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$CATALINA_HOME/lib"
JAVA_OPTS="$JAVA_OPTS -Dcatalina.home=$CATALINA_HOME"
JAVA_OPTS="$JAVA_OPTS -Dlocal.logs=$CATALINA_HOME/logs"
JAVA_OPTS="$JAVA_OPTS -Dcas.log.dir=$CATALINA_HOME/logs"
JAVA_OPTS="$JAVA_OPTS -Dcas.standalone.config=/etc/cas/5/dev516"
rm -rf $CATALINA_HOME/logs/* >/dev/null 2>&1

-Jeff

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/bfe6c835-bf1e-4f24-b507-025d7c0e3172%40apereo.org.

[Martin Bohun]

I have seen the behavior you are describing when people ran cas (tomcat, mysql, etc.) on a (what I would consider a misconfigured) Linux box with 0 swap.

However you are saying you have 4gb of swap.

I still do prefer to set my swap to 2 * $MY_RAM; can you try that? adjust or add a swapfile to your swap (so you have 8gb RAM / 16gb swap), I am curious if that would help / solve your problem?

What error messages are you getting in the jvm and syslog/systemd journal from the OS?

regards,

martin

[William E.]

Martin,

Thank you.  You might be on to something.  I was quoting from memory and I was wrong on swap.  Of the two nodes, both in my mind identical VM's, the secondary node has 8GB of swap and a tiny bit used, but the primary, the one that is crashing, has no swap configured.  I have requested our systems team add 8GB of swap to the primary.

Primary server:

              total        used        free      shared  buff/cache   available

Mem:        8010840     4872660      420488      107484     2717692     2679336

Swap:             0           0           0

Secondary server:

              total        used        free      shared  buff/cache   available

Mem:        8010972     1192296     1530500       23196     5288176     6449948

Swap:       8388604        4604     8384000

Not sure I understand why it would matter since in theory swap should not be needed on a server with 8GB of ram with jvm limit set to 6GB though.  Any more insight on why, because I would really like to understand the reason.


Additionally, I've put the shibboleth IDP back into play, effectively rendering the saml services in cas "unused".  I am using proxy_ajp to front tomcat with apache so it was easy to copy the idp.war into tomcat and re-enable the shib-cas-authenticator. I guess my hope of moving from cas+shibb. to just cas will have to wait....

Thanks, 

William

P.S. Jeff, thank you for posting your catalina opts!

[Uxío Prego]

Swap is good, generally, but as more dedicated is the server, it should make less of a difference, because of your -Xmx configuration.

I don't know about the specific numbers of version 5, but (pending knowing how many concurrent sessions do you normally manage) maybe the server is having a deployment problem (maybe not).

I think 8G should be enough for your case, but I don't really know. While you keep investigating, maybe adding swap and more memory can help you... maybe not!

Good luck with it,

Uxío Prego

             

Madiva Soluciones
CL / SERRANO GALVACHE 56
BLOQUE ABEDUL PLANTA 4
28033 MADRID
+34 917 56 84 94
www.madiva.com

www.bbva.com

The activity of email inboxes can be systematically tracked by colleagues, business partners and third parties. Turn off automatic loading of images to hamper it.

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/7e36f7d2-3bf7-49d2-bcd8-bbc0e22b901b%40apereo.org.

[William E.]

Thanks Uxio.

Luckily, since changing to delegating IDP functions to Shibboleth, as it was on cas 3 setup on this same server, memory seems to be stable.  That and adding the -XX:-UseCompressedOops opt recommended by Unicon support(many thanks!).  Not entirely sure which item gets the kudos, perhaps both, but I will test a bit more with one or the other over the holidays to try and determine which was the factor.

Screenshot of stats.

Primary server now has 8GB of swap.  Added to server while up without disruption.  Linux rocks!

Thanks,

William

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.

[David Curry]

Once you're satisfied that it's working correctly, could you share your options/settings in this thread? I know I (and probably others) will be coming to this point Real Soon Now and the additional knowledge would be helpful.

Thanks,

--Dave

--

DAVID A. CURRY, CISSP
DIRECTOR OF INFORMATION SECURITY
INFORMATION TECHNOLOGY

71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003
+1 212 229-5300 x4728 • david...@newschool.edu

To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+unsubscribe@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/0a78c6f4-9f32-4456-8adc-9f82ff6d7c56%40apereo.org.

[William E.]

Our campus is shutting down for the holidays today and it looks like thus far we haven't crashed in almost 24 hours.  So FWIW below are my config params in hopes it might help others.  I learn best by examples myself.  If those on this list have suggestions on better ways to do the things we're doing, please let me know.  Happy to learn.

One item of note, cas 5.1 and 5.2 saw a json config param change:

# json service registry

#

# cas 5.1

cas.serviceRegistry.config.location=file:/etc/cas/config/services

#

# cas 5.2

cas.serviceRegistry.json.location=file:/etc/cas/config/services

My cas.properties:

cas.server.name: https://sso.uah.edu

cas.server.prefix: https://sso.uah.edu/cas

cas.adminPagesSecurity.ip=127.0.0.1

cas.adminPagesSecurity.loginUrl=${cas.server.prefix}/login

cas.adminPagesSecurity.service=${cas.server.prefix}/status/dashboard

cas.adminPagesSecurity.users=file:/etc/cas/config/users.properties

cas.adminPagesSecurity.adminRoles[0]=ROLE_ADMIN

logging.config: file:/etc/cas/config/log4j2.xml

cas.slo.disabled=true

cas.ticket.tgt.maxTimeToLiveInSeconds=36000

cas.ticket.tgt.timeToKillInSeconds=14400

cas.authn.ldap[0].type=AUTHENTICATED

cas.authn.ldap[0].ldapUrl=<redacted>

cas.authn.ldap[0].useSsl=true

cas.authn.ldap[0].baseDn=ou=People,dc=uah,dc=edu

cas.authn.ldap[0].userFilter=(|(uid={user})(mail={user}))

cas.authn.ldap[0].bindDn=<redacted>

cas.authn.ldap[0].bindCredential=<redacted>

cas.authn.ldap[0].principalAttributeList=uid,mail

cas.authn.attributeRepository.ldap[0].order=0

cas.authn.attributeRepository.ldap[0].ldapUrl=<redacted>

cas.authn.attributeRepository.ldap[0].useSsl=true

cas.authn.attributeRepository.ldap[0].useStartTls=false

cas.authn.attributeRepository.ldap[0].baseDn=ou=People,dc=uah,dc=edu

cas.authn.attributeRepository.ldap[0].bindDn=<redacted>

cas.authn.attributeRepository.ldap[0].bindCredential=<redacted>

cas.authn.attributeRepository.ldap[0].userFilter=(|(uid={user})(mail={user}))

cas.authn.attributeRepository.ldap[0].attributes.uid=uid

cas.authn.attributeRepository.ldap[0].attributes.ou=ou

cas.authn.attributeRepository.ldap[0].attributes.o=o

cas.authn.attributeRepository.ldap[0].attributes.displayName=displayName

cas.authn.attributeRepository.ldap[0].attributes.cn=cn

cas.authn.attributeRepository.ldap[0].attributes.mail=mail

cas.authn.attributeRepository.ldap[0].attributes.mailLocalAddress=mailLocalAddress

cas.authn.attributeRepository.ldap[0].attributes.member=member

cas.authn.attributeRepository.ldap[0].attributes.memberof=memberof

cas.authn.attributeRepository.ldap[0].attributes.sn=sn

cas.authn.attributeRepository.ldap[0].attributes.uahUDCID=UDC_IDENTIFIER

cas.authn.attributeRepository.ldap[0].attributes.displayName=displayName

cas.authn.attributeRepository.ldap[0].attributes.givenName=givenName

cas.authn.attributeRepository.ldap[0].attributes.telephoneNumber=telephoneNumber

cas.authn.attributeRepository.ldap[0].attributes.title=title

cas.authn.attributeRepository.ldap[0].attributes.employeeNumber=employeeNumber

cas.authn.attributeRepository.ldap[0].attributes.eduPersonAffiliation=eduPersonAffiliation

cas.authn.attributeRepository.ldap[0].attributes.eduPersonPrimaryAffiliation=eduPersonPrimaryAffiliation

cas.authn.attributeRepository.ldap[0].attributes.eduPersonEntitlement=eduPersonEntitlement

cas.authn.attributeRepository.ldap[0].attributes.eduPersonPrincipalName=eduPersonPrincipalName

cas.authn.attributeRepository.ldap[0].attributes.uahEduHomeLaborAcct=uahEduHomeLaborAcct

cas.authn.attributeRepository.ldap[0].attributes.physicalDeliveryOfficeName=physicalDeliveryOfficeName

cas.personDirectory.principalAttribute=uid,mail

cas.personDirectory.returnNull=false

cas.personDirectory.principalResolutionFailureFatal=false

cas.authn.accept.users=

cas.serviceRegistry.json.location=file:/etc/cas/config/services

cas.serviceRegistry.watcherEnabled=true

cas.serviceRegistry.repeatInterval=240000

cas.serviceRegistry.initFromJson=true

cas.samlSP.inCommon.signatureLocation=/etc/cas/saml/inc-md-public-key.pem

cas.authn.samlIdp.entityId=https://sso.uah.edu/idp/shibboleth

cas.authn.samlIdp.scope=uah.edu

cas.authn.samlIdp.metadata.location=file:/etc/cas/saml

cas.authn.samlIdp.response.useAttributeFriendlyName=true

cas.authn.samlIdp.response.attributeNameFormats=uid->uri,mail->uri

management.contextPath=/status

management.security.enabled=true

management.security.roles=ACTUATOR,ADMIN

management.security.sessions=if_required

cas.monitor.endpoints.enabled=true

cas.monitor.endpoints.sensitive=false

cas.monitor.endpoints.dashboard.enabled=true

cas.monitor.endpoints.dashboard.sensitive=false

cas.monitor.endpoints.status.enabled=true

cas.monitor.endpoints.status.sensitive=false

Current memory usage:

$ free

              total        used        free      shared  buff/cache   available

Mem:        8010840     5539196      480504      107080     1991140     2013464

Swap:       8387580         368     8387212

Happy holidays all.

-W