Ah, JVM stats. Thanks for that tip: I really wasn't looking at my memory closely enough, and I still had a -Xmx128m set (tomcat7 default in ubuntu). I bumped it to 1g (actually this time, confirmed via /status this time) so I'm hoping this helps resolve.
Ah! That should do it. I am surprised you were able to last a few months with that setting. My preference is somewhere between 2g and 4g.
/status wasn't working for localhost by default, but this was due to IPv6 on my end. Would you be open to changing the default cas.securityContext.adminpages.ip=(127\.0\.0\.1|0:0:0:0:0:0:0:1) to support IPv6 localhost?
No problem. File an issue please.
I'm planning on throwing on a lot more memory monitoring near future. I'm assuming this would be a very likely root cause for these sorts off issues - sorry I didn't catch that earlier.Hoping this is the ticket (hurr hurr) - I'll let you know if anything else interesting comes up, but I really appreciate your support!
After about a month of working perfectly on 4.2.2 deployed to tomcat7, running under java8, randomly the in-memory ticketing system would not create any more tickets. Restarting the tomcat instance fixed it, but I'm wondering why CAS would randomly break on me after working so well! Using a LDAP (AD) backed user base with a mysql backed attribute DB. We have pretty minimal traffic, so I'm not sure why I am seeing issues after such a small amount of time.Despite having an <AsyncRoot level="error">, no errors have been thrown at the time of issue.Unfortunately, I've only had the org.springframework.jdbc logger set to debug, and all others at info, so I have very minimal logging around the issue.I'm noticing both the ldap auth AND the jdbc handlers returning without issues (no errors). ...But no tickets?Here's a sample of the logs:2016-07-19 16:28:16,399 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user1>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:16,401 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>
2016-07-19 16:28:19,015 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user2>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:19,017 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>Immediately before this, I've seen tickets that are created (an audit log is posted that a ticket granting ticket has been created and validated, and all is good). There are no exceptions thrown between when the tickets were able to be created and when there was this bottleneck.On the front end, after the logs say 'success' without a ticket created, they are redirected to the main cas login page. Reproducing this is also difficult as it will stop intermittently, without warning.What are the best ways to debug or resolve these sorts of issues? What could be causing this issue?Thanks in advance,Jeff
After about a month of working perfectly on 4.2.2 deployed to tomcat7, running under java8, randomly the in-memory ticketing system would not create any more tickets. Restarting the tomcat instance fixed it, but I'm wondering why CAS would randomly break on me after working so well! Using a LDAP (AD) backed user base with a mysql backed attribute DB. We have pretty minimal traffic, so I'm not sure why I am seeing issues after such a small amount of time.Despite having an <AsyncRoot level="error">, no errors have been thrown at the time of issue.Unfortunately, I've only had the org.springframework.jdbc logger set to debug, and all others at info, so I have very minimal logging around the issue.I'm noticing both the ldap auth AND the jdbc handlers returning without issues (no errors). ...But no tickets?Here's a sample of the logs:2016-07-19 16:28:16,399 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user1>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:16,401 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>
2016-07-19 16:28:19,015 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user2>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:19,017 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>Immediately before this, I've seen tickets that are created (an audit log is posted that a ticket granting ticket has been created and validated, and all is good). There are no exceptions thrown between when the tickets were able to be created and when there was this bottleneck.On the front end, after the logs say 'success' without a ticket created, they are redirected to the main cas login page. Reproducing this is also difficult as it will stop intermittently, without warning.What are the best ways to debug or resolve these sorts of issues? What could be causing this issue?Thanks in advance,Jeff
After about a month of working perfectly on 4.2.2 deployed to tomcat7, running under java8, randomly the in-memory ticketing system would not create any more tickets. Restarting the tomcat instance fixed it, but I'm wondering why CAS would randomly break on me after working so well! Using a LDAP (AD) backed user base with a mysql backed attribute DB. We have pretty minimal traffic, so I'm not sure why I am seeing issues after such a small amount of time.Despite having an <AsyncRoot level="error">, no errors have been thrown at the time of issue.Unfortunately, I've only had the org.springframework.jdbc logger set to debug, and all others at info, so I have very minimal logging around the issue.I'm noticing both the ldap auth AND the jdbc handlers returning without issues (no errors). ...But no tickets?Here's a sample of the logs:2016-07-19 16:28:16,399 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user1>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:16,401 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>
2016-07-19 16:28:19,015 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user2>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:19,017 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>Immediately before this, I've seen tickets that are created (an audit log is posted that a ticket granting ticket has been created and validated, and all is good). There are no exceptions thrown between when the tickets were able to be created and when there was this bottleneck.On the front end, after the logs say 'success' without a ticket created, they are redirected to the main cas login page. Reproducing this is also difficult as it will stop intermittently, without warning.What are the best ways to debug or resolve these sorts of issues? What could be causing this issue?Thanks in advance,Jeff
After about a month of working perfectly on 4.2.2 deployed to tomcat7, running under java8, randomly the in-memory ticketing system would not create any more tickets. Restarting the tomcat instance fixed it, but I'm wondering why CAS would randomly break on me after working so well! Using a LDAP (AD) backed user base with a mysql backed attribute DB. We have pretty minimal traffic, so I'm not sure why I am seeing issues after such a small amount of time.Despite having an <AsyncRoot level="error">, no errors have been thrown at the time of issue.Unfortunately, I've only had the org.springframework.jdbc logger set to debug, and all others at info, so I have very minimal logging around the issue.I'm noticing both the ldap auth AND the jdbc handlers returning without issues (no errors). ...But no tickets?Here's a sample of the logs:2016-07-19 16:28:16,399 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user1>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:16,401 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>
2016-07-19 16:28:19,015 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user2>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:19,017 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>Immediately before this, I've seen tickets that are created (an audit log is posted that a ticket granting ticket has been created and validated, and all is good). There are no exceptions thrown between when the tickets were able to be created and when there was this bottleneck.On the front end, after the logs say 'success' without a ticket created, they are redirected to the main cas login page. Reproducing this is also difficult as it will stop intermittently, without warning.What are the best ways to debug or resolve these sorts of issues? What could be causing this issue?Thanks in advance,Jeff
Excellent analysis, thank you.
It’s worth a try, I guess, to see if adding Echo to the Cascading DAO removes the issue. I admit person-directory is mostly a mystery to most of us (and 5 does not use the Cascading DAO at all by default). Its evolution has been sinusy.
--Misagh
From: cas-...@apereo.org [mailto:cas-...@apereo.org] On Behalf Of Jeffrey Wong
Sent: Friday, November 18, 2016 12:01 PM
To: CAS Community <cas-...@apereo.org>
After about a month of working perfectly on 4.2.2 deployed to tomcat7, running under java8, randomly the in-memory ticketing system would not create any more tickets. Restarting the tomcat instance fixed it, but I'm wondering why CAS would randomly break on me after working so well! Using a LDAP (AD) backed user base with a mysql backed attribute DB. We have pretty minimal traffic, so I'm not sure why I am seeing issues after such a small amount of time.Despite having an <AsyncRoot level="error">, no errors have been thrown at the time of issue.Unfortunately, I've only had the org.springframework.jdbc logger set to debug, and all others at info, so I have very minimal logging around the issue.I'm noticing both the ldap auth AND the jdbc handlers returning without issues (no errors). ...But no tickets?Here's a sample of the logs:2016-07-19 16:28:16,399 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user1>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:16,400 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:16,401 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>
2016-07-19 16:28:19,015 INFO [org.jasig.cas.authentication.PolicyBasedAuthenticationManager] - <LdapAuthenticationHandler successfully authenticated user2>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL query>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.core.JdbcTemplate] - <Executing prepared SQL statement[SELECT ID,username,FirstName,LastName,Email FROM User WHERE UserName = ?]>
2016-07-19 16:28:19,015 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Fetching JDBC Connection from DataSource>
2016-07-19 16:28:19,017 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - <Returning JDBC Connection to DataSource>Immediately before this, I've seen tickets that are created (an audit log is posted that a ticket granting ticket has been created and validated, and all is good). There are no exceptions thrown between when the tickets were able to be created and when there was this bottleneck.On the front end, after the logs say 'success' without a ticket created, they are redirected to the main cas login page. Reproducing this is also difficult as it will stop intermittently, without warning.What are the best ways to debug or resolve these sorts of issues? What could be causing this issue?Thanks in advance,Jeff