[cas-user] Clustered CAS 3.4.11: Stops working at nightly reboot

[Nicolaie Szabadkai]

Hello,

out 2 clustered Tomcats each with its own CAS working as cluster (tomcat session-replication via TCP and JBOSS-TreeCache for tickets) are usualy rebooted nightly with a gap of one hour.
Last night one of them failed to restart with the following error that we never saw before.
In the morning one of our admins realized that and manually restarted but it is somehow strange.

Do you have an idea what the cause might be?

There are running on tomcat 6.0.32, java 1.6.0_32_b05.


2012-08-01 01:54:42,536 ERROR [org.jasig.cas.web.init.SafeContextLoaderListener] - SafeContextLoaderListener:
The Spring ContextLoaderListener we wrap threw on contextInitialized.
But for our having caught this error, the web application context would not have initialized.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ticketRegistry' defined in ServletContext resource [/WEB-INF/spring-c
onfiguration/ticketRegistry.xml]: Cannot resolve reference to bean 'cache' while setting bean property 'cache'; nested exception is org.springframework.beans
.factory.BeanCreationException: Error creating bean with name 'cache' defined in ServletContext resource [/WEB-INF/spring-configuration/ticketRegistry.xml]:
Invocation of init method failed; nested exception is org.jboss.cache.CacheException: Unable to invoke method public void org.jboss.cache.RPCManagerImpl.star
t() on object instance org.jboss.cache.RPCManagerImpl@3432a325
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1325)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1086)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
at org.jasig.cas.web.init.SafeContextLoaderListener.contextInitialized_aroundBody0(SafeContextLoaderListener.java:62)
at org.jasig.cas.web.init.SafeContextLoaderListener.contextInitialized_aroundBody1$advice(SafeContextLoaderListener.java:44)
at org.jasig.cas.web.init.SafeContextLoaderListener.contextInitialized(SafeContextLoaderListener.java:1)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cache' defined in ServletContext resource [/WEB-INF/spring-configuration/ticketRegistry.xml]: Invocation of init method failed; nested exception is org.jboss.cache.CacheException: Unable to invoke method public void org.jboss.cache.RPCManagerImpl.start() on object instance org.jboss.cache.RPCManagerImpl@3432a325
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1420)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322)
... 42 more
Caused by: org.jboss.cache.CacheException: Unable to invoke method public void org.jboss.cache.RPCManagerImpl.start() on object instance org.jboss.cache.RPCManagerImpl@3432a325
at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:127)
at org.jboss.cache.factories.ComponentRegistry$PrioritizedMethod.invoke(ComponentRegistry.java:932)
at org.jboss.cache.factories.ComponentRegistry.internalStart(ComponentRegistry.java:718)
at org.jboss.cache.factories.ComponentRegistry.start(ComponentRegistry.java:582)
at org.jboss.cache.invocation.CacheInvocationDelegate.start(CacheInvocationDelegate.java:279)
at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:95)
at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:79)
at org.jboss.cache.DefaultCacheFactory.createCache(DefaultCacheFactory.java:146)
at org.jasig.cas.util.JBossCacheFactoryBean.afterPropertiesSet(JBossCacheFactoryBean.java:49)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1477)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1417)
... 49 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:123)
... 59 more
Caused by: org.jboss.cache.CacheException: Unable to fetch state on startup
at org.jboss.cache.RPCManagerImpl.start(RPCManagerImpl.java:159)
... 64 more
Caused by: org.jgroups.StateTransferException: 1.8.6.35:58525 could not fetch state null from null
at org.jgroups.JChannel.connect(JChannel.java:466)
at org.jboss.cache.RPCManagerImpl.start(RPCManagerImpl.java:147)
... 64 more
Caused by: org.jgroups.StateTransferException: 1.8.6.35:58525 could not fetch state null from null
at org.jgroups.JChannel.connect(JChannel.java:459)
... 65 more


Cheers
Nicolaie

--
You are currently subscribed to cas-...@lists.jasig.org as: jasig-cas-user...@googlegroups.com
To unsubscribe, change settings or access archives, see http://www.ja-sig.org/wiki/display/JSG/cas-user

[Marvin Addison]

> out 2 clustered Tomcats each with its own CAS working as cluster (tomcat session-replication via TCP and JBOSS-TreeCache for tickets) are usualy rebooted nightly with a gap of one hour.

I'm curious why you do a nightly restart. While some truly
misbehaving applications need this, it's very uncommon for such
drastic measures to be required for CAS. I've never heard of it in my
many years in the CAS community.

> Do you have an idea what the cause might be?

No idea, but from the stack trace it looks like a TreeCache issue that
is bubbling up to CAS.

> Caused by: org.jgroups.StateTransferException: 1.8.6.35:58525 could not fetch state null from null
> at org.jgroups.JChannel.connect(JChannel.java:466)
> at org.jboss.cache.RPCManagerImpl.start(RPCManagerImpl.java:147)
> ... 64 more
> Caused by: org.jgroups.StateTransferException: 1.8.6.35:58525 could not fetch state null from null
> at org.jgroups.JChannel.connect(JChannel.java:459)
> ... 65 more

This smells like a networking problem. I'm only vaguely familiar with
TreeCache and JGroups, but it's my experience that many if not most
cache issues are due to replication and state transfer failures. That
looks to be the case here. I'm curious what replication method you're
using; TCP or UDP multicast. In any case you should probably
concentrate on replication and possibly do some network traffic
capture and analysis, and possibly post to the jgroups forum as a
follow up.

M