Weird and sudden error during deployment.

39 views
Skip to first unread message

Carl-Magnus Björkell

unread,
Nov 17, 2011, 8:18:03 AM11/17/11
to mobicents-public
Hey all, I run into a very puzzling problem today, and I was hoping
that someone might have some ideas about where to look.

Background: We have had working code running on a 2.5.0 SLEE server
now for quite some time. Yesterday we merged a branch that added two
new Sbbs to the DUs, and did modifications to two others. These were
pretty minor changes, and I don't see how they could affect
deployment. Either way, now we began seing the following stack trace
when starting the SLEE:

(org.mobicents category using TRACE level)

12:47:27,747 TRACE [ConcreteClassGeneratorUtils]
com.yyy.zzz.slee.calllayer.base.CallSbbUsageParametersImpl Implements
link with com.yyy.zzz.slee.calllayer.base.CallSbbUsageParameters
interface created
12:47:27,747 TRACE [ConcreteClassGeneratorUtils]
com.yyy.zzz.slee.calllayer.base.CallSbbUsageParametersImpl Inheritance
link with org.mobicents.slee.runtime.usage.AbstractUsageParameterSet
class created
12:47:27,747 TRACE [ConcreteUsageParameterClassGenerator]
generateConcreteMethod(): USAGEPARAM variable is currentCalls
12:47:27,747 TRACE [ConcreteUsageParameterClassGenerator] Generating
usage method = incrementCurrentCalls
12:47:27,747 TRACE [ConcreteUsageParameterClassGenerator] Adding SLEE
1.0 usage param getter with source public long
getCurrentCalls( boolean reset) { return
super.getParameter("currentCalls",reset);}
12:47:27,755 INFO [DeploymentMBeanImpl] Uninstalled
SbbID[name=CapCallSbb,vendor=com.yyy.zzz.slee,version=0.9.1-SNAPSHOT]
due to tx rollback
12:47:27,755 INFO [DeploymentMBeanImpl] Uninstalled
ServiceID[name=CAP Call Layer,vendor=com.yyy.zzz.slee,version=0.9.1-
SNAPSHOT] due to tx rollback
12:47:27,756 DEBUG [DeploymentMBeanImpl] Exception while completing
transaction
javax.slee.management.ManagementException: Exception while completing
transaction
at
org.mobicents.slee.container.management.jmx.DeploymentMBeanImpl.install(DeploymentMBeanImpl.java:
309)
at
org.mobicents.slee.container.deployment.jboss.action.InstallDeployableUnitAction.invoke(InstallDeployableUnitAction.java:
41)
at
org.mobicents.slee.container.deployment.jboss.DeploymentManager.sciAction(DeploymentManager.java:
323)
at
org.mobicents.slee.container.deployment.jboss.DeploymentManager.installDeployableUnit(DeploymentManager.java:
188)
at
org.mobicents.slee.container.deployment.jboss.SLEESubDeployer.start(SLEESubDeployer.java:
281)
at
org.mobicents.slee.container.deployment.jboss.SleeContainerDeployerImpl.start(SleeContainerDeployerImpl.java:
262)
at
org.mobicents.slee.container.deployment.jboss.SLEEDeployer.callSubDeployer(SLEEDeployer.java:
124)
at
org.mobicents.slee.container.deployment.jboss.SLEEDeployer.setInternalDeployer(SLEEDeployer.java:
170)
at
org.mobicents.slee.container.deployment.jboss.SleeContainerDeployerImpl
$1.run(SleeContainerDeployerImpl.java:133)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
at
org.mobicents.slee.container.component.deployment.DeployableUnitImpl.deletePath(DeployableUnitImpl.java:
309)
at
org.mobicents.slee.container.component.deployment.DeployableUnitImpl.undeploy(DeployableUnitImpl.java:
299)
at
org.mobicents.slee.container.management.jmx.DeploymentMBeanImpl.install(DeploymentMBeanImpl.java:
302)
... 14 more
12:47:27,758 ERROR [SleeContainerDeployerImpl]
java.lang.NoClassDefFoundError: org/mobicents/slee/container/
deployment/jboss/action/ManagementAction$Type
java.util.concurrent.ExecutionException:
java.lang.NoClassDefFoundError: org/mobicents/slee/container/
deployment/jboss/action/ManagementAction$Type
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.mobicents.slee.container.deployment.jboss.SleeContainerDeployerImpl.sleeInitialization(SleeContainerDeployerImpl.java:
201)
at
org.mobicents.slee.container.SleeContainer.initSlee(SleeContainer.java:
572)
at
org.mobicents.slee.container.management.jmx.SleeManagementMBeanImpl.startSlee(SleeManagementMBeanImpl.java:
645)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:
59)
at
org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:
150)
at
org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:
66)
at org.jboss.kernel.plugins.dependency.KernelControllerContextAction
$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:
241)
at
org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:
47)
at
org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:
109)
at
org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:
70)
at
org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:
221)
at
org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:
54)
at
org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:
42)
at
org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:
62)
at
org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:
71)
at
org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:
51)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
774)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
540)
at
org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:
121)
at
org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:
51)
at
org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:
62)
at
org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:
50)
at
org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:
171)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:
1439)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:
1157)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:
1178)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:
1098)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
553)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:
781)
at
org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:
702)
at
org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:
117)
at
org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:
70)
at
org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:
53)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:
361)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
553)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(AbstractProfileService.java:
306)
at
org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:
271)
at
org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:
461)
at org.jboss.Main.boot(Main.java:221)
at org.jboss.Main$1.run(Main.java:556)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NoClassDefFoundError: org/mobicents/slee/
container/deployment/jboss/action/ManagementAction$Type
at
org.mobicents.slee.container.deployment.jboss.action.DeployManagementAction.getType(DeployManagementAction.java:
55)
at
org.mobicents.slee.container.deployment.jboss.DeploymentManager.sciAction(DeploymentManager.java:
372)
at
org.mobicents.slee.container.deployment.jboss.DeploymentManager.installDeployableUnit(DeploymentManager.java:
188)
at
org.mobicents.slee.container.deployment.jboss.SLEESubDeployer.start(SLEESubDeployer.java:
281)
at
org.mobicents.slee.container.deployment.jboss.SleeContainerDeployerImpl.start(SleeContainerDeployerImpl.java:
262)
at
org.mobicents.slee.container.deployment.jboss.SLEEDeployer.callSubDeployer(SLEEDeployer.java:
124)
at
org.mobicents.slee.container.deployment.jboss.SLEEDeployer.setInternalDeployer(SLEEDeployer.java:
170)
at
org.mobicents.slee.container.deployment.jboss.SleeContainerDeployerImpl
$1.run(SleeContainerDeployerImpl.java:133)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:
471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
... 1 more
Caused by: java.lang.ClassNotFoundException: Failed to load class
org.mobicents.slee.container.deployment.jboss.action.ManagementAction
$Type
at
org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:
170)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:
265)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:
1119)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:
798)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:
441)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 14 more
12:47:27,766 ERROR [AbstractKernelController] Error installing to
Start: name=Mobicents.JAINSLEE.SleeManagement state=Create
java.lang.NoClassDefFoundError: org/jboss/cache/eviction/EvictionEvent
at org.jboss.cache.RegionManagerImpl.getRegion(RegionManagerImpl.java:
244)
at org.jboss.cache.RegionManagerImpl.getRegion(RegionManagerImpl.java:
207)
at
org.jboss.cache.invocation.CacheInvocationDelegate.getRegion(CacheInvocationDelegate.java:
421)
at
org.mobicents.cache.MobicentsCache.setReplicationClassLoader(MobicentsCache.java:
133)
at
org.mobicents.cache.MobicentsCache.setReplicationClassLoader(MobicentsCache.java:
149)
at
org.mobicents.slee.container.SleeContainer.afterModulesInitialization(SleeContainer.java:
717)
at
org.mobicents.slee.container.SleeContainer.initSlee(SleeContainer.java:
574)
at
org.mobicents.slee.container.management.jmx.SleeManagementMBeanImpl.startSlee(SleeManagementMBeanImpl.java:
645)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:
59)
at
org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:
150)
at
org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:
66)
at org.jboss.kernel.plugins.dependency.KernelControllerContextAction
$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:
241)
at
org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:
47)
at
org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:
109)
at
org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:
70)
at
org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:
221)
at
org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:
54)
at
org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:
42)
at
org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:
62)
at
org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:
71)
at
org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:
51)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
774)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
540)
at
org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:
121)
at
org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:
51)
at
org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:
62)
at
org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:
50)
at
org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:
171)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:
1439)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:
1157)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:
1178)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:
1098)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
553)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:
781)
at
org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:
702)
at
org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:
117)
at
org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:
70)
at
org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:
53)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:
361)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
553)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(AbstractProfileService.java:
306)
at
org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:
271)
at
org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:
461)
at org.jboss.Main.boot(Main.java:221)
at org.jboss.Main$1.run(Main.java:556)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassNotFoundException: Failed to load class
org.jboss.cache.eviction.EvictionEvent
at
org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:
170)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:
265)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:
1119)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:
798)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:
441)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 67 more
12:47:27,778 ERROR [AbstractKernelController] Error installing to
PreInstall: name=Mobicents.JAINSLEE.LogManagementMBean state=Real
java.lang.NoClassDefFoundError: org/mobicents/slee/container/
management/jmx/log/NotificationHandler
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2404)
at java.lang.Class.getDeclaredConstructors(Class.java:1853)
at
org.jboss.reflect.plugins.introspection.IntrospectionTypeInfoFactoryImpl.getDeclaredConstructors(IntrospectionTypeInfoFactoryImpl.java:
489)
at
org.jboss.reflect.plugins.introspection.IntrospectionTypeInfoFactoryImpl.getConstructors(IntrospectionTypeInfoFactoryImpl.java:
146)
at
org.jboss.reflect.plugins.ClassInfoImpl.getDeclaredConstructors(ClassInfoImpl.java:
424)
at
org.jboss.beans.info.plugins.AbstractBeanInfoFactory.getConstructors(AbstractBeanInfoFactory.java:
206)
at
org.jboss.beans.info.plugins.AbstractBeanInfoFactory.getBeanInfo(AbstractBeanInfoFactory.java:
151)
at
org.jboss.config.plugins.AbstractConfiguration.getBeanInfo(AbstractConfiguration.java:
87)
at
org.jboss.kernel.plugins.config.AbstractKernelConfig.getBeanInfo(AbstractKernelConfig.java:
80)
at
org.jboss.kernel.plugins.config.AbstractKernelConfigurator.getBeanInfo(AbstractKernelConfigurator.java:
77)
at
org.jboss.kernel.plugins.config.AbstractKernelConfigurator.getBeanInfo(AbstractKernelConfigurator.java:
96)
at
org.jboss.kernel.plugins.dependency.PreInstallAction.installActionInternal(PreInstallAction.java:
90)
at
org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:
54)
at
org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:
42)
at
org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:
62)
at
org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:
71)
at
org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:
51)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
774)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
540)
at
org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:
121)
at
org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:
51)
at
org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:
62)
at
org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:
50)
at
org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:
171)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:
1439)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:
1157)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:
1178)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:
1098)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
553)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:
781)
at
org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:
702)
at
org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:
117)
at
org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:
70)
at
org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:
53)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:
361)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
553)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(AbstractProfileService.java:
306)
at
org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:
271)
at
org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:
461)
at org.jboss.Main.boot(Main.java:221)
at org.jboss.Main$1.run(Main.java:556)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassNotFoundException: Failed to load class
org.mobicents.slee.container.management.jmx.log.NotificationHandler
at
org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:
170)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:
265)
at
org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:
1119)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:
798)
at
org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:
441)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 60 more
12:47:27,787 ERROR [AbstractKernelController] Error installing to
Real: name=vfsfile:/opt/mobicents-jainslee-2.5.0/jboss-5.1.0.GA/server/
all/deploy/deploy.last/ state=PreReal mode=Manual requiredState=Real
org.jboss.deployers.spi.DeploymentException: Error deploying
deploy.last: java.io.IOException: Error listing files: /opt/mobicents-
jainslee-2.5.0/jboss-5.1.0.GA/server/all/deploy/deploy.last
at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:
196)
at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:99)
at
org.jboss.deployers.vfs.spi.deployer.AbstractVFSRealDeployer.internalDeploy(AbstractVFSRealDeployer.java:
45)
at
org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:
50)
at
org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:
171)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:
1439)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:
1157)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:
1098)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
553)
at
org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:
781)
at
org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:
702)
at
org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:
117)
at
org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:
70)
at
org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:
53)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:
361)
at
org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:
348)
at
org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:
1631)
at
org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:
934)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
1082)
at
org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:
984)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
822)
at
org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:
553)
at
org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(AbstractProfileService.java:
306)
at
org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:
271)
at
org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:
461)
at org.jboss.Main.boot(Main.java:221)
at org.jboss.Main$1.run(Main.java:556)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: java.io.IOException: Error
listing files: /opt/mobicents-jainslee-2.5.0/jboss-5.1.0.GA/server/all/
deploy/deploy.last
at
org.jboss.ejb3.deployers.JBoss5DeploymentUnit.getResources(JBoss5DeploymentUnit.java:
227)
at org.jboss.ejb3.Ejb3Deployment.deployUrl(Ejb3Deployment.java:639)
at org.jboss.ejb3.Ejb3Deployment.deploy(Ejb3Deployment.java:614)
at org.jboss.ejb3.Ejb3Deployment.create(Ejb3Deployment.java:491)
at org.jboss.ejb3.deployers.Ejb3Deployer.deploy(Ejb3Deployer.java:
182)
... 33 more
Caused by: java.io.IOException: Error listing files: /opt/mobicents-
jainslee-2.5.0/jboss-5.1.0.GA/server/all/deploy/deploy.last
at
org.jboss.virtual.plugins.context.file.FileHandler.getChildren(FileHandler.java:
225)
at
org.jboss.virtual.plugins.context.AbstractVFSContext.getChildren(AbstractVFSContext.java:
219)
at
org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:
336)
at
org.jboss.virtual.plugins.context.AbstractVFSContext.visit(AbstractVFSContext.java:
306)
at org.jboss.virtual.VFS.visit(VFS.java:421)
at org.jboss.virtual.VirtualFile.visit(VirtualFile.java:437)
at
org.jboss.ejb3.deployers.JBoss5DeploymentUnit.getResources(JBoss5DeploymentUnit.java:
223)
... 37 more
12:47:27,793 TRACE [SLEEParserDeployer] Undeploying: vfsfile:/opt/
mobicents-jainslee-2.5.0/jboss-5.1.0.GA/server/all/deploy/deploy.last/
12:47:27,793 TRACE [SLEEParserDeployer] Undeployed: vfsfile:/opt/
mobicents-jainslee-2.5.0/jboss-5.1.0.GA/server/all/deploy/deploy.last/
12:47:27,956 TRACE [SLEEParserDeployer] Deploying: vfsfile:/opt/
mobicents-jainslee-2.5.0/jboss-5.1.0.GA/server/all/farm/cluster-
examples-service.xml
12:47:27,956 TRACE [SLEEParserDeployer] SLEEParserDeployer 'parse'
called:
12:47:27,956 TRACE [SLEEParserDeployer]
du.................AbstractVFSDeploymentContext@662925821{vfsfile:/opt/
mobicents-jainslee-2.5.0/jboss-5.1.0.GA/server/all/farm/cluster-
examples-service.xml}
12:47:27,956 TRACE [SLEEParserDeployer] str1...............null
12:47:27,956 TRACE [SLEEParserDeployer] str2................jar
12:47:27,956 TRACE [SLEEParserDeployer] sdmd...............null
12:47:27,956 TRACE [SLEEParserDeployer] Deployed: vfsfile:/opt/
mobicents-jainslee-2.5.0/jboss-5.1.0.GA/server/all/farm/cluster-
examples-service.xml
12:47:27,965 ERROR [ProfileServiceBootstrap] Failed to load profile:
Summary of incomplete deployments (SEE PREVIOUS ERRORS FOR DETAILS):

DEPLOYMENTS MISSING DEPENDENCIES:
Deployment "Mobicents.JAINSLEE.PolicyManagementMBean" is missing the
following dependencies:
Dependency "<UNKNOWN
Mobicents.JAINSLEE.PolicyManagementMBean>" (should be in state
"Described", but is actually in state "** UNRESOLVED Demands
'Mobicents.JAINSLEE.SleeManagement' **")

DEPLOYMENTS IN ERROR:
Deployment "Mobicents.JAINSLEE.SleeManagement" is in error due to
the following reason(s): java.lang.ClassNotFoundException: Failed to
load class org.jboss.cache.eviction.EvictionEvent
Deployment "vfsfile:/opt/mobicents-jainslee-2.5.0/jboss-5.1.0.GA/
server/all/deploy/deploy.last/" is in error due to the following
reason(s): java.io.IOException: Error listing files: /opt/mobicents-
jainslee-2.5.0/jboss-5.1.0.GA/server/all/deploy/deploy.last
Deployment "<UNKNOWN Mobicents.JAINSLEE.PolicyManagementMBean>" is
in error due to the following reason(s): ** UNRESOLVED Demands
'Mobicents.JAINSLEE.SleeManagement' **
Deployment "Mobicents.JAINSLEE.LogManagementMBean" is in error due
to the following reason(s): java.lang.ClassNotFoundException: Failed
to load class
org.mobicents.slee.container.management.jmx.log.NotificationHandler

12:47:27,974 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on
http-172.16.133.123-8080
12:47:27,999 INFO [AjpProtocol] Starting Coyote AJP/1.3 on
ajp-172.16.133.123-8009
12:47:28,009 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA
(build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 1m:59s:
822ms

The weird thing here is that it only does this something like 3 start-
ups out of 5. Also, it's not possible to reproduce on all servers. We
have it now on two of our systems, but we also have two systems where
we can't seem to reproduce it (all running 2.5.0).

It seems to me that the stack trace is actually caused by a failed
rollback, and I can't seem to figure out what would be the actual
cause of the rollback. Does anyone have any kinds of ideas about where
to start looking, or what could be wrong?

Thanks,
-Calle

Carl-Magnus Björkell

unread,
Nov 18, 2011, 3:13:14 AM11/18/11
to mobicents-public
Hey again,

As it turns out, we did identify the core issue behind this problem
yesterday evening. The problem was that the number of allowed open
files were exceeded, and apparently the deployment code fails silently
in this case and just does a transaction rollback. We circumvented the
problem by increasing the allowed open files by setting "ulimit -n
8192" in the shell before starting the SLEE.

It's still a bit weird to me that the SLEE would exceed the default
1024 open file limit during deployment though... we saw the number of
open files jumping from ~200 to above 1000 when a new DU was
investigated. I guess this has something to do with the Sbb concrete
class generation?

I added an issue for this as well at http://code.google.com/p/mobicents/issues/detail?id=2967
. At least there should be more information given about why the
deployment fails.

-Calle

Alexandre Mendonça

unread,
Nov 20, 2011, 7:59:52 PM11/20/11
to mobicent...@googlegroups.com
Hi Calle,

I think it should be related to the DU contents being completely exploded (even lib jars) into a tmp folder, even if from 200 to 1000 it's still a big jump, but for XMPP RA for instance, I've checked it explodes into 424 file entries.

Thanks for the issue, we will investigate what may be going on.

Regards,

--
Alexandre Mendonça // JBoss R&D
http://ammendonca.blogspot.com/



2011/11/18 Carl-Magnus Björkell <nrgiz...@gmail.com>
Reply all
Reply to author
Forward
0 new messages