ec2 plugin bug

89 views

Skip to first unread message

Stas Rudenko

unread,

Nov 17, 2015, 2:06:51 PM11/17/15

to rundeck-discuss

Hi,

I think I'm facing with ec2 plugin bug. Sometimes my ec2 nodelist stops to update.

-rw-rw-r-- 1 rundeck rundeck 53970 Nov 16 15:07 2.aws-ec2.xml

root@ml-batch01:/rundeck/var/resourceModelSourceCache/Machine-Learning# date

Tue Nov 17 13:53:27 GMT+5 2015

I was trying to pull nodelist by visiting the node page or running any job - It doesn't resolve anything. Sometimes it's just sticking like that. Rundeck restart helps tho.

Log:

WARN LoggingResourceModelSourceCache: [ResourceModelSource: 2.aws-ec2 (AWS EC2 Resources), project: Machine-Learning] Returning cached model data

ERROR ExceptionCatchingResourceModelSource: [ResourceModelSource: 2.aws-ec2 (AWS EC2 Resources), project: Machine-Learning]

java.lang.NoClassDefFoundError: com/amazonaws/services/ec2/AmazonEC2AsyncClient

at com.dtolabs.rundeck.plugin.resources.ec2.InstanceToNodeMapper.performQueryAsync(InstanceToNodeMapper.java:105)

at com.dtolabs.rundeck.plugin.resources.ec2.EC2ResourceModelSource.getNodes(EC2ResourceModelSource.java:208)

at com.dtolabs.rundeck.core.resources.ExceptionCatchingResourceModelSource.getNodes(ExceptionCatchingResourceModelSource.java:45)

at com.dtolabs.rundeck.core.common.ProjectNodeSupport.getNodeSet(ProjectNodeSupport.java:88)

at com.dtolabs.rundeck.core.common.FrameworkProject.getNodeSet(FrameworkProject.java:380)

at com.dtolabs.rundeck.core.common.IRundeckProject$getNodeSet$2.call(Unknown Source)

at rundeck.services.FrameworkService$_filterNodeSet_closure4.doCall(FrameworkService.groovy:283)

at rundeck.services.FrameworkService$_filterNodeSet_closure4.doCall(FrameworkService.groovy)

at sun.reflect.GeneratedMethodAccessor851.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)

at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)

at groovy.lang.Closure.call(Closure.java:423)

at groovy.lang.Closure.call(Closure.java:417)

at com.codahale.metrics.Timer.time(Timer.java:99)

at com.codahale.metrics.Timer$time$0.call(Unknown Source)

at MetricswebGrailsPlugin$_addDynamicMetricMethods_closure26.doCall(MetricswebGrailsPlugin.groovy:174)

at sun.reflect.GeneratedMethodAccessor578.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)

at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)

at groovy.lang.Closure.call(Closure.java:423)

at org.codehaus.groovy.runtime.metaclass.ClosureStaticMetaMethod.invoke(ClosureStaticMetaMethod.java:59)

at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:248)

at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:68)

at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)

at rundeck.services.FrameworkService.filterNodeSet(FrameworkService.groovy:282)

at rundeck.services.FrameworkService$filterNodeSet$24.call(Unknown Source)

at rundeck.services.ExecutionService.createContext(ExecutionService.groovy:1061)

at rundeck.services.ExecutionService$createContext$26.callCurrent(Unknown Source)

at rundeck.services.ExecutionService.executeAsyncBegin(ExecutionService.groovy:794)

at rundeck.services.ExecutionService.executeAsyncBegin(ExecutionService.groovy)

at rundeck.services.ExecutionService$$FastClassBySpringCGLIB$$ffb73873.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)

at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)

at rundeck.services.ExecutionService$$EnhancerBySpringCGLIB$$e0364ca7.executeAsyncBegin(<generated>)

at rundeck.services.ExecutionService$executeAsyncBegin$15.call(Unknown Source)

at rundeck.quartzjobs.ExecutionJob.executeCommand(ExecutionJob.groovy:290)

at rundeck.quartzjobs.ExecutionJob$executeCommand$6.callCurrent(Unknown Source)

at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:125)

at sun.reflect.GeneratedMethodAccessor1281.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)

at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1016)

at groovy.lang.Closure.call(Closure.java:423)

at groovy.lang.Closure.call(Closure.java:417)

at com.codahale.metrics.Timer.time(Timer.java:99)

at com.codahale.metrics.Timer$time$0.call(Unknown Source)

at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:73)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

Is there any workaround for this? It's really hard/inconvenient to restart production box every time when it's happening :(

Stas Rudenko

unread,

Nov 18, 2015, 2:16:15 PM11/18/15

to rundeck-discuss

Hey Greg, I filled a ticket for this issue https://github.com/rundeck-plugins/rundeck-ec2-nodes-plugin/issues/34 .

When you will be able please check this ticket. It's kind of critical as rundeck doesn't refresh list nodes.