[JIRA] (JENKINS-38481) Error while launching new ec2 slave

[vtians.ad@gmail.com (JIRA)]

Tim Feng created an issue

Jenkins / JENKINS-38481 Error while launching new ec2 slave Issue Type: Bug Assignee: Francis Upton Components: ec2-plugin Created: 2016/Sep/24 9:32 AM Environment: Jenkins 2.7.1 ec2-plugin v1.35 Priority: Minor Reporter: Tim Feng Got the following error while trying to launch new EC2 build slave. I noticed that a new EC2 instance has been launched successfully, but the error still pop up. Stack trace javax.servlet.ServletException: java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V at hudson.plugins.ec2.SlaveTemplate.updateRemoteTags(SlaveTemplate.java:882) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:585) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:377) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:515) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:340) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) ... 58 more Add Comment

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)

[sebastian.hillig@artory.com (JIRA)]

Sebastian Hillig commented on JENKINS-38481

Re: Error while launching new ec2 slave Bug is caused by upgrading Amazon Web Services SDK to 1.11.37

Add Comment

[james@jameshuston.net (JIRA)]

James Huston commented on JENKINS-38481

Re: Error while launching new ec2 slave

I can confirm that downgrading the Amazon Web Services SDK plugin to 1.10.50 resolves this issue for me.

Add Comment

[mika.jo.karjalainen@gmail.com (JIRA)]

Mika Karjalainen updated an issue

Jenkins / JENKINS-38481

Error while launching new ec2 slave

Change By: Mika Karjalainen Component/s: aws-java-sdk-plugin

Add Comment

[stephan@angrybytes.com (JIRA)]

Stéphan Kochen commented on JENKINS-38481

Re: Error while launching new ec2 slave As a side note, this has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances. Doesn't look like there's a way to limit instances per IAM user on the AWS side either. Perhaps, as a defensive measure against future bugs, the code could also try to terminate the instance if provisioning fails?

Add Comment

[stephan@angrybytes.com (JIRA)]

Stéphan Kochen commented on JENKINS-38481

Re: Error while launching new ec2 slave

Temporary workaround: remove tags from your slave configuration. (If your Jenkins is offline, remove the <tags> element from config.xml.) This way, tags are not included in the filter when Jenkins looks for existing instances, and will match instances for which tagging failed. (The filter is otherwise sufficiently narrow to not match unrelated instances in most cases. It includes, e.g., instance type, AMI and SSH key name, security group, etc.)

Add Comment

[christopher.ernenwein@here.com (JIRA)]

Chris Ernenwein commented on JENKINS-38481

Re: Error while launching new ec2 slave

can we increase the priority on this? we also "has bitten us quite a bit, because Jenkins proceeded to fill our AWS account up to the limit with EC2 instances"

Add Comment

[pjbecotte@gmail.com (JIRA)]

Paul Becotte commented on JENKINS-38481

Re: Error while launching new ec2 slave

Same here. I also cannot figure out how to proceed. downgrading this plugin to 1.34 and the sdk to 1.10.50 results in slaves being terminated immediately after launch...with them both at the latest version, I get the above error message (even with tags removed from all my AMI configs). Any idea on how to get back up and running?

Add Comment

[pjbecotte@gmail.com (JIRA)]

Paul Becotte edited a comment on JENKINS-38481

Re: Error while launching new ec2 slave

Same here.  I also cannot figure out how to proceed.  downgrading this plugin to 1.34 and the sdk to 1.10.50 results in slaves being terminated immediately after launch...with them both at the latest version, I get the above error message (even with tags removed from all my AMI configs). Any idea on how to get back up and running?

EDIT:  The second time I tried downgrading them both, the system worked properly.

Add Comment

[michal.cichra@gmail.com (JIRA)]

Michal Cichra commented on JENKINS-38481

Re: Error while launching new ec2 slave

We just paid 250$ in AWS because the plugin created 20 pretty big machines. I'm stunned that no one was able to issue some hotfix for days now. People can be paying a lot and not know. I can confirm that downgrading the Amazon Web Services SDK plugin fixes the issue.

Add Comment

[michal.cichra@gmail.com (JIRA)]

Michal Cichra edited a comment on JENKINS-38481

Re: Error while launching new ec2 slave

We just paid 250$ in AWS because the plugin created 20 pretty big machines.

I'm stunned that no one was able to issue some hotfix for days now. People can be paying a lot and not know. Or why it has priority `Minor`.

I can confirm that downgrading the Amazon Web Services SDK plugin fixes the issue.

Add Comment

[jared.biel@bolderthinking.com (JIRA)]

Jared Biel commented on JENKINS-38481

Re: Error while launching new ec2 slave

Another confirmation that downgrading EC2 to 1.34 and the AWS SDK to 1.10.50 works for now. I too am surprised that this is a minor. Luckily due to the instance size that we use and account restrictions this bug only cost us ~$4 + time.

Add Comment

[hamza.chouaibi@gmail.com (JIRA)]

Hamza Chouaibi updated an issue

Jenkins / JENKINS-38481

Error while launching new ec2 slave

Change By: Hamza Chouaibi Priority: Minor Critical

Add Comment

[nicolas.deloof@gmail.com (JIRA)]

Nicolas De Loof commented on JENKINS-38481

Re: Error while launching new ec2 slave Method is unchanged in SDK 1.11.37, so I don't understand we get such an error https://github.com/aws/aws-sdk-java/blob/1.11.37/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2.java#L1199

Add Comment

[nicolas.deloof@gmail.com (JIRA)]

Nicolas De Loof edited a comment on JENKINS-38481

Re: Error while launching new ec2 slave

Method is seems unchanged in SDK 1.11.37, so I don't understand we get such an error but actually a return type was added https://github.com/aws/aws-sdk-java/blob/1.11.37/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2.java#L1199 which introduce binary compatibility issue. Proposed https://github.com/jenkinsci/ec2-plugin/pull/221

Add Comment

[pgmjsd@gmail.com (JIRA)]

Joshua Davis commented on JENKINS-38481

Re: Error while launching new ec2 slave

This bug is bad news... I just had to terminate over 40 m4.large instances.

Add Comment

[peter@reidweb.com (JIRA)]

Peter Reid commented on JENKINS-38481

Re: Error while launching new ec2 slave

I am also experiencing this issue

Add Comment

[chriskulinski@yahoo.com (JIRA)]

Chris Kulinski commented on JENKINS-38481

Re: Error while launching new ec2 slave

We also had this issue. Spun up slaves until we hit our EC2 instance cap. Not fun. Tested the PR fix linked above and it solved the problem. Release this fix soon!

Add Comment

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-38481

Re: Error while launching new ec2 slave

Code changed in jenkins User: Nicolas De Loof Path: pom.xml src/test/java/hudson/plugins/ec2/SlaveTemplateUnitTest.java http://jenkins-ci.org/commit/ec2-plugin/c8983e4a8c3b784ae5a9a58127c88f3013db490d Log: JENKINS-38481 1.11.37 changed com.amazonaws.services.ec2.AmazonEC2#createTags return type, so incompatible binary change

Add Comment