[JIRA] [ec2-plugin] (JENKINS-35465) Instance launch is not terminated when launch hangs

[bartlomiej.pelc@nokia.com (JIRA)]

Bartlomiej Pelc created an issue

Jenkins / JENKINS-35465 Instance launch is not terminated when launch hangs Issue Type: Bug Assignee: Francis Upton Components: ec2-plugin Created: 2016/Jun/08 11:39 AM Environment: awt.toolkit sun.awt.X11.XToolkit com.sun.akuma.Daemon daemonized com.sun.org.apache.xml.internal.dtm.DTMManager com.sun.org.apache.xml.internal.dtm.ref.DTMManagerDefault executable-war /usr/lib/jenkins/jenkins.war file.encoding UTF-8 file.encoding.pkg sun.io file.separator / hudson.diyChunking false java.awt.graphicsenv sun.awt.X11GraphicsEnvironment java.awt.headless true java.awt.printerjob sun.print.PSPrinterJob java.class.path /usr/lib/jenkins/jenkins.war java.class.version 52.0 java.endorsed.dirs /usr/java/jdk1.8.0_92/jre/lib/endorsed java.ext.dirs /usr/java/jdk1.8.0_92/jre/lib/ext:/usr/java/packages/lib/ext java.home /usr/java/jdk1.8.0_92/jre java.io.tmpdir /tmp java.library.path /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.runtime.name Java(TM) SE Runtime Environment java.runtime.version 1.8.0_92-b14 java.specification.name Java Platform API Specification java.specification.vendor Oracle Corporation java.specification.version 1.8 java.vendor Oracle Corporation java.vendor.url http://java.oracle.com/ java.vendor.url.bug http://bugreport.sun.com/bugreport/ java.version 1.8.0_92 java.vm.info mixed mode java.vm.name Java HotSpot(TM) 64-Bit Server VM java.vm.specification.name Java Virtual Machine Specification java.vm.specification.vendor Oracle Corporation java.vm.specification.version 1.8 java.vm.vendor Oracle Corporation java.vm.version 25.92-b14 javamelody.About Monitoring https://wiki.jenkins-ci.org/display/JENKINS/Monitoring javamelody.analytics-id UA-1335263-7 javamelody.custom-reports Jenkins Info,About Monitoring javamelody.gzip-compression-disabled true javamelody.http-transform-pattern /\d+/|/site/.+|avadoc/.+|/ws/.+|obertura/.+|estReport/.+|iolations/file/.+|/user/.+|/static/\w+/|/adjuncts/\w+/|/bound/[\w\-]+ javamelody.Jenkins Info /systemInfo javamelody.no-database true javamelody.storage-directory //jenkins/monitoring javamelody.system-actions-enabled true javax.xml.parsers.DocumentBuilderFactory com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl JENKINS_HOME /jenkins jna.loaded true jna.platform.library.path /usr/lib64:/lib64:/usr/lib:/lib:/usr/lib64/dyninst:/usr/lib64/mysql:/usr/lib64/iscsi jnidispatch.path /tmp/jna--1712433994/jna7944967999299947335.tmp line.separator mail.smtp.sendpartial true mail.smtps.sendpartial true os.arch amd64 os.name Linux os.version 3.10.0-327.el7.x86_64 path.separator : pid 20516 sun.arch.data.model 64 sun.boot.class.path /usr/java/jdk1.8.0_92/jre/lib/resources.jar:/usr/java/jdk1.8.0_92/jre/lib/rt.jar:/usr/java/jdk1.8.0_92/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_92/jre/lib/jsse.jar:/usr/java/jdk1.8.0_92/jre/lib/jce.jar:/usr/java/jdk1.8.0_92/jre/lib/charsets.jar:/usr/java/jdk1.8.0_92/jre/lib/jfr.jar:/usr/java/jdk1.8.0_92/jre/classes sun.boot.library.path /usr/java/jdk1.8.0_92/jre/lib/amd64 sun.cpu.endian little sun.cpu.isalist sun.font.fontmanager sun.awt.X11FontManager sun.io.unicode.encoding UnicodeLittle sun.java.command /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=-1 --httpsPort=8443 --httpsKeyStore=/jenkins/keystore.jks --httpsKeyStorePassword=jenkins --httpsListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20 sun.java.launcher SUN_STANDARD sun.jnu.encoding UTF-8 sun.management.compiler HotSpot 64-Bit Tiered Compilers sun.os.patch.level unknown svnkit.http.methods Digest,Basic,NTLM,Negotiate svnkit.ssh2.persistent false user.country US user.dir / user.home /jenkins user.language en user.name jenkins user.timezone Europe/Helsinki Priority: Major Reporter: Bartlomiej Pelc I have Jenkins 2.7 with ec2-plugin 1.33 configured with Eucalyptus cloud. When launching nodes in the cloud, sometimes it so happens that Eucalyptus is taking way too long to launch the instance (more than an hour). This is because the cloud is near max capacity so it is expected that some instances might take a long time to launch. I would expect that the ec2-plugin node startup procedure terminates after certain time when node fails to launch (as indicated by https://issues.jenkins-ci.org/browse/JENKINS-23790 where 30 minutes timeout is mentioned) However, the launch process is not terminated and it blocks launching new nodes that might start up successfully, for example in a different availability zone. Here is a snippet from instance launch log file: {{FINEST: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is still pending/launching, waiting 5s Jun 07, 2016 7:06:45 PM hudson.plugins.ec2.EC2Cloud log FINEST: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is still pending/launching, waiting 5s ... FINEST: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is still pending/launching, waiting 5s Jun 07, 2016 9:03:23 PM hudson.plugins.ec2.EC2Cloud log FINEST: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is still pending/launching, waiting 5s Jun 07, 2016 9:03:28 PM hudson.plugins.ec2.EC2Cloud log INFO: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is terminated or terminating, aborting launch}} As you can see above, the instance was launching for almost two hours, then Eucalyptus timeout triggered and ec2-plugin gave up with launching this instance. Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[bartlomiej.pelc@nokia.com (JIRA)]

Bartlomiej Pelc updated an issue

Jenkins / JENKINS-35465 Instance launch is not terminated when launch hangs

Change By: Bartlomiej Pelc

I have Jenkins 2.7 with ec2-plugin 1.33 configured with Eucalyptus cloud. When launching nodes in the cloud, sometimes it so happens that Eucalyptus is taking way too long to launch the instance (more than an hour). This is because the cloud is near max capacity so it is expected that some instances might take a long time to launch. I would expect that the ec2-plugin node startup procedure terminates after certain time when node fails to launch (as indicated by https://issues.jenkins-ci.org/browse/JENKINS-23790 where 30 minutes timeout is mentioned) However, the launch process is not terminated and it blocks launching new nodes that might start up successfully, for example in a different availability zone. Here is a snippet from instance launch log file: {{ FINEST: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is still pending/launching, waiting 5s Jun 07, 2016 7:06:45 PM hudson.plugins.ec2.EC2Cloud log FINEST: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is still pending/launching, waiting 5s ... FINEST: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is still pending/launching, waiting 5s Jun 07, 2016 9:03:23 PM hudson.plugins.ec2.EC2Cloud log FINEST: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is still pending/launching, waiting 5s Jun 07, 2016 9:03:28 PM hudson.plugins.ec2.EC2Cloud log INFO: Node CentOS 7.2 2016-04-19 (i-5f209379)(i-5f209379) is terminated or terminating, aborting launch }} As you can see above, the instance was launching for almost two hours, then Eucalyptus timeout triggered and ec2-plugin gave up with launching this instance. Add Comment

[fabrizio.manfredi@gmail.com (JIRA)]

FABRIZIO MANFREDI closed an issue as Cannot Reproduce

Jenkins / JENKINS-35465 Instance launch is not terminated when launch hangs

Change By: FABRIZIO MANFREDI Status: Open Closed Resolution: Cannot Reproduce

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)