Shutting down a Jenkins instance that was still starting up

[Steve K]

Hello,

We have been experiencing periodic crashes of the Jenkins server (almost one per week).

It's bad enough that the system goes down, but bringing it back up has been problematic as well.

We are running Jenkins version 2.89.2 on Windows Server 2008 R2 Standard.

Here's a look at my jenkins.err.log:

Aug 15, 2018 9:20:20 AM Main deleteWinstoneTempContents
WARNING: Failed to delete the temporary Winstone file C:\Users\lx00057\AppData\Local\Temp\winstone\jenkins.war
Aug 15, 2018 9:20:20 AM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @669ms to org.eclipse.jetty.util.log.JavaUtilLog
Aug 15, 2018 9:20:20 AM winstone.Logger logInternal
INFO: Beginning extraction from war file
Aug 15, 2018 9:20:20 AM org.eclipse.jetty.server.handler.ContextHandler setContextPath
WARNING: Empty contextPath
Aug 15, 2018 9:20:20 AM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.4.z-SNAPSHOT
Aug 15, 2018 9:20:21 AM org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Aug 15, 2018 9:20:21 AM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: DefaultSessionIdManager workerName=node0
Aug 15, 2018 9:20:21 AM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: No SessionScavenger set, using defaults
Aug 15, 2018 9:20:21 AM org.eclipse.jetty.server.session.HouseKeeper startScavenging
INFO: Scavenging every 600000ms
Aug 15, 2018 9:20:21 AM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started w.@3cbe{/,file:///D:/Jenkins/war/,AVAILABLE}{D:\Jenkins\war}
Aug 15, 2018 9:20:21 AM org.eclipse.jetty.server.AbstractConnector doStop
INFO: Stopped ServerConnector@172e513{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
Aug 15, 2018 9:20:21 AM org.eclipse.jetty.server.session.HouseKeeper stopScavenging
INFO: Stopped scavenging
Aug 15, 2018 9:20:21 AM hudson.WebAppMain contextDestroyed
INFO: Shutting down a Jenkins instance that was still starting up
java.lang.Throwable: reason
    at hudson.WebAppMain.contextDestroyed(WebAppMain.java:388)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:898)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:545)
    at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:873)
    at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:355)
    at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1507)
    at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1471)
    at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:927)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:271)
    at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:569)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:142)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:160)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:124)
    at org.eclipse.jetty.server.Server.doStop(Server.java:523)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
    at winstone.Launcher.shutdown(Launcher.java:307)
    at winstone.Launcher.<init>(Launcher.java:167)
    at winstone.Launcher.main(Launcher.java:354)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at Main._main(Main.java:294)
    at Main.main(Main.java:132)

Aug 15, 2018 9:20:21 AM org.eclipse.jetty.server.handler.ContextHandler doStop
INFO: Stopped w.@3cbe{/,null,UNAVAILABLE}{D:\Jenkins\war}
Exception in thread "Jenkins initialization thread" java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:247)
Caused by: java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:560)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 1 more
Aug 15, 2018 9:20:21 AM winstone.Logger logInternal
INFO: Winstone shutdown successfully
java.io.IOException: Failed to start Jetty
    at winstone.Launcher.<init>(Launcher.java:156)
    at winstone.Launcher.main(Launcher.java:354)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at Main._main(Main.java:294)
    at Main.main(Main.java:132)
Caused by: java.net.BindException: Address already in use: bind
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Unknown Source)
    at sun.nio.ch.Net.bind(Unknown Source)
    at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
    at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:298)
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.server.Server.doStart(Server.java:431)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at winstone.Launcher.<init>(Launcher.java:154)
    ... 7 more
Aug 15, 2018 9:20:21 AM winstone.Logger logInternal
SEVERE: Container startup failed
java.io.IOException: Failed to start Jetty
    at winstone.Launcher.<init>(Launcher.java:156)
    at winstone.Launcher.main(Launcher.java:354)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at Main._main(Main.java:294)
    at Main.main(Main.java:132)
Caused by: java.net.BindException: Address already in use: bind
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Unknown Source)
    at sun.nio.ch.Net.bind(Unknown Source)
    at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
    at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
    at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:298)
    at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
    at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.server.Server.doStart(Server.java:431)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at winstone.Launcher.<init>(Launcher.java:154)
    ... 7 more

Interesting info from the jenkins.wrapper.log

2018-08-15 09:19:15,056 INFO  - Starting ServiceWrapper in the service mode
2018-08-15 09:19:15,572 INFO  - Starting D:\Jenkins\jre\bin\java -Xrs -Xmx1024m   -XX:+HeapDumpOnOutOfMemoryError -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "D:\Jenkins\jenkins.war" --httpPort=8080
2018-08-15 09:19:15,588 INFO  - Starting D:\Jenkins\jre\bin\java -Xrs -Xmx1024m   -XX:+HeapDumpOnOutOfMemoryError -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "D:\Jenkins\jenkins.war" --httpPort=8080
2018-08-15 09:19:15,603 INFO  - Started process 3716
2018-08-15 09:19:15,619 DEBUG - Forwarding logs of the process System.Diagnostics.Process (java) to winsw.SizeBasedRollingLogAppender

I don't understand why there are two "Starting D:\Jenkins\jre\bin\java..." commands.

My jenkins.xml
<!--
  Windows service definition for Jenkins

  To uninstall, run "jenkins.exe stop" to stop the service, then "jenkins.exe uninstall" to uninstall the service.
  Both commands don't produce any output if the execution is successful.
-->
<service>
  <id>jenkins</id>
  <name>Jenkins</name>
  <description>This service runs Jenkins continuous integration system.</description>
  <env name="JENKINS_HOME" value="%BASE%"/>
  <!--
    if you'd like to run Jenkins with a specific version of Java, specify a full path to java.exe.
    The following value assumes that you have java in your PATH.
  -->
  <executable>%BASE%\jre\bin\java</executable>
  <arguments>-Xrs -Xmx1024m   -XX:+HeapDumpOnOutOfMemoryError -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycl
e -jar "%BASE%\jenkins.war" --httpPort=8080</arguments>
  <!--
    interactive flag causes the empty black Java window to be displayed.
    I'm still debugging this.
  <interactive />
  -->
  <logmode>rotate</logmode>
</service>

I'd appreciate any helpful information you may be able to provide.  If there is additional information that I could provide to help determine a cause/fix for the problems I'm experiencing, please let me know.

Thanks in advance.

Steve K.

[Burrows, John]

Steve,

Shut down Jenkins, then using Task Manager, check for any rogue Jenkins or Java processes running on that machine and if any found, kill those processes.

Start Jenkins back up.

You should only have one Jenkins and one Java process running on the Jenkins Master, unless you are also running jobs on that server.

See if that helps.

John

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/8f81da76-deb6-47a3-a781-a33ba86ee7d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

---

This email message and any attachments may contain confidential, proprietary or non-public information. The information is intended solely for the designated recipient(s). If an addressing or transmission error has misdirected this email, please notify the sender immediately and destroy this email. Any review, dissemination, use or reliance upon this information by unintended recipients is prohibited. Any opinions expressed in this email are those of the author personally.

[Steve K]

Thanks for your ideas John.

I have attempted what you suggested multiple times, but it does not go smoothly. 

• I shut down Jenkins if necessary (often, it is already down).
• Via task manager, I see that Java is running.
• I attempt to kill the Java process, but, even though I am an administrator, I get "Access Denied".
• I attempt to start Jenkins, but that fails (I believe that's when the subject of this thread is generated "Shutting down a Jenkins instance that was still starting up".
  
• I resort to -- Disabling the Jenkins service.
• Reboot the machine (seems to be the only way to get rid of the Java process).
• After the machine completes the reboot, I enable the service and start Jenkins.
• Again, I'll see two two "Starting D:\Jenkins\jre\bin\java..." commands in the jenkins.wrapper.log  I don't know where the second command comes from.
  

[Burrows, John]

A couple more things to check:

 

1)      Check where the Jenkins Service is pointing to, should be a Jenkins.exe file located in folder where Jenkins is installed. Listed under Path to executable in Service properties:

For example, this is my entry - D:\SCM\Apps\Jenkins\jenkins.exe

 

In same folder as listed in Service, check your Jenkins.xml file to ensure it only has one <arguments> key line in  the xml file, also make sure that you have your <executable> key in the xml file and it points to a valid Java 1.8 version of Java. Include the path in this key, if Java is not in the system path, then you need to point directly at it.

 

<service>

<id>jenkins</id>

<name>Jenkins</name>

<description>This service runs Jenkins continuous integration system.</description>

<env name="JENKINS_HOME" value="%BASE%"/>

 

<executable>D:\SCM\Apps\Java\Ora_jdk1.8.0_144x64\bin\java</executable>

 

                              <arguments>-Xrs -Xms2048m -Xmx8192m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=8080</arguments>

 

<logmode>rotate</logmode>

<onfailure action="restart" />

</service>

 

2)      Make sure there are no other applications using port 8080 (80), if you have something like Apache Tomcat, or IIS, it could be using the same port and you could have a conflict. You can always change your httpPort in your Jenkins.xml file to 8082 or 8090, etc.

 

Hope this helps.

 

Thanks,

John

This email message and any attachments may contain confidential, proprietary or non-public information. The information is intended solely for the designated recipient(s). If an addressing or transmission error has misdirected this email, please notify the sender immediately and destroy this email. Any review, dissemination, use or reliance upon this information by unintended recipients is prohibited. Any opinions expressed in this email are those of the author personally.

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/814ff085-07d6-434e-a4e8-4d3d600dc35a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Steve K]

Thanks again John,

re 1) my Path to exe in Service props is: D:\Jenkins\jenkins.exe    [confirmed as valid]

         I included the non-comment content of my Jenkins.xml file in my initial post. There is only one arguments line.
         I believe that the path to Java (<executable>%BASE%\jre\bin\java</executable>) is the "out-of-the-box" value that points to the java included in the install [confirmed as v1.8.0_20]

re 2) I don't see a conflict on 8080

D:\Jenkins>netstat -aon | findstr 8080 | findstr LISTENING
  TCP    0.0.0.0:8080           0.0.0.0:0              LISTENING       236
  TCP    [::]:8080              [::]:0                 LISTENING       236

D:\Jenkins>tasklist | findstr 236
java.exe                       236 Services                   0  1,260,892 K

I do appreciate the help.  Any other ideas?

On Thursday, August 16, 2018 at 11:21:04 AM UTC-4, Burrows, John wrote:

A couple more things to check:

...