Jenkins not correctly executing Maven task starting forked jetty.

[Thomas Peeters]

Jenkins version: 1.461


We are attempting to start a Jetty server in forked mode at the end of a build pipeline.  It all works, except for the forking part, that doesn't seem to work eventhough a non-forked attempt does work.  Even executing the exact same Maven command on the same system as of which Jenkins is installed it works, both in forked as non-forked variation.

I had no clue as to what is going on, no logging, no errors, no nothing actually.  So starting the Jenkins job with switches -e -X gave me the following command that gets ran:
(bear with me)

[DEBUG] [/usr/lib/jvm/jdk1.6.0_29/jre/bin/java, -Denv=thomas, -cp, /home/jenkins/.m2/repository/org/eclipse/jetty/jetty-util/8.1.5.v20120716/jetty-util-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-webapp/8.1.5.v20120716/jetty-webapp-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-xml/8.1.5.v20120716/jetty-xml-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-servlet/8.1.5.v20120716/jetty-servlet-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-security/8.1.5.v20120716/jetty-security-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-server/8.1.5.v20120716/jetty-server-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-continuation/8.1.5.v20120716/jetty-continuation-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/sonatype/sisu/sisu-inject-bean/2.1.1/sisu-inject-bean-2.1.1.jar:/home/jenkins/.m2/repository/org/sonatype/sisu/sisu-guice/2.9.4/sisu-guice-2.9.4-no_aop.jar:/home/jenkins/.m2/repository/org/codehaus/plexus/plexus-utils/2.0.6/plexus-utils-2.0.6.jar:/home/jenkins/.m2/repository/org/sonatype/aether/aether-util/1.11/aether-util-1.11.jar:/home/jenkins/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.14/plexus-interpolation-1.14.jar:/home/jenkins/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar:/home/jenkins/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar:/home/jenkins/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar:/home/jenkins/.m2/repository/org/apache/maven/plugin-tools/maven-plugin-tools-api/2.9/maven-plugin-tools-api-2.9.jar:/home/jenkins/.m2/repository/org/apache/maven/reporting/maven-reporting-api/2.0.6/maven-reporting-api-2.0.6.jar:/home/jenkins/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.0-alpha-7/doxia-sink-api-1.0-alpha-7.jar:/home/jenkins/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/home/jenkins/.m2/repository/net/sf/jtidy/jtidy/r938/jtidy-r938.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-plus/8.1.5.v20120716/jetty-plus-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.transaction/1.1.1.v201105210645/javax.transaction-1.1.1.v201105210645.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-jndi/8.1.5.v20120716/jetty-jndi-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.mail.glassfish/1.4.1.v201005082020/javax.mail.glassfish-1.4.1.v201005082020.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.activation/1.1.0.v201105071233/javax.activation-1.1.0.v201105071233.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-jmx/8.1.5.v20120716/jetty-jmx-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-annotations/8.1.5.v20120716/jetty-annotations-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.annotation/1.1.0.v201108011116/javax.annotation-1.1.0.v201108011116.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/org.objectweb.asm/3.1.0.v200803061910/org.objectweb.asm-3.1.0.v200803061910.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-websocket/8.1.5.v20120716/jetty-websocket-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-io/8.1.5.v20120716/jetty-io-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-http/8.1.5.v20120716/jetty-http-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-jsp/8.1.5.v20120716/jetty-jsp-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.servlet.jsp/2.2.0.v201112011158/javax.servlet.jsp-2.2.0.v201112011158.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/3.0.0.v201112011016/javax.servlet-3.0.0.v201112011016.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/org.apache.jasper.glassfish/2.2.2.v201112011158/org.apache.jasper.glassfish-2.2.2.v201112011158.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.servlet.jsp.jstl/1.2.0.v201105211821/javax.servlet.jsp.jstl-1.2.0.v201105211821.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/org.apache.taglibs.standard.glassfish/1.2.0.v201112081803/org.apache.taglibs.standard.glassfish-1.2.0.v201112081803.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.el/2.2.0.v201108011116/javax.el-2.2.0.v201108011116.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/com.sun.el/2.2.0.v201108011116/com.sun.el-2.2.0.v201108011116.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/org.eclipse.jdt.core/3.7.1/org.eclipse.jdt.core-3.7.1.jar:/home/jenkins/.m2/repository/org/mortbay/jetty/jetty-maven-plugin/8.1.5.v20120716/jetty-maven-plugin-8.1.5.v20120716.jar, org.mortbay.jetty.plugin.Starter, --stop-port, 9990, --stop-key, stopherding, --jetty-xml, src/main/resources/env/thomas/jetty.xml, --props, /home/jenkins/.jenkins/jobs/start-test-bulkuploader/workspace/target/fork.props]
[INFO] Forked process started

Now: notice those "," characters after every space.  Running that command directly causes a "path not found" kind of error.  And...  Removing all of the "," characters lets the command run correctly.  Something might be up with that, but after 8 hours of looking I'm not even sure anymore what my name is.  Anyway.  This is the command that works:
But I do want this to work in Jenkins, and in forked mode.

/usr/lib/jvm/jdk1.6.0_29/jre/bin/java -Denv=thomas -cp /home/jenkins/.m2/repository/org/eclipse/jetty/jetty-util/8.1.5.v20120716/jetty-util-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-webapp/8.1.5.v20120716/jetty-webapp-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-xml/8.1.5.v20120716/jetty-xml-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-servlet/8.1.5.v20120716/jetty-servlet-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-security/8.1.5.v20120716/jetty-security-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-server/8.1.5.v20120716/jetty-server-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-continuation/8.1.5.v20120716/jetty-continuation-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/sonatype/sisu/sisu-inject-bean/2.1.1/sisu-inject-bean-2.1.1.jar:/home/jenkins/.m2/repository/org/sonatype/sisu/sisu-guice/2.9.4/sisu-guice-2.9.4-no_aop.jar:/home/jenkins/.m2/repository/org/codehaus/plexus/plexus-utils/2.0.6/plexus-utils-2.0.6.jar:/home/jenkins/.m2/repository/org/sonatype/aether/aether-util/1.11/aether-util-1.11.jar:/home/jenkins/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.14/plexus-interpolation-1.14.jar:/home/jenkins/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar:/home/jenkins/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar:/home/jenkins/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar:/home/jenkins/.m2/repository/org/apache/maven/plugin-tools/maven-plugin-tools-api/2.9/maven-plugin-tools-api-2.9.jar:/home/jenkins/.m2/repository/org/apache/maven/reporting/maven-reporting-api/2.0.6/maven-reporting-api-2.0.6.jar:/home/jenkins/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.0-alpha-7/doxia-sink-api-1.0-alpha-7.jar:/home/jenkins/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/home/jenkins/.m2/repository/net/sf/jtidy/jtidy/r938/jtidy-r938.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-plus/8.1.5.v20120716/jetty-plus-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.transaction/1.1.1.v201105210645/javax.transaction-1.1.1.v201105210645.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-jndi/8.1.5.v20120716/jetty-jndi-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.mail.glassfish/1.4.1.v201005082020/javax.mail.glassfish-1.4.1.v201005082020.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.activation/1.1.0.v201105071233/javax.activation-1.1.0.v201105071233.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-jmx/8.1.5.v20120716/jetty-jmx-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-annotations/8.1.5.v20120716/jetty-annotations-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.annotation/1.1.0.v201108011116/javax.annotation-1.1.0.v201108011116.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/org.objectweb.asm/3.1.0.v200803061910/org.objectweb.asm-3.1.0.v200803061910.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-websocket/8.1.5.v20120716/jetty-websocket-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-io/8.1.5.v20120716/jetty-io-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-http/8.1.5.v20120716/jetty-http-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/jetty-jsp/8.1.5.v20120716/jetty-jsp-8.1.5.v20120716.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.servlet.jsp/2.2.0.v201112011158/javax.servlet.jsp-2.2.0.v201112011158.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/3.0.0.v201112011016/javax.servlet-3.0.0.v201112011016.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/org.apache.jasper.glassfish/2.2.2.v201112011158/org.apache.jasper.glassfish-2.2.2.v201112011158.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.servlet.jsp.jstl/1.2.0.v201105211821/javax.servlet.jsp.jstl-1.2.0.v201105211821.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/org.apache.taglibs.standard.glassfish/1.2.0.v201112081803/org.apache.taglibs.standard.glassfish-1.2.0.v201112081803.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/javax.el/2.2.0.v201108011116/javax.el-2.2.0.v201108011116.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/com.sun.el/2.2.0.v201108011116/com.sun.el-2.2.0.v201108011116.jar:/home/jenkins/.m2/repository/org/eclipse/jetty/orbit/org.eclipse.jdt.core/3.7.1/org.eclipse.jdt.core-3.7.1.jar:/home/jenkins/.m2/repository/org/mortbay/jetty/jetty-maven-plugin/8.1.5.v20120716/jetty-maven-plugin-8.1.5.v20120716.jar org.mortbay.jetty.plugin.Starter --stop-port 9990 --stop-key stopherding --jetty-xml src/main/resources/env/thomas/jetty.xml --props /home/jenkins/.jenkins/jobs/start-test-bulkuploader/workspace/target/fork.props

[Thomas Peeters]

No suggestions at all? 

[Josselin Pierre]

Hi,

Here's my 2 cents, for what it's worth :
Are you sure the forking itself fails ?
One issue that often appears with spawning process from Jenkins is that the child process might be killed once the job is finished.

You might find quite a few references on this issue on google
https://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build

As far as I remember, one of the work around was to set a specific environment variable BUILD_ID before spawning the process, such as BUILD_ID=dontKillMe.

Hope that helped

[Thomas Peeters]

Well..  The process is already in deamon mode, and a switch called "waitForChild" in the Jetty configuration is set to false, so it is an independent process.  Without those two the Maven job would already be either held by the jetty server or would start and stop as soon as the job is finished.

I would expect Jenkins to behave the same as its Maven job does, because it is a Maven implementation that is used by Jenkins.  But fact is that without daemon mode on and waitForChild=true the Jenkins job keeps running, but works.  The Jetty server is reachable.  However it never evaluates the job as succes/fail and it also takes up a run spot.

Thanks for the info though.  I am looking at your suggested path.  Something might be off with the daemon part.

[Thomas Peeters]

I fixed the Jenkins issue by trying a workaround.  Instead of asking jenkins to run my Maven job I used a plugin that runs scripts over ssh.  This script does that Maven command that doesn't work in Jenkins.  This works without a hitch.  I still think this is a Jenkins bug, or that's what it looks like to me.  I mean, a Maven job that doesn't work when Jenkins runs it, but when Jenkins invokes the script it works. 

[yogesh khandelwal]

I am facing the same problem with jeninks and jetty org.mortbay maven plugin. while starting jetty using run-forked, it's getting hung in jenkins. Any solution will be appricaited. I tried the scripts workaround but it didnot work for me.