[JIRA] [docker-custom-build-environment-plugin] (JENKINS-32542) Maven project failed with "Connection refused" when built in a docker Container

30 views
Skip to first unread message

syl.carrier@laposte.net (JIRA)

unread,
Jan 20, 2016, 2:23:02 PM1/20/16
to jenkinsc...@googlegroups.com
Sylvie Carrier created an issue
 
Jenkins / Bug JENKINS-32542
Maven project failed with "Connection refused" when built in a docker Container
Issue Type: Bug Bug
Assignee: Unassigned
Components: docker-custom-build-environment-plugin
Created: 20/Jan/16 7:22 PM
Environment: Jenkins ver. 1.625.3.1
Plugin Version: 1.6.4
Priority: Major Major
Reporter: Sylvie Carrier

Building a "Maven project" in a Docker container fails with a "connection refused" error message whereas the same build is successful in a "Freestyle project" (also build in a Docker container plugin).

[ael.validate-in-docker-mvnproject] $ docker exec --tty --user 500: dc69be1aed975dbff977348d060329d75d1377ecbefaf929a269a11fc5e40373 env ANT_HOME=/opt/apache-ant-1.8.2 CLASSPATH= EXECUTOR_NUMBER=0 JAVA_HOME=/opt/jdk1.7.0_45 JENKINS_HOME=/data/EDT/jenkins M2_HOME=/opt/apache-maven-3.0.4 MAVEN_HOME=/opt/apache-maven-3.0.4 "MAVEN_OPTS=-Xms1g -Xmx1g -XX:MaxPermSize=256m -Dfile.encoding=Cp1252" no_proxy=/var/run/docker.sock ORACLE_HOME=/opt/oracle11/product/11.2.0/client/ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/jdk1.7.0_45/bin:/opt/apache-ant-1.8.2/bin:/opt/apache-maven-3.0.4/bin TERM=xterm WORKSPACE=/data/EDT/jenkinsSlave_recette_1/c3f621fe/workspace/ael.validate-in-docker-mvnproject java -cp /data/EDT/jenkinsSlave_recette_1/c3f621fe/maven3-agent.jar:/opt/apache-maven-3.0.4/boot/plexus-classworlds-2.4.jar org.jvnet.hudson.maven3.agent.Maven3Main /opt/apache-maven-3.0.4 /data/EDT/jenkinsSlave_recette_1/slave.jar /data/EDT/jenkinsSlave_recette_1/c3f621fe/maven3-interceptor.jar /data/EDT/jenkinsSlave_recette_1/c3f621fe/maven3-interceptor-commons.jar 44848
Exception in thread "main" java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:136)
at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:70)
Stopping Docker container after build completion

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

clausfod@gmail.com (JIRA)

unread,
Jan 20, 2016, 3:42:03 PM1/20/16
to jenkinsc...@googlegroups.com

dbeck@cloudbees.com (JIRA)

unread,
Feb 9, 2016, 9:12:01 AM2/9/16
to jenkinsc...@googlegroups.com

syl.carrier@laposte.net (JIRA)

unread,
Feb 9, 2016, 11:27:04 AM2/9/16
to jenkinsc...@googlegroups.com

No. We always had this behaviour. So we currently progress with freestyle job only...

dwyst01@ca.com (JIRA)

unread,
Feb 22, 2016, 11:14:02 AM2/22/16
to jenkinsc...@googlegroups.com

syl.carrier@laposte.net (JIRA)

unread,
Feb 23, 2016, 5:17:01 AM2/23/16
to jenkinsc...@googlegroups.com
Sylvie Carrier updated an issue
 
Change By: Sylvie Carrier
Environment: Jenkins ver. 1.625.3.1
Plugin Version: 1.6.
4 5

nicolas.deloof@gmail.com (JIRA)

unread,
Feb 29, 2016, 9:55:02 AM2/29/16
to jenkinsc...@googlegroups.com
Nicolas De Loof commented on Bug JENKINS-32542
 
Re: Maven project failed with "Connection refused" when built in a docker Container

is your jenkins slave itself running in a docker container ?

tomiphone3G@gmail.com (JIRA)

unread,
Feb 29, 2016, 3:55:05 PM2/29/16
to jenkinsc...@googlegroups.com

No the jenkins slave is "classic".
Jenkins master (Linux RHEL) => jenkins slave (another Linux RHEL)

aaryn101@gmail.com (JIRA)

unread,
Mar 7, 2016, 4:33:02 PM3/7/16
to jenkinsc...@googlegroups.com

My company almost exclusively uses Maven projects, and we are seeing this error. I'd really like to use this plugin but without Maven job support it is pretty much useless to us.

aaryn101@gmail.com (JIRA)

unread,
Mar 8, 2016, 11:23:03 AM3/8/16
to jenkinsc...@googlegroups.com

I discovered a workaround: changing the advanced setting "Network bridge" to "host" caused the error to go away and the build to proceed normally.

zcahill@zomp.net (JIRA)

unread,
Mar 11, 2016, 11:11:04 AM3/11/16
to jenkinsc...@googlegroups.com

We're having the same issue... In our case, as Nicolas De Loof considered, the slave is running within our JNLP docker container deployed upon Amazon EC2 Container Service, using his plugin.

The problem, it seems, stems from the network bridging; each container being in its own network and no "advanced" option within the CloudBees Docker Custom Build Environment Plugin to link containers. Although with a dynamically produced slave container name by the Amazon EC2 Container Service plugin, I guess this may need to utilise an environment variable if/when this feature becomes available?

Setting the "Network bridge" to host as Aaron Corley suggest does not help in our case as the containers are still on separate networks. However, I'm presuming if Nicolas De Loof's plugin had an advanced option to configure the "Network Bridge" for the Slave and setting it to "host" could cure this issue. However there may be other consequences to this...

For the time being we can build with a Freestyle project, but if I get a chance to review the plugin(s) sources I'll see if I can contribute a fix.

peter@retep.org (JIRA)

unread,
May 18, 2016, 6:29:03 AM5/18/16
to jenkinsc...@googlegroups.com

I've had a docker java slave running happily for a few weeks now but after adding the Cloudbees docker build environment plugin (may be a coincidence) it failed with this same issue, with one minor difference in that it included a hostname in the logs:

Modules changed, recalculating dependency graph
Established TCP socket on dockerhost:48877
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[opendata-common] $ /opt/jdk//bin/java -cp /opt/jenkins/maven32-agent.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/boot/plexus-classworlds-2.5.2.jar:/opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3/conf/logging jenkins.maven3.agent.Maven32Main /opt/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3 /opt/jenkins/slave.jar /opt/jenkins/maven32-interceptor.jar /opt/jenkins/maven3-interceptor-commons.jar dockerhost:48877
Exception in thread "main" java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:141)
	at jenkins.maven3.agent.Maven32Main.main(Maven32Main.java:71)
ERROR: Failed to launch Maven. Exit code = 1

Whilst looking for other possible causes for this I found

JENKINS-29674 which stated:

This is by intention, maven job type won't work with a docker container as it open an arbitrary socket for communication between slave agent and maven process, which we can't discover at container launch time

So as the exception gave me a hostname & as this was using my own docker images (both jenkins server and slaves) I added as part of the slaves entrypoint script the following:

echo "127.0.0.1 dockerhost" >>/etc/hosts

Now this may be a fluke that it's working & I don't know where dockerhost is being defined (it's not in any of my config) but this may help in finding the underlying issue.

Environment details: This was with Jenkins 2.5 (first spotted with 2.3) and an SSH based slave, both running within separate docker images. My docker images that have this fix in are area51/jenkins:2.5 & area51/jenkins-slave:2.5

ryan@shamu.ch (JIRA)

unread,
May 31, 2016, 10:47:03 AM5/31/16
to jenkinsc...@googlegroups.com

I'm using the kubernetes plugin to provision fungible slaves in a cluster for each build and I have this issue too.

Maybe the "dockerhost" hostname should actually be "jenkinshost" and be obtained by running somethig like "hostname -i" on the jenkins host performing the build (the slave container in my case) instead of the "ip route in an alpine:3.2 container" technique.

Some other issues remain as the port needs to be exposed in the slave pod.

  • The kubernetes plugin needs to be able to specify port exposition in slave templates
  • The jenkins-maven plugin needs to support using a fixed port for the jvm agent -> build host connection. (this would cause collisions in most cases, but not when using the kubernetes plugin as each build has it's own IP address).

ryan@shamu.ch (JIRA)

unread,
May 31, 2016, 12:21:02 PM5/31/16
to jenkinsc...@googlegroups.com

It all boils down to the "getDocker0" method in the custom-build-plugin. We should not assume that the address of the jenkins (master or slave) is the default gateway of the container it starts. The basic example is jenkins running in a container with a shared docker socket.

I think a portable solution could be to allow master/slave provisionners to specify and environment variable (for the master/slave process) with the value computed depending on their networking environment (there are so many different scenarios, we can give sensible defaults, but we also need to allow full customization).

The environment variable could be something like JENKINS_MAVEN_PLUGIN_HOST. I'll try to submit a PR tomorrow for this.

tobias.prinz@tngtech.com (JIRA)

unread,
Jul 5, 2016, 10:50:03 AM7/5/16
to jenkinsc...@googlegroups.com

Have the same issue. Any update on the PR?

This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages