Docker cloud plugin and pipeline using docker

[Christian Gagneraud]

Hi there,

I have a machine A that runs Jenkins and Bitbucket-server in docker containers.
I have a machine B, that just has docker listening over tcp.
I have a project that uses Declarative pipeline with "agent { docker
'foobar:X.Y'}"
I'm using "jenkinsci/blueocean:latest" docker image with
"docker-workflow v1.15" plugin.

Ignoring machine B (for now), my setup is working:
- 1. A dev open a pull-request on bitbucket
- 2. The declarative pipline is executed by jenkins
- 3. The docker container(s) specified in the Jenkinsfile are executed
by machine A
- 4. The pipeline finished
- 5. Jenkins notify Bitbucket.

From a docker point of view, for the above to work, I simply need to
run the Jenkins container with --volume
/var/run/docker.sock:/var/run/docker.sock

Now, I would like to achieve 3 things:
- Leave machine A runs non-build services only
- Use machine B to spawn containers as directed by declarative pipelines.
- Eventually make machine B a docker swarm manager, and let other
docker swarm workers join in if/when needed.

The first modification I did to the jenkins container is to remove the
"/var/run/docker.sock" volume. This is to make sure jenkins can only
talk to the remote docker I configure.

I have then tried 3 plugins:
- "docker plugin" https://plugins.jenkins.io/docker-plugin
- "docker swarm" https://plugins.jenkins.io/docker-swarm
- "Yet another docker plugin"
https://plugins.jenkins.io/yet-another-docker-plugin

All of them add a type of "cloud providers", but I never managed to
get my pipeline executed using one.

The "docker plugin" is the only one that gave me the feeling I was almost there:
The job was handled by a docker container spawn on machine B, which i
had to specify the image name (From examples and tutorials, i settled
down on "jenkinsci/jnlp-slave" in "Launch attached" mode).
But at that stage, I got an error "docker: command not found" emitted
from the jnlp-slave container.
The reason being that Jenkins master contacted docker on machine B to
execute a "jenkinsci/jnlp-slave", and then ask this
"jenkinsci/jnlp-slave" to execute the pipeline, which is an operation
that requires to instantiate the docker containers specified in the
project's pipeline...

So I would like to know if the "docker-plugin" is the right tool for
my job or should I head in another direction.
In case "docker-plugin" is the way to go, then how do I tell my
customised jnlp-slave to connect to the right docker daemon? Is there
an off-the-shelf docker image to play that role? I couldn't find
documentation on how to create it, and the specificites of having
docker inside it.

As a side note:
I found a bit inefficient to have to instantiate Jenkins slave
container whose sole purpose is to read the pipeline and spawn
containers as directed by the pipeline.
Ideally I would like the pipeline to be parsed by Machine A (jenkins
master) and the required containers to be executed on machine B, I do
not know if volumes (jenkins workspace) can be shared across 2 docker
daemons thought...

Any opinion or advice are very welcome, point-out of any sort would be
greatly appreciated too.

Thanks,
Chris

[nicolas de loof]

docker support in declarative pipeline relies on docker-workfow plugin, which require the jenkins slave to have access to a LOCAL docker daemon, which means the docker agent you are provisioning on computer B using docker plugin HAS to be configured with /var/run/docker.sock (so, the one from B) bind mounted. Doing so it can then create some side container to host the build steps.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CABxGUTgbu2BPOO1MRcPVPaxqqGhe306kU_LBAWf3H2jO2RooFA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Christian Gagneraud]

On 26 February 2018 at 03:10, nicolas de loof <nicolas...@gmail.com> wrote:
> docker support in declarative pipeline relies on docker-workfow plugin,
> which require the jenkins slave to have access to a LOCAL docker daemon,
> which means the docker agent you are provisioning on computer B using docker
> plugin HAS to be configured with /var/run/docker.sock (so, the one from B)
> bind mounted. Doing so it can then create some side container to host the
> build steps.

Hi Nicolas,

Thanks for the tips, i think i'm getting there, but i'm still missing some bits:
- I have created a docker image based on jenkinsci/jnlp-slave, the
only customisation is installing docker in it
- The pipeline now get checkout inside tmy slave (on machine B)
- The docker image from the Jenkinsfile is pulled and executed.

But i get this weird error (see full log at the end of this email):
------------------------
docker-44734997459c on work-leader seems to be running inside
container e7f7a4381e32c456b806695b9b7334751d88f1a3985867bb964e9fd3d2e8c948
but /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent]
but /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp
could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent]
$ docker run -t -d -u 10000:10000 -w
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
-v /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q:/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q:rw,z
-v /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp:/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp:rw,z
-e ******** -e ******** -e ******** -e ******** -e ******** -e
******** -e ******** -e ******** -e ******** -e ******** -e ********
-e ******** -e ******** -e ******** -e ******** -e ******** -e
******** -e ******** -e ******** -e ******** -e ******** -e ********
-e ******** -e ******** -e ******** -e ******** -e ******** -e
******** python:2.7 cat
$ docker top 20801bdc7f7f4777b89a864c2cd52b03e3906d02ad3c708df4c4cdc5ebe4f45b
-eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (dummy)
[Pipeline] sh

[aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q]
Running shell script
sh: 1: cannot create
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp/durable-f19371c8/jenkins-log.txt:
Directory nonexistent
sh: 1: cannot create
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp/durable-f19371c8/jenkins-result.txt:
Directory nonexistent

process apparently never started in
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp/durable-f19371c8
-----------------------

work-leader is machine B, docker-44734997459c is, I believe, an
instance of my custom jnlp-slave, as you can see the slave fails to
run the python container

I wonder if this could be some permissions issues, for now machine B
is a VBox managed using vagrant, on it I can see that:
vagrant@leader:~$ ls -l /home/jenkins/workspace/
total 8
drwxr-xr-x 2 root root 4096 Feb 26 02:02
aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
drwxr-xr-x 2 root root 4096 Feb 26 02:02
aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp
vagrant@leader:~$ ls -la /home/jenkins/workspace/*
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q:
total 8
drwxr-xr-x 2 root root 4096 Feb 26 02:02 .
drwxr-xr-x 4 root root 4096 Feb 26 02:02 ..

/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp:
total 8
drwxr-xr-x 2 root root 4096 Feb 26 02:02 .
drwxr-xr-x 4 root root 4096 Feb 26 02:02 ..

Any idea?
Chris

Here is my Dockerfile (jenkins-slave-agent):
-------------------------
FROM jenkinsci/jnlp-slave
USER root
curl -sfL -o docker.tgz https://.... && tar -C /usr/bin -xzf
docker.tgz && rm docker.tgz
USER jenkins
-------------------------

And the project's Jenkinsfile:
------------------------------
pipeline {
agent { docker 'python:2.7'}
stages { stage('dummy') { steps { sh 'python --version' } } }
}
------------------------------

Full build log:
------------------------------
Started by user admin
Obtained Jenkinsfile from 0be91558399793412462578e0061fef50a7919c3
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on docker-44734997459c on work-leader in
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://code.navico.cloud/scm/AA/aaa.git
> git init /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
# timeout=10
Fetching upstream changes from https://code.navico.cloud/scm/AA/aaa.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials bb
> git fetch --no-tags --progress
https://code.navico.cloud/scm/AA/aaa.git
+refs/heads/master:refs/remotes/origin/master
> git config remote.origin.url
https://code.navico.cloud/scm/AA/aaa.git # timeout=10
> git config --add remote.origin.fetch
+refs/heads/master:refs/remotes/origin/master # timeout=10
> git config remote.origin.url
https://code.navico.cloud/scm/AA/aaa.git # timeout=10
Fetching without tags
Fetching upstream changes from https://code.navico.cloud/scm/AA/aaa.git
using GIT_ASKPASS to set credentials bb
> git fetch --no-tags --progress
https://code.navico.cloud/scm/AA/aaa.git
+refs/heads/master:refs/remotes/origin/master
Checking out Revision 0be91558399793412462578e0061fef50a7919c3 (master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 0be91558399793412462578e0061fef50a7919c3
Commit message: "WIP"
> git rev-list --no-walk 0be91558399793412462578e0061fef50a7919c3 # timeout=10
[Bitbucket] Notifying commit build result
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Agent Setup)
[Pipeline] sh
[aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q]
Running shell script
+ docker pull python:2.7
2.7: Pulling from library/python
Digest: sha256:d52278db7a8a4608e2b3c4b7a1a1eff76849542481144603bb9072512395a6d9
Status: Image is up to date for python:2.7
[Pipeline] }
[Pipeline] // stage
[Pipeline] sh
[aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q]
Running shell script
+ docker inspect -f . python:2.7
.
[Pipeline] withDockerContainer
docker-44734997459c on work-leader seems to be running inside
container e7f7a4381e32c456b806695b9b7334751d88f1a3985867bb964e9fd3d2e8c948
but /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent]
but /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp
could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent]
$ docker run -t -d -u 10000:10000 -w
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
-v /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q:/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q:rw,z
-v /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp:/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp:rw,z
-e ******** -e ******** -e ******** -e ******** -e ******** -e
******** -e ******** -e ******** -e ******** -e ******** -e ********
-e ******** -e ******** -e ******** -e ******** -e ******** -e
******** -e ******** -e ******** -e ******** -e ******** -e ********
-e ******** -e ******** -e ******** -e ******** -e ******** -e
******** python:2.7 cat
$ docker top 20801bdc7f7f4777b89a864c2cd52b03e3906d02ad3c708df4c4cdc5ebe4f45b
-eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (dummy)
[Pipeline] sh
[aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q]
Running shell script
sh: 1: cannot create
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp/durable-f19371c8/jenkins-log.txt:
Directory nonexistent
sh: 1: cannot create
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp/durable-f19371c8/jenkins-result.txt:
Directory nonexistent
process apparently never started in
/home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q@tmp/durable-f19371c8
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1
20801bdc7f7f4777b89a864c2cd52b03e3906d02ad3c708df4c4cdc5ebe4f45b
$ docker rm -f 20801bdc7f7f4777b89a864c2cd52b03e3906d02ad3c708df4c4cdc5ebe4f45b
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[Bitbucket] Notifying commit build result
[Bitbucket] Build result notified
ERROR: script returned exit code -2
Finished: FAILURE
----------------------

[Christian Gagneraud]

On 26 February 2018 at 15:29, Christian Gagneraud <chg...@gmail.com> wrote:
> On 26 February 2018 at 03:10, nicolas de loof <nicolas...@gmail.com> wrote:
>> docker support in declarative pipeline relies on docker-workfow plugin,
>> which require the jenkins slave to have access to a LOCAL docker daemon,
>> which means the docker agent you are provisioning on computer B using docker
>> plugin HAS to be configured with /var/run/docker.sock (so, the one from B)
>> bind mounted. Doing so it can then create some side container to host the
>> build steps.
>
> Hi Nicolas,
>
> Thanks for the tips, i think i'm getting there, but i'm still missing some bits:
> - I have created a docker image based on jenkinsci/jnlp-slave, the
> only customisation is installing docker in it
> - The pipeline now get checkout inside tmy slave (on machine B)
> - The docker image from the Jenkinsfile is pulled and executed.
>
> But i get this weird error (see full log at the end of this email):
> ------------------------
> docker-44734997459c on work-leader seems to be running inside
> container e7f7a4381e32c456b806695b9b7334751d88f1a3985867bb964e9fd3d2e8c948
> but /home/jenkins/workspace/aa_aaa_master-GQ43GQETZSZCK3GMCGYSLEW5TKL7YKHSBXYJPJLEX7A2AM6K5M6Q
> could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent]

Google didn't return much with "could not be found among"

https://stackoverflow.com/questions/47559469/jenkins-declarative-pipeline-docker-agent-timeout-run?rq=1
https://tutel.me/c/programming/questions/46530963/run+jenkins+agent+in+docker+container+issue+with+jenkins+pipeline+and+varrundockersocket

Not very useful, but using google's "repeat the search with the
omitted results included.", I got a few extra results, one of them, an
IRD log, that gave me the answer:
https://echelog.com/logs/browse/jenkins/1497304800

I had to add /home/jenkins in the "Volumes" of the "Docker Agent
templates" (as well as "Remoting Filling System Root", but this was
set by default).
I noticed as well that if no "user" is specified in the "Connect
method"/"Attach Docker Container", the Python image script will be run
with id 1000:1000, which does not exists. But this doesn't seem to be
a problem. (user=root is not required, nor is "Run container
privileged").

Thanks!
Chris

[nicolas de loof]

For a dockerized agent to be able to run docker pipeline, it has to use a volume for workspace directory so this one can be shared with other containers it creates.

Your docker agent template should be configured to use "/home/jenkins/agent" as agent root directory, as this one is a volume and as such workspace sub-folder will be shareable with other containers.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CABxGUTjm_P7nc0pk2WntGjq1BX2Eoacn6tD9xqwskjLpxg2_ZA%40mail.gmail.com.

[Christian Gagneraud]

On 27 February 2018 at 04:55, nicolas de loof <nicolas...@gmail.com> wrote:
> For a dockerized agent to be able to run docker pipeline, it has to use a
> volume for workspace directory so this one can be shared with other
> containers it creates.
> Your docker agent template should be configured to use "/home/jenkins/agent"
> as agent root directory, as this one is a volume and as such workspace
> sub-folder will be shareable with other containers.

Thanks a lot for the tip, greatly appreciated!

Chris