Problems with (Declarative) pipeline in AWS ECS

[Christian Gagneraud]

Hi there,

Running Jenkins 2.89.4 (BlueOcean), with Amazon EC2 Container Service
plugin v 1.11

I have successfully setup my self-hosted docker host connected to
jenkins using the "docker cloud" plugin, i can get this cloud execute
declarative pipelines that needs docker container such as:
----------------------------------------------------------
pipeline {
agent { docker 'python:2.7'}
stages {
stage('Dummy stage') {
steps {
sh 'python --version'
}
}
}
}
----------------------------------------------------------

I have successfully setup an Amazon ECS connected to jenkins using the
"Amazon EC2 Container Service plugin", i can get this cloud execute
dead-simple job: no git repo, just execute a few shell commands such
as:
----------------------------------------------------
echo "Hello world! From Amazon ECS"
----------------------------------------------------

The console output says:
-----------------------------------------------------
Started by user admin
Building remotely on ecs-builder-de-35016db04c851 (ecs-de.cloud) in
workspace /home/jenkins/workspace/test-aws
[test-aws] $ /bin/sh -xe /tmp/jenkins8406299037257519290.sh
+ echo Hello world! From Amazon ECS.
Hello world! From Amazon ECS.
Finished: SUCCESS
------------------------------------------------------

When i try to execute a job from my project that uses the declarative
pipeline, the last step (the "Dummy stage") fails with:
-------------------------------------------------------
sh: 1: cannot create
/home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA@tmp/durable-e5c611c5/jenkins-log.txt:
Directory nonexistent
-------------------------------------------------------

To reach this stage i'm using a custom ECS slave templates:
-------------------------------------------------------
Template Name = jenkins-slave
Label = docker.jenkins.slave
Docker Image = chgans/jenkins-slave-ecs:latest
Filesystem root = /home/jenkins
Soft Memory Reservation = 1024
Hard Memory Reservation = 0
CPU units = 1
DNS Search Domains
Task Role ARN
Override entrypoint
JVM arguments
Privileged
Logging Driver
Logging Configuration
Environments
Extra Hosts
Container Mount Points
Name = docker_socket
source Path = /var/run/docker.sock
Container Path = /var/run/docker.sock
Read Only
-------------------------------------------------------

The custom docker image deals mainly with user/group issues.

When i execute the pipeline, here is what i get in the console output:
Note: The project name is "application", and the pull request is
"feature/gfdgfgfdgfdgfd"
-------------------------------------------------------
Started by user admin
Obtained Jenkinsfile from 955ff5122ac1ab67ff8b70743d0a2064a63ceb9a
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on ecs-builder-de-352c7774fa56f in
/home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
< ... snipped git stuff ...>
> git checkout -f 955ff5122ac1ab67ff8b70743d0a2064a63ceb9a
Commit message: "Jenkinsfile edited online with Bitbucket"
> git rev-list --no-walk 955ff5122ac1ab67ff8b70743d0a2064a63ceb9a # timeout=10
[Bitbucket] Notifying commit build result
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Agent Setup)
[Pipeline] sh
[tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA]
Running shell script
+ docker pull python:2.7
2.7: Pulling from library/python
Digest: sha256:83b5b650aadb1c85a9764da0e7f6e4caa8dbb4c387ced0aca6ff510980576f31
Status: Image is up to date for python:2.7
[Pipeline] }
[Pipeline] // stage
[Pipeline] sh
[tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA]
Running shell script
+ docker inspect -f . python:2.7
.
[Pipeline] withDockerContainer
ecs-builder-de-352c7774fa56f seems to be running inside container
52fcf60dcae91c7bab6a1ec17209482018f1dc528264360b2d1706209e681086
but /home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA
could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent,
/var/run/docker.sock]
but /home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA@tmp
could not be found among [/home/jenkins/.jenkins, /home/jenkins/agent,
/var/run/docker.sock]
$ docker run -t -d -u 10000:10000 -w
/home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA
-v /home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA:/home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA:rw,z
-v /home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA@tmp:/home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA@tmp:rw,z
-e ******** -e ******** -e ******** -e ******** -e ******** -e
******** -e ******** -e ******** -e ******** -e ******** -e ********
-e ******** -e ******** -e ******** -e ******** -e ******** -e
******** -e ******** -e ******** -e ******** -e ******** -e ********
-e ******** -e ******** -e ******** -e ******** -e ******** -e
******** -e ******** python:2.7 cat
$ docker top 81ff4619d3fab2e7bfad36ba988e0c99116d888503dad070d2b99ef71f14cc54
-eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Dummy stage)
[Pipeline] sh
[tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA]
Running shell script
sh: 1: cannot create
/home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA@tmp/durable-e5c611c5/jenkins-log.txt:
Directory nonexistent
sh: 1: cannot create
/home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA@tmp/durable-e5c611c5/jenkins-result.txt:
Directory nonexistent
process apparently never started in
/home/jenkins/workspace/tion_feature_gfdgfgfdgfdgfd-KZ7OXHZWKOBIKEE7D6XZOM3UNTYU5DGJSLNIET2I5IITHIBHBWPA@tmp/durable-e5c611c5
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1
81ff4619d3fab2e7bfad36ba988e0c99116d888503dad070d2b99ef71f14cc54
$ docker rm -f 81ff4619d3fab2e7bfad36ba988e0c99116d888503dad070d2b99ef71f14cc54
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[Bitbucket] Notifying commit build result
[Bitbucket] Build result notified
ERROR: script returned exit code -2
Finished: FAILURE
-------------------------------------------------------------------


I have the feeling that i have almost everything in place, but for
some reason, the docker container used for a given stage has an
incomplete /home/jenkins/workspace.

I have tried to play with the mounting container mounting points of
the ECS slave templates, but i'm unlucky so far (permission issues on
checkout.

It looks to me that the templated jenkins-slave container doesn't
share /homme/jenkins with it's host, and so side containers
(python:2.7 in my case) cannot inherit them.

Does anyone know what sort of tweaking are required for such a setup?
Do i need to customised the EC2 docker host? I'm running the default
AWS AMI Linux.

Any point out or tips appreciated.

Thanks,
Chris

[Christian Gagneraud]

On 17 March 2018 at 21:15, Christian Gagneraud <chg...@gmail.com> wrote:
[...]

> It looks to me that the templated jenkins-slave container doesn't
> share /homme/jenkins with it's host, and so side containers
> (python:2.7 in my case) cannot inherit them.
>
> Does anyone know what sort of tweaking are required for such a setup?
> Do i need to customised the EC2 docker host? I'm running the default
> AWS AMI Linux.

I managed to troubleshoot a bit the problem, I don't have a proper
fix, but this should help understand what's going wrong:

I first noticed on the EC2 docker host that /home/jenkins and all
subdirs were owned by root.
So i decided to create a jenkins user/group with id 10000 in the EC2
docker host.
I then made sure that my chgans/jenkins-slave-ecs container run the
entry point as user jenkins.
I added the mount point
"/home/jenkins/workspace:/home/jenkins/workspace" in my ECS slave
template.

But his was not enough, /home/jenkins was owned by jenkins.jenkins
(created via adduser command), but the workspace sub-directory was
automatically created and owned by root.

So I manually created /home/jenkins/workspace and made sure it was
owned by jenkins.jenkins.

Now when i run a job, the templated jenkins slave can clone stuff in
it's workspace, with the right ownership with regards to the docker
host.
The declarative pipeline plugin then creates the sister containers
(python:2.7 in my case) from within the jenkins slave, using
'--volumes-from <jenkins-slave>' and '-u 10000:10000'.

Which lead me to the following questions:
- do I have to setup my own EC2 and make sure it has a jenkins user?
- Who is creating /home/jenkins/workspace as root, and why?
- Why do I have to share workspace, shouldn't /home/jenkins/agent be
all what is necessary?

Thanks,
Chris