[JIRA] (JENKINS-52144) unable to provision pods w/ plugin > 1.4.1

5 views
Skip to first unread message

jgangemi@gmail.com (JIRA)

unread,
Jun 22, 2018, 10:47:03 AM6/22/18
to jenkinsc...@googlegroups.com
Jae Gangemi created an issue
 
Jenkins / Bug JENKINS-52144
unable to provision pods w/ plugin > 1.4.1
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Components: kubernetes-plugin
Created: 2018-06-22 14:46
Priority: Blocker Blocker
Reporter: Jae Gangemi

after upgrading from 1.4.1 to any new version, the latest being 1.8.4 i am unable to provision any pods in kubernetes. NOTHING has changed in my configuration, which means something is different in the plugin.

in 1.4.1 i did NOT have to specify any working directory as part of the configuration, it is an empty value. for 1.8.4 it seems i am forced to pick a value and it defaults to /home/jenkins (ugh) if nothing is selected. i have explicitly created a /home/jenkins/.jenkins-agent directory and set the value to that, but still no luck.

here is the pod template (from the logs) that gets created:

Pod built: Pod(apiVersion=v1, kind=Pod, metadata=ObjectMeta(annotations={}, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={jenkins=slave, jenkins/maven-8-debian=true}, name=maven-8-debian-s9xnf, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[d71ba3254f49281ef132d9bd8c841dc52386c1b46c919a64e863a57912408399, maven-8-debian-s9xnf], command=[], env=[EnvVar(name=JENKINS_SECRET, value=d71ba3254f49281ef132d9bd8c841dc52386c1b46c919a64e863a57912408399, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_NAME, value=maven-8-debian-s9xnf, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_URL, value=http://jenkins-beta.infra.us-east-1.conductor.sh/, valueFrom=null, additionalProperties={}), EnvVar(name=HOME, value=/home/jenkins/.jenkins-agent, valueFrom=null, additionalProperties={})], envFrom=[], image=registry.battery-park.conductor.com/jenkins-slave-8-maven-debian, imagePullPolicy=Always, lifecycle=null, livenessProbe=null, name=jnlp, ports=[], readinessProbe=null, resources=ResourceRequirements(limits={}, requests={}, additionalProperties={}), securityContext=SecurityContext(capabilities=null, privileged=false, readOnlyRootFilesystem=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, additionalProperties={}), stdin=null, stdinOnce=null, terminationMessagePath=null, terminationMessagePolicy=null, tty=true, volumeMounts=[VolumeMount(mountPath=/home/jenkins/.jenkins-agent, name=workspace-volume, readOnly=false, subPath=null, additionalProperties={})], workingDir=/home/jenkins/.jenkins-agent, additionalProperties={})], dnsPolicy=null, hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector={}, restartPolicy=Never, schedulerName=null, securityContext=null, serviceAccount=null, serviceAccountName=null, subdomain=null, terminationGracePeriodSeconds=null, tolerations=[], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, downwardAPI=null, emptyDir=EmptyDirVolumeSource(medium=, sizeLimit=null, additionalProperties={}), fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=workspace-volume, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={}), status=null, additionalProperties={})

and this the resulting error i get back

Error in provisioning; agent=KubernetesSlave name: maven-8-debian-s9xnf, template=PodTemplate{inheritFrom='', name='maven-8-debian', namespace='', label='maven-8-debian', nodeSelector='', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], containers=[ContainerTemplate{name='jnlp', image='registry.battery-park.conductor.com/jenkins-slave-8-maven-debian', alwaysPullImage=true, workingDir='/home/jenkins/.jenkins-agent', command='', args='${computer.jnlpmac} ${computer.name}', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@5f5745d0}], yaml=}
java.lang.IllegalStateException: Containers are terminated with exit codes: {jnlp=127}
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:156)
	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

i am unable to get any details about the pod b/c it gets deleted before i can get any information about it.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Jun 22, 2018, 10:54:02 AM6/22/18
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-52144
 
Re: unable to provision pods w/ plugin > 1.4.1

you need to get the container logs, without that there's not enough info

You can continuously run something like

for pod in $(kubectl get pods -o name | grep mypod); do kubectl logs $pod; done

jgangemi@gmail.com (JIRA)

unread,
Jun 22, 2018, 11:50:03 AM6/22/18
to jenkinsc...@googlegroups.com

this seems to be related to whatever is happening to the working directory / jenkins home when the container is started.

in my `Dockerfile` i have this:

USER jenkins
WORKDIR ${JENKINS_HOME}

ENTRYPOINT ["slave-scripts/jenkins-slave.sh"]

where JENKINS_HOME == /home/jenkins

and i have found this in the system logs:

Jun 22 15:45:28 ip-172-22-1-63.ec2.internal containerd[1004]: time="2018-06-22T15:45:28.017677203Z" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"slave-scripts/jenkins-slave.sh\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\"\n" id=27bc57569f3ce34c9cf120b5fb80d77a44e59b46d8a557190d46835232f6ee28
Jun 22 15:45:28 ip-172-22-1-63.ec2.internal dockerd[1212]: time="2018-06-22T15:45:28.018189578Z" level=error msg="Create container failed with error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"exec: \\\\\\\"slave-scripts/jenkins-slave.sh\\\\\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\"\\n\""
Jun 22 15:45:28 ip-172-22-1-63.ec2.internal dockerd[1212]: time="2018-06-22T15:45:28.036649033Z" level=error msg="Handler for POST /containers/27bc57569f3ce34c9cf120b5fb80d77a44e59b46d8a557190d46835232f6ee28/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"exec: \\\\\\\"slave-scripts/jenkins-slave.sh\\\\\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\"\\n\""
Jun 22 15:45:28 ip-172-22-1-63.ec2.internal kubelet[1773]: E0622 15:45:28.036968    1773 docker_manager.go:2250] container start failed: RunContainerError: runContainer: Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"exec: \\\\\\\"slave-scripts/jenkins-slave.sh\\\\\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\"\\n\""}
Jun 22 15:45:28 ip-172-22-1-63.ec2.internal kubelet[1773]: E0622 15:45:28.037029    1773 pod_workers.go:184] Error syncing pod 48cbf1ff-7633-11e8-b3fa-0ef76f0169b6, skipping: failed to "StartContainer" for "jnlp" with RunContainerError: "runContainer: Error response from daemon: {\"message\":\"invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"exec: \\\\\\\\\\\\\\\"slave-scripts/jenkins-slave.sh\\\\\\\\\\\\\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\\\\\"\\\\n\\\"\"}"

which seems to be indicating that it can't find the script in /home/jenkins.

setting the working directory to something completely different /tmp/jenkins-agent doesn't solve anything either.

but this seems to be related to something potentially mounting over /home/jenkns when it shouldn't be.

 

 

 

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Jun 22, 2018, 1:45:02 PM6/22/18
to jenkinsc...@googlegroups.com

/home/jenkins is mounted as a wolume shared across all containers. There is an option to customize it in the UI, I don't recall if it can also remove it altogether.

But if you move your startup script somewhere else it should work

jgangemi@gmail.com (JIRA)

unread,
Jun 22, 2018, 1:52:01 PM6/22/18
to jenkinsc...@googlegroups.com

i have other things in the home directory that get used so i can't just up and move everything.

i used to be able to remove the working directory entirely in 1.4.1 but i can no longer do that in the latest version. i have even tried setting the working dir to a different directory and i still get the same behavior.

i don't need a shared volume for my use case, i only run a single container, and this change is forcing me to have one.

 

jgangemi@gmail.com (JIRA)

unread,
Sep 19, 2018, 6:23:02 PM9/19/18
to jenkinsc...@googlegroups.com

any chance of seeing movement on this? JENKINS-50659 references the same issue i am where the behavior changed from 1.4 to 1.5 and it seems to all be related to the working directory between versions.

i tried the latest version 1.12.2 and it's still broken.

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

stanislav.pugachev@gmail.com (JIRA)

unread,
Oct 24, 2018, 7:13:02 AM10/24/18
to jenkinsc...@googlegroups.com

I'm getting same behaviour but error is "{jnlp=0}"
Is it related to this ticket or is it a different problem?

jglick@cloudbees.com (JIRA)

unread,
Jul 16, 2019, 3:43:25 PM7/16/19
to jenkinsc...@googlegroups.com
Jesse Glick assigned an issue to Unassigned
 
Change By: Jesse Glick
Assignee: Carlos Sanchez
Reply all
Reply to author
Forward
0 new messages