Problems with Jenkins

[kurr...@gmail.com]

Hello

We are running Fabric8 on versions:

OpenShift Master: v3.3.1.7

Kubernetes Master: v1.3.0+52492b4

There are several problems with Jenkins.

Based on all the info below, is anyone able to tell us how we can fix this? Any help would be greatly appreciated.

Jenkins Build Error

When attempting to run a Jenkins build, the build error is:

NOTE: running pipelines for the first time will take longer as build and base docker images are pulled onto the node

[Pipeline] container

[Pipeline] {

[Pipeline] stage (Build Release)

Using the ‘stage’ step without a block argument is deprecated

Entering stage Build Release

Proceeding

[Pipeline] sh

[shiftwork-staffservice-pipeline] Running shell script

Waiting for container container [maven] of pod [jenkins-17-fzkj9] to become ready.

[Pipeline] }

[Pipeline] // container

[Pipeline] }

[Pipeline] // node

[Pipeline] }

[Pipeline] // podTemplate

[Pipeline] End of Pipeline

java.io.IOException: Failed to execute shell script inside container [maven] of pod [jenkins-17-fzkj9]. Timed out waiting for container to become ready!

at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:88)

at hudson.Launcher$ProcStarter.start(Launcher.java:381)

at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:147)

at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:61)

at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:158)

at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)

at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)

at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)

at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)

at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)

at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)

at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)

at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)

at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)

at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:103)

at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)

at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)

at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)

at mavenCanaryRelease.call(/var/jenkins_home/jobs/shiftwork-staffservice-pipeline/builds/59/libs/github.com/fabric8io/fabric8-pipeline-library/vars/mavenCanaryRelease.groovy:11)

at WorkflowScript.run(WorkflowScript:39)

at ___cps.transform___(Native Method)

at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:48)

at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)

at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)

at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)

at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)

at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)

at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)

at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:50)

at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)

at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)

at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)

at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)

at com.cloudbees.groovy.cps.Next.step(Next.java:58)

at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)

at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)

at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)

at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)

at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)

at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)

at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)

at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)

at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)

at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)

at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)

at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)

at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Finished: FAILURE

Kubernetes Events

17s        17s         1         kubernetes-becd7c1893f64c7588794ee45551586c-1517d1f995738   Pod                                                      Normal    Scheduled               {default-scheduler }                   Successfully assigned kubernetes-becd7c1893f64c7588794ee45551586c-1517d1f995738 to openshift.hughestech.space
13m        13m         1         kubernetes-e6c242de4a494b4d90987fa58b875e74-150caf5aef7c6   Pod                                                      Normal    Scheduled               {default-scheduler }                   Successfully assigned kubernetes-e6c242de4a494b4d90987fa58b875e74-150caf5aef7c6 to openshift.hughestech.space
4m         11m         4         kubernetes-e6c242de4a494b4d90987fa58b875e74-150caf5aef7c6   Pod                                                      Warning   FailedMount             {kubelet openshift.hughestech.space}   Unable to mount volumes for pod "kubernetes-e6c242de4a494b4d90987fa58b875e74-150caf5aef7c6_fabric8(6e18bd90-d761-11e6-8e7a-002421dde3d7)": timeout expired waiting for volumes to attach/mount for pod "kubernetes-e6c242de4a494b4d90987fa58b875e74-150caf5aef7c6"/"fabric8". list of unattached/unmounted volumes=[volume-0]
4m         11m         4         kubernetes-e6c242de4a494b4d90987fa58b875e74-150caf5aef7c6   Pod                                                      Warning   FailedSync              {kubelet openshift.hughestech.space}   Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "kubernetes-e6c242de4a494b4d90987fa58b875e74-150caf5aef7c6"/"fabric8". list of unattached/unmounted volumes=[volume-0]
4m         4m          1         kubernetes-e6c242de4a494b4d90987fa58b875e74-1514404f7b419   Pod                                                      Normal    Scheduled               {default-scheduler }                   Successfully assigned kubernetes-e6c242de4a494b4d90987fa58b875e74-1514404f7b419 to openshift.hughestech.space
9s         2m          2         kubernetes-e6c242de4a494b4d90987fa58b875e74-1514404f7b419   Pod                                                      Warning   FailedMount             {kubelet openshift.hughestech.space}   Unable to mount volumes for pod "kubernetes-e6c242de4a494b4d90987fa58b875e74-1514404f7b419_fabric8(a40176c1-d762-11e6-8e7a-002421dde3d7)": timeout expired waiting for volumes to attach/mount for pod "kubernetes-e6c242de4a494b4d90987fa58b875e74-1514404f7b419"/"fabric8". list of unattached/unmounted volumes=[volume-0]
9s         2m          2         kubernetes-e6c242de4a494b4d90987fa58b875e74-1514404f7b419   Pod                                                      Warning   FailedSync              {kubelet openshift.hughestech.space}   Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "kubernetes-e6c242de4a494b4d90987fa58b875e74-1514404f7b419"/"fabric8". list of unattached/unmounted volumes=[volume-0]
40m        40m         1         kubernetes-f80d502fcb2f4360a28499a14f50230c-14f487674f40d   Pod                                                      Normal    Scheduled               {default-scheduler }                   Successfully assigned kubernetes-f80d502fcb2f4360a28499a14f50230c-14f487674f40d to openshift.hughestech.space
34m        38m         3         kubernetes-f80d502fcb2f4360a28499a14f50230c-14f487674f40d   Pod                                                      Warning   FailedMount             {kubelet openshift.hughestech.space}   Unable to mount volumes for pod "kubernetes-f80d502fcb2f4360a28499a14f50230c-14f487674f40d_fabric8(90be918a-d75d-11e6-8e7a-002421dde3d7)": timeout expired waiting for volumes to attach/mount for pod "kubernetes-f80d502fcb2f4360a28499a14f50230c-14f487674f40d"/"fabric8". list of unattached/unmounted volumes=[volume-0]
34m        38m         3         kubernetes-f80d502fcb2f4360a28499a14f50230c-14f487674f40d   Pod                                                      Warning   FailedSync              {kubelet openshift.hughestech.space}   Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "kubernetes-f80d502fcb2f4360a28499a14f50230c-14f487674f40d"/"fabric8". list of unattached/unmounted volumes=[volume-0]
34m        34m         1         kubernetes-f80d502fcb2f4360a28499a14f50230c-14fa594d26add   Pod                                                      Normal    Scheduled               {default-scheduler }                   Successfully assigned kubernetes-f80d502fcb2f4360a28499a14f50230c-14fa594d26add to openshift.hughestech.space
27m        32m         3         kubernetes-f80d502fcb2f4360a28499a14f50230c-14fa594d26add   Pod                                                      Warning   FailedMount             {kubelet openshift.hughestech.space}   Unable to mount volumes for pod "kubernetes-f80d502fcb2f4360a28499a14f50230c-14fa594d26add_fabric8(7f0ca76f-d75e-11e6-8e7a-002421dde3d7)": timeout expired waiting for volumes to attach/mount for pod "kubernetes-f80d502fcb2f4360a28499a14f50230c-14fa594d26add"/"fabric8". list of unattached/unmounted volumes=[volume-0]
27m        32m         3         kubernetes-f80d502fcb2f4360a28499a14f50230c-14fa594d26add   Pod                                                      Warning   FailedSync              {kubelet openshift.hughestech.space}   Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "kubernetes-f80d502fcb2f4360a28499a14f50230c-14fa594d26add"/"fabric8". list of unattached/unmounted volumes=[volume-0]
27m        27m         1         kubernetes-f80d502fcb2f4360a28499a14f50230c-15002b68a6ded   Pod                                                      Normal    Scheduled               {default-scheduler }                   Successfully assigned kubernetes-f80d502fcb2f4360a28499a14f50230c-15002b68a6ded to openshift.hughestech.space
20m        25m         3         kubernetes-f80d502fcb2f4360a28499a14f50230c-15002b68a6ded   Pod                                                      Warning   FailedMount             {kubelet openshift.hughestech.space}   Unable to mount volumes for pod "kubernetes-f80d502fcb2f4360a28499a14f50230c-15002b68a6ded_fabric8(6d765605-d75f-11e6-8e7a-002421dde3d7)": timeout expired waiting for volumes to attach/mount for pod "kubernetes-f80d502fcb2f4360a28499a14f50230c-15002b68a6ded"/"fabric8". list of unattached/unmounted volumes=[volume-0]
20m        25m         3         kubernetes-f80d502fcb2f4360a28499a14f50230c-15002b68a6ded   Pod                                                      Warning   FailedSync              {kubelet openshift.hughestech.space}   Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "kubernetes-f80d502fcb2f4360a28499a14f50230c-15002b68a6ded"/"fabric8". list of unattached/unmounted volumes=[volume-0]

Jenkinsfile

Note, when using def label = "buildpod.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_') then jenkins gets stuck on:

Still waiting to schedule task
kubernetes-e6c242de4a494b4d90987fa58b875e74-150caf5aef7c6 is offline

#!/usr/bin/groovy
@Library('github.com/fabric8io/fabric8-pipeline-library@master')


// lets allow the VERSION_PREFIX to be specified as a parameter to the build
// but if not lets just default to 1.0
def versionPrefix = ""
try {
  versionPrefix = VERSION_PREFIX
} catch (Throwable e) {
  versionPrefix = "1.0"
}


def canaryVersion = "${versionPrefix}.${env.BUILD_NUMBER}"
//def label = "buildpod.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
def label = "master"




podTemplate(label: label, serviceAccount: 'jenkins', containers: [
        [name: 'maven', image: 'fabric8/maven-builder', command: 'cat', ttyEnabled: true, envVars: [
                [key: 'MAVEN_OPTS', value:'-Duser.home=/home/jenkins/'],
                [key: 'DOCKER_CONFIG', value:'/home/jenkins/.docker/'],
                [key: 'KUBERNETES_MASTER', value:  'kubernetes.default'] ]],
        [name: 'jnlp', image: 'iocanel/jenkins-jnlp-client:latest', command:'/usr/local/bin/start.sh', args: '${computer.jnlpmac} ${computer.name}', ttyEnabled: false,
                envVars: [[key: 'DOCKER_HOST', value: 'unix:/var/run/docker.sock']]]],
        volumes: [
                [$class: 'PersistentVolumeClaim', mountPath: '/home/jenkins/.mvnrepository', claimName: 'jenkins-mvn-local-repo'],
                [$class: 'SecretVolume', mountPath: '/home/jenkins/.m2/', secretName: 'jenkins-maven-settings'],
                [$class: 'SecretVolume', mountPath: '/home/jenkins/.docker', secretName: 'jenkins-docker-cfg'],
                [$class: 'HostPathVolume', mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock']
        ]) {


  node(label) {
    git = checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'f5cbf2f9-0028-42e5-8fc5-1e0c60240992', url: 'https://gitlab.com/myrepo/myservice.git']]])


    echo 'NOTE: running pipelines for the first time will take longer as build and base docker images are pulled onto the node'
    container(name: 'maven') {


      stage 'Build Release'
      mavenCanaryRelease {
        version = canaryVersion
      }
     }
   
  }
}

[James Rawlings]

from your kubernetes events it looks like its a PV problem.

Unable to mount volumes for pod

FWIW I've seen problems like this in the past when I was on older versions of kubernetes, there were quite a few PV problems.  I cant remember what version that was but 1.3 is a little old so maybe related. Unfortunately I dont know the best way to debug that, perhaps check the kubelet logs on the node that its failing?

I've a feeling secrets and volumes get mounted in the same way so I'm not sure if this is going to help but you could try and remove the maven repo volume in the Jenkinsfile, i.e. remove this line..

[$class: 'PersistentVolumeClaim', mountPath: '/home/jenkins/.mvnrepository', claimName: 'jenkins-mvn-local-repo'],

and see if that helps.  But still my gut feel is its a kubernetes / PV issues.

--
James Rawlings
Engineering
Red Hat, Inc.
Blog: https://medium.com/@jdrawlings
Twitter: @jdrawlings

https://fabric8.io/
https://blog.fabric8.io/

--
You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kurt T Stam]

Yes, and if you this is an older version of gofabric8 or an existing opnshift installation you may not have run the PV deployment all together, which is part of the gofabric8 installation now.

[kurr...@gmail.com]

Thanks James 

[$class: 'PersistentVolumeClaim', mountPath: '/home/jenkins/.mvnrepository', claimName: 'jenkins-mvn-local-repo'],

Where / how do I remove the above line?

 

[Anton]

Hi Kurt

On Tue, Jan 10, 2017 at 11:19 PM, Kurt T Stam <kurt...@gmail.com> wrote:

Yes, and if you this is an older version of gofabric8 or an existing opnshift installation you may not have run the PV deployment all together, which is part of the gofabric8 installation now.

Can you please clarify?

Are you recommending that I run PV deployment using gofabric8?

If yes, how do I do this? 

[kurr...@gmail.com]

I have tried running

gofabric8 volumes

But, alas, no chhange.

Still getting

java.lang.IllegalArgumentException: Container with name:[maven] not found in pod:[jenkins-19-kkjf6]

On Wednesday, January 11, 2017 at 11:20:17 PM UTC+1, Anton wrote:

Hi Kurt

[James Rawlings]

Sorry I should have explained that.  You can manually edit a deployment using `kubectl edit deployment foo` which opens up a vi editor or sort, here you can edit the yaml and when you :wq a new pod is automatically created.

Having said that this issue was fixed https://github.com/fabric8io/gofabric8/issues/325 and released last night to the fabric8-platform 2.4.18 should work fine.

[kurr...@gmail.com]

Thanks James

But, when I go to see what deployments I have, I get the following:

$ kubectl get deployments

No resources found.

Does this make sense? Yet when I log into Fabric8 I see:

[Anton]

Furthermore, when I try using Jenkins:

kubectl edit deployment jenkins

Error from server (NotFound): deployments.extensions "jenkins" not found

[Anton]

I tried running one of the OpenShift examples, the drools decision server.

After it pulled down all the maven jars, and and built, the last log output was:

Pushing image 172.30.164.161:5000/tmp/jboss-decisionserver63-o:latest ...

Registry server Address: 

Registry server User Name: serviceaccount

Registry server Email: service...@example.org

Registry server Password: <<non-empty>>

error: build error: Failed to push image: unauthorized: authentication required

[kurr...@gmail.com]

I discovered the /etc/sysconfig/docker was not correct.

I added INSECURE_REGISTRY='--insecure-registry 172.30.0.0/16'

Now when I run the same jenkins job it errors with:

Caused by: java.net.UnknownHostException: kubernetes.default: Name does not resolve
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at okhttp3.Dns$1.lookup(Dns.java:39)

Anyone know how to solve this?