[JIRA] (JENKINS-40294) Exec Failure: HTTP:404. Message:Not Found

227 views
Skip to first unread message

dserodio@gmail.com (JIRA)

unread,
Dec 7, 2016, 3:52:01 PM12/7/16
to jenkinsc...@googlegroups.com
Daniel Serodio created an issue
 
Jenkins / Bug JENKINS-40294
Exec Failure: HTTP:404. Message:Not Found
Issue Type: Bug Bug
Assignee: Carlos Sanchez
Attachments: jenkins-kubernetes.log
Components: kubernetes-plugin
Created: 2016/Dec/07 8:51 PM
Environment: Jenkins 2.35, Kubernetes Plugin 0.9, Minikube 0.13.1
Labels: kubernetes
Priority: Minor Minor
Reporter: Daniel Serodio

I'm trying to run a simple pipeline based on the example on the README:

podTemplate(label: 'mypod', containers: [
    containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'jnlp', image: 'jenkinsci/jnlp-slave', args: '${computer.jnlpmac} ${computer.name}'),
]) {

    node ('mypod') {
        stage 'Get a Maven project'
        git 'https://github.com/jenkinsci/kubernetes-plugin.git'
        container('maven') {
            stage 'Build a Maven project'
            sh 'mvn clean install'
        }
    }
}

The pod is created and started, but when Jenkins tries to run the shell script for the build, I get:

[kubernetes-plugin-pipeline-example] Running shell script
Executing shell script inside container [maven] of pod [kubernetes-f914ee7f49c44f7eaa182af5fda74c93-60beb1f2a295]
java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
	at com.squareup.okhttp.ws.WebSocketCall.createWebSocket(WebSocketCall.java:123)
	at com.squareup.okhttp.ws.WebSocketCall.access$000(WebSocketCall.java:40)
	at com.squareup.okhttp.ws.WebSocketCall$1.onResponse(WebSocketCall.java:98)
	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:177)
	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

The relevant section from Jenkins' log is attached. I increased the log verbosity of io.fabric8.kubernetes, org.csanchez.jenkins.plugins.kubernetes, com.squareup.okhttp and org.apache.http to ALL but it doesn't show which HTTP request got a 404 response.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

jean.neveux@gmail.com (JIRA)

unread,
Dec 10, 2016, 4:09:02 PM12/10/16
to jenkinsc...@googlegroups.com
Jean Neveux commented on Bug JENKINS-40294
 
Re: Exec Failure: HTTP:404. Message:Not Found

I'm having the same exception with Kubernet version:

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

And I'm also having another exception at the end of the script:

io.fabric8.kubernetes.client.KubernetesClientException: Not Found
	at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onFailure(ExecWebSocketListener.java:159)
	at com.squareup.okhttp.ws.WebSocketCall$1.onResponse(WebSocketCall.java:100)
	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:177)
	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

valorekhov@gmail.com (JIRA)

unread,
Dec 12, 2016, 8:57:01 PM12/12/16
to jenkinsc...@googlegroups.com

I'm having an exact same issue on a K8N cluster configured on bare metal. Tried configuring a custom logger on org.apache.http to capture the calls but it's not showing any entries. Were you able to figure out a solution to this problem?
Thanks!
Val

valorekhov@gmail.com (JIRA)

unread,
Dec 12, 2016, 9:11:01 PM12/12/16
to jenkinsc...@googlegroups.com
Val Orekhov updated an issue
 
Change By: Val Orekhov
Priority: Minor Critical

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 18, 2016, 2:28:01 PM12/18/16
to jenkinsc...@googlegroups.com
Carlos Sanchez commented on Bug JENKINS-40294
 
Re: Exec Failure: HTTP:404. Message:Not Found

are there any errors in the slave log? under /computer/xxx/log

jredl@vendasta.com (JIRA)

unread,
Dec 21, 2016, 6:57:01 PM12/21/16
to jenkinsc...@googlegroups.com

Carlos Sanchez I also am encountering this exact error. I pulled the following logs out of my GKE logs:

Dec 21, 2016 11:51:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Spot instances: 1
Dec 21, 2016 11:51:37 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Dec 21, 2016 11:51:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Created Pod: default-11ea535e917446
Dec 21, 2016 11:51:37 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
INFO: Waiting for Pod to be scheduled (0/100): default-11ea535e917446
Dec 21, 2016 11:51:39 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #21 from /10.156.3.229:38600
Dec 21, 2016 11:51:39 PM hudson.TcpSlaveAgentListener$ConnectionHandler error
WARNING: Connection #21 is aborted: Disabled protocol:Protocol:JNLP3-connect
Dec 21, 2016 11:51:39 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #22 from /10.156.3.229:38602
Dec 21, 2016 11:51:43 PM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:400. Message:Bad Request
java.net.ProtocolException: Expected HTTP 101 response but was '400 Bad Request'
	at com.squareup.okhttp.ws.WebSocketCall.createWebSocket(WebSocketCall.java:123)
	at com.squareup.okhttp.ws.WebSocketCall.access$000(WebSocketCall.java:40)
	at com.squareup.okhttp.ws.WebSocketCall$1.onResponse(WebSocketCall.java:98)
	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:177)
	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for slave default-11ea535e917446
Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminated Kubernetes instance for slave default-11ea535e917446
Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Disconnected computer default-11ea535e917446
Dec 21, 2016 11:51:43 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: vendasta/AA/jenkins-build #58 completed: FAILURE
Dec 21, 2016 11:51:43 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
SEVERE: Error in provisioning; slave=KubernetesSlave name: default-11ea535e917446, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@28a6c5c2
java.lang.IllegalStateException: Node was deleted, computer is null
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:556)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:482)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Dec 21, 2016 11:51:47 PM hudson.slaves.NodeProvisioner$2 run
WARNING: Provisioned agent Kubernetes Pod Template failed to launch
java.lang.IllegalStateException: Node was deleted, computer is null
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:556)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:482)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

jenkins-ci@carlossanchez.eu (JIRA)

unread,
Dec 22, 2016, 2:56:09 AM12/22/16
to jenkinsc...@googlegroups.com

as I said, is there anything in the slave logs in jenkins web ui under /computer/xxx/log ?

jenkins-ci@carlossanchez.eu (JIRA)

unread,
May 25, 2018, 5:22:02 AM5/25/18
to jenkinsc...@googlegroups.com
Carlos Sanchez resolved as Not A Defect
 

so your pipeline is probably getting a different pod template because you have one in the global config with the same label. And that template doesn't have the container you are trying to exec into

Change By: Carlos Sanchez
Status: Open Resolved
Resolution: Not A Defect
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

root@junwuhui.cn (JIRA)

unread,
Sep 18, 2019, 1:58:05 AM9/18/19
to jenkinsc...@googlegroups.com
runze xia reopened an issue
 

i also encountered the same problem

There is only one podtemplate with label xx in my jenkins.

This problem occurs when I start a large number of agents at the same time.

Belog is the agent log

Inbound agent connected from 10.233.103.249/10.233.103.249
Waiting for agent to connect (0/100): maven-dws8p
Waiting for agent to connect (1/100): maven-dws8p
Remoting version: 3.27
This is a Unix agent
Waiting for agent to connect (2/100): maven-dws8p
Waiting for agent to connect (3/100): maven-dws8p
Waiting for agent to connect (4/100): maven-dws8p
Waiting for agent to connect (5/100): maven-dws8p
Waiting for agent to connect (6/100): maven-dws8p
Waiting for agent to connect (7/100): maven-dws8p
Waiting for agent to connect (8/100): maven-dws8p
Waiting for agent to connect (9/100): maven-dws8p
Waiting for agent to connect (10/100): maven-dws8p
Waiting for agent to connect (11/100): maven-dws8p
Waiting for agent to connect (12/100): maven-dws8p
Waiting for agent to connect (13/100): maven-dws8p
Waiting for agent to connect (14/100): maven-dws8p
Waiting for agent to connect (15/100): maven-dws8p
Waiting for agent to connect (16/100): maven-dws8p
ERROR: Failed to monitor for Free Temp Space
java.util.concurrent.TimeoutException
	at hudson.remoting.Request$1.get(Request.java:316)
	at hudson.remoting.Request$1.get(Request.java:240)
	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:78)
	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:303)
ERROR: Failed to monitor for Free Disk Space
java.util.concurrent.TimeoutException
	at hudson.remoting.Request$1.get(Request.java:316)
	at hudson.remoting.Request$1.get(Request.java:240)
	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:78)
	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:303)
Agent successfully connected and online

Change By: runze xia
Resolution: Not A Defect
Status: Resolved Reopened
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

root@junwuhui.cn (JIRA)

unread,
Sep 18, 2019, 2:01:02 AM9/18/19
to jenkinsc...@googlegroups.com
runze xia edited a comment on Bug JENKINS-40294
 
Re: Exec Failure: HTTP:404. Message:Not Found
Jenkins v2.176.2 kubernetes plugin 1.15.10

i also encountered the same problem

There is only one podtemplate with label xx in my jenkins.

This problem occurs when I start a large number of agents at the same time.

Belog is the agent log
{code:java}
{code}

root@junwuhui.cn (JIRA)

unread,
Sep 18, 2019, 2:04:02 AM9/18/19
to jenkinsc...@googlegroups.com


At the same time, there is a very interesting behavior. Some of the steps in the container step are executed successfully, and some steps have a probability of failing.
The state of pod is Running. If a task is assigned to that agent again, it may succeed.

root@junwuhui.cn (JIRA)

unread,
Sep 18, 2019, 2:05:04 AM9/18/19
to jenkinsc...@googlegroups.com


!image-2019-09-18-14-04-48-431.png|thumbnail!
Reply all
Reply to author
Forward
0 new messages