"WARNING: Failed to construct Command" when dynamic slave is terminated

178 views
Skip to first unread message

Mark Petrovic

unread,
Jul 18, 2015, 9:38:22 PM7/18/15
to jenkins...@googlegroups.com
I am developing a cloud plugin that spins up nodes on demand to build projects inside Docker containers.

Everything is worked well, but I see these exceptions in the log when a node finishes its job and I _terminate it:

Jul 18, 2015 5:41:21 PM hudson.model.Run execute
INFO: boot-continuous-develop #15 main build action completed: SUCCESS
Jul 18, 2015 5:41:21 PM com.xoom.inf.jenkins.plugins.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for slave 25cf5bd9-1bb8-468a-bca3-faeba0a84bde
Jul 18, 2015 5:41:22 PM com.xoom.inf.jenkins.plugins.KubernetesSlave _terminate
INFO: Terminated Kubernetes instance for slave 25cf5bd9-1bb8-468a-bca3-faeba0a84bde
Jul 18, 2015 5:41:22 PM hudson.remoting.AbstractByteArrayCommandTransport$1 handle
WARNING: Failed to construct Command
java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2328)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:40)
        at hudson.remoting.AbstractByteArrayCommandTransport$1.handle(AbstractByteArrayCommandTransport.java:61)
        at org.jenkinsci.remoting.nio.NioChannelHub$2.run(NioChannelHub.java:594)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


What does this mean and how can I prevent this?  

Thank you.

Mark Petrovic

unread,
Jul 19, 2015, 11:10:52 PM7/19/15
to jenkins...@googlegroups.com
A little more information:  here is how the AbstractCloudSlave._terminate() method is implemented in my custom slave:

    @Override
    protected void _terminate(TaskListener listener) throws IOException, InterruptedException {
        LOGGER.log(Level.INFO, "Terminating Kubernetes instance for slave {0}", name);

        try {
            cloud.client().deletePod(name, KubernetesCloud.POD_NAMESPACE);
            LOGGER.log(Level.INFO, "Terminated Kubernetes instance for slave {0}", name);
            toComputer().disconnect(null);
        } catch (Exception e) {
            LOGGER.log(Level.SEVERE, "Failure to terminate instance for slave " + name, e);
        }
    }


I'm wondering if the toComputer().disconnect(null) is appropriate.  

Mark Petrovic

unread,
Jul 28, 2015, 7:00:10 PM7/28/15
to Jenkins Users, mspet...@gmail.com
And a bit more information...

Here is another stacktrace that occurs when jobs finish:

Jul 28, 2015 3:56:29 PM jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed

WARNING: Computer.threadPoolForRemoting [#75] for ac027ba0-74df-4d1d-a92c-1bca931c61d1 terminated

java.io.EOFException

at org.jenkinsci.remoting.nio.NioChannelHub$3.run(NioChannelHub.java:613)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)

at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)


I am working in a fork of the Jenkins Kubernetes plugin, which I should have said at the outset.  


https://github.com/jenkinsci/kubernetes-plugin


Beyond these exceptions, the plugin otherwise works fine.


I'm concerned that if I deploy this plugin to support a production build farm, these exceptions may indicate a condition in the master that will lead to memory leaks or some such.  iow, can I safely ignore these exceptions (which I really hate to do).

Reply all
Reply to author
Forward
0 new messages