Troubleshooting remoting related input/output stream piping issues

[Jerome Lacoste]

Hello,

I am trying to diagnose a few remoting related issues in the unity3d plugin (https://github.com/jenkinsci/unity3d-plugin/)

I am unable to reproduce the issues so far but would appreciate any insight into my analysis (and design of the plugin).

Plugin function and design:

The plugins basically executes a program (Unity3d) which happens to write its output into a file (hereafter called logFile). I pipe this file into the console as it is written. To achieve this remote piping I used the following design:

3 elements are created:

* a org.jenkinsci.plugins.unity3d.io.Pipe consisting of a PipedInputStream and a PipedOutputStream. This outputstream is wrapped into a RemoteOutputStream if the launcher is to be executed remotely.

* a long running Async Callable is passed to the launcher channel. This closure, a PipeFileAfterModificationAction instance, detects that the logFile is started being written and copies from it recursively into the pipe until interrupted. It then closes the output in a finally

* a org.jenkinsci.plugins.unity3d.io.StreamCopyThread reads from the pipe on the master and copies it into the job console.

The issues that some users are experiencing:

* the logFile isn't fully pasted into the console

* the pipe is broken:

java.io.IOException: Pipe broken

  at java.io.PipedInputStream.read(PipedInputStream.java:322)

  at java.io.PipedInputStream.read(PipedInputStream.java:378)

  at java.io.InputStream.read(InputStream.java:101)

  at org.jenkinsci.plugins.unity3d.io.StreamCopyThread.run(StreamCopyThread.java:64)

It seems like these errors

* only happen when used in distributed environmenrts

* when the second happens, the first one as well.

Analysis:

I looked into the "Pipe broken" issue and I believe something is fishy in my design.

Ideally I want to:

* wait for the Unity3d command to finish execution

* stop the thread reading the logFile on the remote side

* pipe everything back into the console

* close things gracefully

To achieve this, I do this in the finally block of the unity3d plugin execution:

    // wait for some time - see https://github.com/jenkinsci/unity3d-plugin/commit/d792376afd67579f313043e7c701cf2559df02bd

            Thread.sleep(1000);

            // stop the piping

            if (!futureReadBytes.isDone()) {

                futureReadBytes.cancel(true);

            }

            // wait for the copy 

            try {

                copierThread.join();

                if (copierThread.getFailure() != null) {

For the "Pipe broken" issue, my clean closing objective isn't reached. The StreamCopyThread fails while copying, i.e. it detects that the Thread that writes into the PipedInputStream (named "Computer.threadPoolForRemoting(...)" is dead (isAlive()==false) before the read() detects the eof (-1).

Why does it die, I am not sure, this thread is managed by the remoting framework.

Question #1: futureReadBytes.cancel(true)

This if my understanding is correct should send a Cancel command asynchronously to the remote end.

I expect this one to interrupt the reading of the PipeFileAfterModificationAction which would close the stream in its finally and thus terminate the async operation then the thread that reads into the pipe.

But could the cancel cause the local thread to die before the remote action is terminated properly ?

Question #2: lack of flushing could be the cause of the incomplete piping

Am I missing some flushing somewhere ? This could happen at lots of levels. I don't see much flushing in the jenkins StreamCopyTread either.

Question #3: better design

I believe it would be better if I didn't have a long running async Callable to pipe the logFile, but something I could signal to stop from the client side instead of having to use cancel(). Any tip on achieving this in a way that fits with the remoting library ?

Some ideas I've had:

* get rid of the PipeFileAfterModificationAction action by forcing the writing of the output into a newly created logFile and use a standard StreamCopyThread to pipe it. That way I don't have to detect the start of the operation. This would still require the synchronization of the stop of the command.

* create a wrapper script that uses OS capabilities to run the command and pipe the logFile to its standard output, letting the jenkins side of it execute a single command, and not caring about the piping. I know how to do this on Unix but not on Windows. Otherwise the script could be a Java wrapper, but this is a bit overkill it seems.

Thanks for your insights,

Cheers,

Jerome