occasional exception from junit call

40 views

Skip to first unread message

jeremy mordkoff

unread,

Aug 21, 2020, 12:28:40 PM8/21/20

to Jenkins Users

I use scripted pipelines with a library. The master and the slaves are running in ubuntu 16.04 docker containers.

About a week ago we started getting occasional failures from this call:

junit testResults: ".build/ub18_debug/artifacts/systemtest/*xml"

There are about 60 junit xml files in that directory.

Since then I have tried updating jenkins to 2.251 (including a container restart), which warned me about a place where a function marked

NONCPS was calling CPS code, so I fixed that and restarted all of the slaves. But the problem persists. I would estimate it happens 30%

of the time.

For now I have wrapped that call in a try block that sets the build result to UNSTABLE if the call fails. But not having the test results

available is messing up all of our reporting tooks. I also tried reordering some code thinking that maybe the previous step had caused the

problem but the problem stayed with the junit call.

I am willing to try just about anything to get past this issue. Any suggestions at all are welcome.

The Jenkins console for that job reports

[Pipeline] junit

Recording test results

java.io.IOException: Unexpected termination of the channel

[Pipeline] }

[Pipeline] // node

[Pipeline] }

[Pipeline] // stage

[Pipeline] End of Pipeline

java.io.EOFException

at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2353)

at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822)

at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)

at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)

at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)

at hudson.remoting.Command.readFrom(Command.java:140)

at hudson.remoting.Command.readFrom(Command.java:126)

at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)

at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)

Caused: java.io.IOException: Unexpected termination of the channel

at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)

Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to j7-ssh2

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)

at hudson.remoting.Request.call(Request.java:202)

at hudson.remoting.Channel.call(Channel.java:954)

at hudson.FilePath.act(FilePath.java:1069)

at hudson.FilePath.act(FilePath.java:1058)

at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)

at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)

at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)

at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)

at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)

at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused: hudson.remoting.RequestAbortedException

at hudson.remoting.Request.abort(Request.java:340)

at hudson.remoting.Channel.terminate(Channel.java:1038)

at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:94)

Finished: FAILURE

The slave's log reports

Aug 11, 2020 4:19:21 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn

WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/

Aug 12, 2020 7:55:32 AM hudson.slaves.ChannelPinger$1 onDead

INFO: Ping failed. Terminating the channel channel.

java.io.IOException: Broken pipe

at java.io.FileOutputStream.writeBytes(Native Method)

at java.io.FileOutputStream.write(FileOutputStream.java:313)

at hudson.remoting.StandardOutputStream.write(StandardOutputStream.java:83)

at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:89)

at hudson.remoting.ChunkedOutputStream.sendBreak(ChunkedOutputStream.java:62)

at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:46)

at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:46)

at hudson.remoting.Channel.send(Channel.java:766)

at hudson.remoting.Request.callAsync(Request.java:237)

at hudson.remoting.Channel.callAsync(Channel.java:1032)

at hudson.remoting.PingThread.ping(PingThread.java:110)

at hudson.remoting.PingThread.run(PingThread.java:90)

Aug 12, 2020 7:55:32 AM hudson.remoting.Request$2 run

WARNING: Failed to send back a reply to the request hudson.remoting.Request$2@1dd89b20

java.io.IOException: Broken pipe

and the master's logs include

Agent JVM has not reported exit code. Is it still running?

[08/12/20 03:53:02] [SSH] Connection closed.

ERROR: Connection terminated