Run out of idea to debug "JENKINS-37575 Jenkins node keeps sending the sames logs indefinitely"

[Quentin Dufour]

Hi everyone,

I don't know if this mailing list is appropriate for this question, but I'm currently encountering a bug with Jenkins on my production environment that I can't reproduce.

I don't know how to reproduce it, it doesn't occur every time. I can have 20 builds without any problem, and suddenly one build will fail, or more precisely hangs indefinitely.
It seems related to the network, as the slave always send the same 20-30 line of logs to the master indefinitely.
I don't know if the master asks to the slave to resend this logs or the slave is just sending this logs in an infinite loop, filling the master harddrive with the same 20-30 lines of log.
I've seen an InterruptedIOExcpetion when the slave try to write the logs in the OutputStream (of the master).

You can find more information here with many screenshots of Netbeans.
https://issues.jenkins-ci.org/browse/JENKINS-37575

I've tried many things, but I'm not sure of what I'm doing, as it's really difficult to debug network threads.
For example, there is a timeout of 10 seconds in the Durable Task Plugin. So, I don't know if the exception I caught is due to the real bug or this timeout.

The Jenkins Wiki is not really encouraging when it comes to Logging (not the job, but Jenkins here).
Unfortunately, there is no simple way in Jenkins to send log messages generated by Jenkins himself or by any plugins to the console, to a file or anywhere else. No matter what command line options you will use, you just won't succeed. It won't even help if you inject a java.util.logging configuration file directly into the JVM as Jenkins will override any settings in there programmatically.
https://wiki.jenkins-ci.org/display/JENKINS/Logging

I set all logs to finest in the Jenkins Interface, but didn't manage to see the logs from durable-task-plugin and workflow-durable-task-step-plugin.
I think that I did something wrong.

If you have some tips to help me, I would be really glad.
Every ideas, tips or spells are welcome

Thanks by advance,