Real-time log parsing?

[Mandeville, Rob]

I’m running multi-hour test cycles and my users have a demand for real-time results.  If test #50 failed 45 minutes in, they want to be able to see it without waiting five hours for the rest of the tests to run.  We’ve had this problem for longer than we’ve had Jenkins, so our solution is to have a log parser separate from Jenkins “tail” the build log, reading it as it’s being written and writing test results to a database powering a non-Jenkins web site.  Said process is also reading other auxiliary files (far too many to turn into Jenkins artifacts), so it has to run on the host the slave node and the build are running on, not the machine hosting the Jenkins server.

 

The problem with this is that we have to make sure that everything gets appended to a log file on disk (so the parser can “tail” it) and to standard output (so that we can see it).  Accidentally opening the log file for “write” rather than “append” truncates the log and the parser gets lost.

Is there a way for a process on the slave node machine to “tail” the build log that Jenkins is getting?  Can Jenkins’ output log be replicated in real time in the workspace?  Can a groovy JAR be run asynchronously inside the slave as it’s building, and watch the bytes go by?  Any other ideas?

 

--Rob

 

The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.

[Gareth Bowles]

You can certainly tail the build log file while the build is in progress; it will be in $JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_NUMBER, where $BUILD_NUMBER is the active build.  I'm not sure of an easy way to get this number from the command line, you might just have to list the build subdirectories in date order and pick the one at the end.

You could probably do it with a system Groovy script as well; I'll take a quick look at that, so let me know if you'd like to pursue this option or if the command line is good enough.

[Michael Barbine]

https://github.com/mbarbine/JenkinsDB

[anthco...@gmail.com]

Hello Rob,

Did you ever come up with a satisfactory approach to  asynchronously processing the build-log?

cheers,

Anth