[JIRA] (JENKINS-55024) Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

43 views
Skip to first unread message

marco.andries@itextpdf.com (JIRA)

unread,
Dec 5, 2018, 4:11:01 AM12/5/18
to jenkinsc...@googlegroups.com
Marco Andries created an issue
 
Jenkins / Bug JENKINS-55024
Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs
Issue Type: Bug Bug
Assignee: Unassigned
Attachments: empty_console.png, stacktrace.log
Components: workflow-job-plugin
Created: 2018-12-05 09:10
Environment: Jenkins version 2.138.3 running on Linux

Labels: plugin pipeline
Priority: Major Major
Reporter: Marco Andries

When upgrading to version 2.29 of the "Pipeline: Job" plugin, users had a completely empty console output in case of an error.

Downgrading to version 2.28 fixed the issue.

When investigating we found an error log that a log file could not be located:

java.io.FileNotFoundException: /var/lib/jenkins/jobs/iText-merge-pipeline/builds/319/log (No such file or directory)

See attachments for screenshots of the console output and the log.

If I read the change log for version 2.29 

  • JENKINS-54128: Change the implementation of WorkflowRun#getLogFile to avoid creating a new temporary file each time the method is called.
  • Fix: Do not call WorkflowRun#getLogFile when an error occurs while opening the log file for a Pipeline to avoid logging an additional stack trace

My guess would be that WorkflowRun#getLogFile is also not called when an error is thrown in a pipeline script.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

dnusbaum@cloudbees.com (JIRA)

unread,
Dec 14, 2018, 2:52:02 PM12/14/18
to jenkinsc...@googlegroups.com
Devin Nusbaum commented on Bug JENKINS-55024
 
Re: Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

I don't think the changes in workflow-job 2.29 and workflow-api 2.32 related to WorkflowRun#getLogFile are related to an empty console view, since the WorkflowRun#getLogFile method is not actually used when displaying logs.

The stack trace you posted shows that the call to new RandomAccessFile here threw a FileNotFoundException when trying to open the logs for a step because the log file did not exist. What does the build folder for the build look like? Does the log file exist? Are there any other errors in the Jenkins logs that seem relevant? It seems like something else might have deleted that file, but I am not sure what could have caused it. Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed? CC Jesse Glick in case this scenario rings a bell to him.

vivek.pandey@gmail.com (JIRA)

unread,
Dec 18, 2018, 10:17:02 AM12/18/18
to jenkinsc...@googlegroups.com
Vivek Pandey updated an issue
 
Change By: Vivek Pandey
Labels: pipeline pipeline-triaged plugin

j.beyer@stoneball.de (JIRA)

unread,
Jan 9, 2019, 2:22:02 AM1/9/19
to jenkinsc...@googlegroups.com
Jens Beyer commented on Bug JENKINS-55024
 
Re: Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

The new workflow-job 2.31 (compared to 2.25) does not write the log files for the individual steps anymore. At least on my system.

For new builds started after the upgrade, the individual build step logs seem to work without the individual build step log files.

But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs.

j.beyer@stoneball.de (JIRA)

unread,
Jan 9, 2019, 2:58:03 AM1/9/19
to jenkinsc...@googlegroups.com
Jens Beyer edited a comment on Bug JENKINS-55024
Regarding the exception:


The new workflow-job 2.31 (compared to 2.25) does not write the log files for the individual steps anymore. At least on my system.

For new builds started after the upgrade, the individual build step logs seem to work without the individual build step log files.

But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs.

 

Regarding the empty console: I opened JENKINS_55465 JENKINS-55465 which may also be the reason for your issue.

j.beyer@stoneball.de (JIRA)

unread,
Jan 9, 2019, 2:58:03 AM1/9/19
to jenkinsc...@googlegroups.com
Jens Beyer edited a comment on Bug JENKINS-55024
Regarding the exception:

 The new workflow-job 2.31 (compared to 2.25) does not write the log files for the individual steps anymore. At least on my system.

For new builds started after the upgrade, the individual build step logs seem to work without the individual build step log files.

But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs.


 

Regarding the empty console: I opened JENKINS_55465 which may also be the reason for your issue.

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 9, 2019, 10:03:02 AM1/9/19
to jenkinsc...@googlegroups.com

But for builds finished before the upgrade, after the upgrade is done, you are unable to get the full build step logs, and the mentioned exception occurs.

For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing? I am not quite sure what you mean by "full build step logs".

j.beyer@stoneball.de (JIRA)

unread,
Jan 10, 2019, 1:49:02 AM1/10/19
to jenkinsc...@googlegroups.com

Devin Nusbaum Sorry I just realized that my issue with the build step logs (the first one I mentioned) was probably also caused by the full console issue with compressedBuildLog (JENKINS-55465, the second one I mentioned). Please disregard.

jglick@cloudbees.com (JIRA)

unread,
Jan 11, 2019, 5:58:02 PM1/11/19
to jenkinsc...@googlegroups.com

After the back and forth here, I am unclear on what the status of this is. Do we have a reproducible bug?

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 14, 2019, 9:18:05 AM1/14/19
to jenkinsc...@googlegroups.com

Jesse Glick Nothing reproducible that I am aware of other than the duplicate related to compressBuildLog. Marco Andries are you still seeing this issue? If so, can you answer the questions in my earlier comment and perhaps attach the Pipeline script that is failing so we can see what it is doing? I am going to close this as incomplete for now, but please feel free to reopen it if you are able to provide additional data.

dnusbaum@cloudbees.com (JIRA)

unread,
Jan 14, 2019, 9:18:05 AM1/14/19
to jenkinsc...@googlegroups.com
Devin Nusbaum resolved as Incomplete
 
Change By: Devin Nusbaum
Status: Open Resolved
Resolution: Incomplete

amedee.vangasse@itextpdf.com (JIRA)

unread,
May 17, 2019, 3:38:02 AM5/17/19
to jenkinsc...@googlegroups.com
Amedee Van Gasse commented on Bug JENKINS-55024
 
Re: Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

Hi! I am a colleague of Marco. Looks like we abandoned this issue, I am sorry about that. Other fish to fry at the time because we had a workaround.

The issue has resurfaced because the Warnings Next Generations plugin is dependent on a Pipeline Job plugin > 2.28.

I will be taking over this issue and answering all your questions about additional information until we get to a solution.

amedee.vangasse@itextpdf.com (JIRA)

unread,
May 17, 2019, 4:01:04 AM5/17/19
to jenkinsc...@googlegroups.com
Amedee Van Gasse edited a comment on Bug JENKINS-55024
Hi! I am a colleague of Marco. Looks like we abandoned this issue, I am sorry about that. Other fish to fry at the time because we had a workaround.

The issue has resurfaced because the Warnings Next Generations plugin is dependent on a Pipeline Job plugin > 2.28.

I will be taking over this issue and answering all your questions about additional information until we get to a solution.


I previously wasn't involved with reporting this issue so it is possible that I will be asking things that have already been asked before, please bear with me.

amedee.vangasse@itextpdf.com (JIRA)

unread,
May 17, 2019, 4:29:06 AM5/17/19
to jenkinsc...@googlegroups.com
Amedee Van Gasse updated an issue
 
Change By: Amedee Van Gasse
Attachment: test-pipeline-Jenkinsfile

amedee.vangasse@itextpdf.com (JIRA)

unread,
May 17, 2019, 4:29:06 AM5/17/19
to jenkinsc...@googlegroups.com
Amedee Van Gasse commented on Bug JENKINS-55024
 
Re: Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

There were several questions for information. I'll try to bring some structure in them, and then answer them one by one.

    1. What does the build folder for the build look like?
    2. Does the log file exist?
    3. Are there any other errors in the Jenkins logs that seem relevant?
    1. Do you know if the build that was broken was building when Jenkins was restarted for the plugin update, or had it already completed?
    1. For builds before the upgrade, the full build log should work ok (some markup will not be displayed correctly), but the step logs are expected to be broken. Is that what you are seeing?
    2. I am not quite sure what you mean by "full build step logs".
    1. Do we have a reproducible bug?
    1. Marco Andries are you still seeing this issue?
    1. Can you answer the questions in my earlier comment?
    2. Can you attach the Pipeline script that is failing so we can see what it is doing?

    Question 1

    I am unsure what kind of answer you want:

    • a screenshot of the build folder?
    • a directory listing of the build folder?
    • a tarball of the build folder?

    I don't have ssh access to the server that runs Jenkins, and I don't know what you want, so I will take a brute force approach. Difficult also works. I am going to make a job that runs ls -ld $(find /var/lib/jenkins). Please let me know what exactly you need, because I guess that the directory listing may also contain confidential information.

    Question 2

    How can I find out if the log file exists?

    Question 3

    Which Jenkins logs exactly? I'd be happy to provide you the information if I know where to look.

    Question 4

    This is an ambiguous question, the literal answer would be "I know", but that would not be helpful. I will instead answer with steps to reproduce:

    1. Pipeline Job Plugin is at version 1.28.
    2. Start a failing job, job 1, and let it finish.
    3. Look at the console output of job 1 -> there is console output for job 1.
    4. Upgrade Pipeline Job Plugin to > 1.28.
    5. Restart Jenkins.
    6. Look at console output of job 1 -> there is console output for job 1.
    7. Start a failing job, job 2, and let it finish.
    8. Look at the console output of job 2 -> the console output for job 2 is empty.
    9. Downgrade Pipeline Job Plugin to 1.28.
    10. Restart Jenkins.
    11. Look at the console output of job 1 -> there is console output for job 1.
    12. Look at the console output of job 2 -> the console output for job 2 is empty.
    13. Start a failing job, job 3, and let it finish.
    14. Look at the console output of job 3 -> there is console output for job 3.

    Question 5

    For builds from before the update, there is indeed a console log, as described in my answer to question 4.

    I do not know about markup, is that important or relevant, and if so, how can I verify this?

    The step logs are expected to be broken.

    -> I don't understand what this means, could you please explain?

    Question 6

    Is this a question for Marco Andries or for Jens Beyer?

    Question 7

    Is this a question for Marco Andries or for Jens Beyer/Devin Nusbaum?

    Question 8

    Yes, we are still seeing this issue.

    Question 9

    I am collaborating with Marco Andries in providing you the answers.

    Question 10

    test-pipeline-Jenkinsfile

    amedee.vangasse@itextpdf.com (JIRA)

    unread,
    May 17, 2019, 4:29:06 AM5/17/19
    to jenkinsc...@googlegroups.com
    Change By: Amedee Van Gasse
    Resolution: Incomplete
    Status: Resolved In Review

    amedee.vangasse@itextpdf.com (JIRA)

    unread,
    May 17, 2019, 4:37:01 AM5/17/19
    to jenkinsc...@googlegroups.com
     
    Re: Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

    Meanwhile the job to make a directory listing, mentioned under question 1, had failed 

    line 2: /usr/bin/ls: Argument list too long

    I should have known... I'll do it again, this time with

    find . -print0 | xargs -0 ls -ld

    amedee.vangasse@itextpdf.com (JIRA)

    unread,
    May 17, 2019, 4:39:01 AM5/17/19
    to jenkinsc...@googlegroups.com
    Amedee Van Gasse edited a comment on Bug JENKINS-55024
    Meanwhile the job to make a directory listing, mentioned under question 1, had failed 
    {code}line 2: /usr/bin/ls: Argument list too long{code}


    I should have known... I'll do it again, this time with
    {code}find . /var/lib/jenkins -print0 | xargs -0 ls -ld{code}

    amedee.vangasse@itextpdf.com (JIRA)

    unread,
    May 22, 2019, 7:09:02 AM5/22/19
    to jenkinsc...@googlegroups.com

    The resulting log file is 310 MiB, with >2M lines. I need some way to narrow this down. Suggestions are welcome!

    amedee.vangasse@itextpdf.com (JIRA)

    unread,
    May 22, 2019, 7:26:01 AM5/22/19
    to jenkinsc...@googlegroups.com
    Amedee Van Gasse updated an issue
     
    Change By: Amedee Van Gasse
    Attachment: consoleText-iTextMergePipeline.txt.xz

    amedee.vangasse@itextpdf.com (JIRA)

    unread,
    May 22, 2019, 7:26:02 AM5/22/19
    to jenkinsc...@googlegroups.com
     
    Re: Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

    I grepped on "iText-merge-pipeline", which is the name of the job, and the result was only 3 MiB.
    After compression with xz, it's only 88 KiB. I have attached the file to this issue. Please let me know if you need more.
    consoleText-iTextMergePipeline.txt.xz

    amedee.vangasse@itextpdf.com (JIRA)

    unread,
    May 22, 2019, 7:29:02 AM5/22/19
    to jenkinsc...@googlegroups.com

    Devin Nusbaum Jesse Glick Jens Beyer I think that I answered all the questions you asked Marco Andries, can you have another look at this issue? Thanks!

    marco.andries@itextpdf.com (JIRA)

    unread,
    May 22, 2019, 7:46:03 AM5/22/19
    to jenkinsc...@googlegroups.com

    Amedee Van Gasse, yes I believe you answered all questions.

    jglick@cloudbees.com (JIRA)

    unread,
    Jun 3, 2019, 3:31:04 PM6/3/19
    to jenkinsc...@googlegroups.com
    Change By: Jesse Glick
    Status: In Review Progress

    jglick@cloudbees.com (JIRA)

    unread,
    Jun 3, 2019, 3:31:04 PM6/3/19
    to jenkinsc...@googlegroups.com
    Jesse Glick stopped work on Bug JENKINS-55024
     
    Change By: Jesse Glick
    Status: In Progress Open

    jglick@cloudbees.com (JIRA)

    unread,
    Jun 3, 2019, 3:32:01 PM6/3/19
    to jenkinsc...@googlegroups.com
    Jesse Glick updated an issue
    Change By: Jesse Glick
    Labels: pipeline pipeline-triaged plugin

    marco.andries@itextpdf.com (JIRA)

    unread,
    Jun 5, 2019, 8:56:04 AM6/5/19
    to jenkinsc...@googlegroups.com
    Marco Andries updated an issue
    Change By: Marco Andries
    Attachment: image-2019-06-05-14-55-49-292.png

    marco.andries@itextpdf.com (JIRA)

    unread,
    Jun 5, 2019, 8:57:03 AM6/5/19
    to jenkinsc...@googlegroups.com
    Marco Andries updated an issue
    Change By: Marco Andries
    Attachment: image-2019-06-05-14-56-21-611.png

    marco.andries@itextpdf.com (JIRA)

    unread,
    Jun 5, 2019, 9:10:04 AM6/5/19
    to jenkinsc...@googlegroups.com
    Marco Andries updated an issue
    Change By: Marco Andries
    Attachment: image-2019-06-05-15-09-19-247.png

    marco.andries@itextpdf.com (JIRA)

    unread,
    Jun 5, 2019, 9:11:03 AM6/5/19
    to jenkinsc...@googlegroups.com
     
    Re: Pipeline Job Plugin (workflow-job-plugin) gives empty console output when an error occurs

    Due to circumstances we needed to update the plugin and now we are experiencing that the console output is empty when the job is completed not only for failed jobs but also for aborted and successful jobs.

    I think to have found the issue we're experiencing.....

    I saw the following when I look at the logs of a stage:

    When I check that location, I find the following:

    + ls -l /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3
    total 288
    drwxr-xr-x 3 jenkins jenkins 16 Jun 5 14:42 archive
    rw-rr- 1 jenkins jenkins 54829 Jun 5 14:43 build.xml
    rw-rr- 1 jenkins jenkins 0 Jun 5 14:42 changelog0.xml
    rw-rr- 1 jenkins jenkins 39174 Jun 5 14:43 log.gz
    rw-rr- 1 jenkins jenkins 329 Jun 5 14:43 log-index
    rw-rr- 1 jenkins jenkins 58 Jun 5 14:43 open-tasks-fixed.xml
    rw-rr- 1 jenkins jenkins 171277 Jun 5 14:43 open-tasks.xml
    drwxr-xr-x 2 jenkins jenkins 8192 Jun 5 14:43 workflow
    drwxr-xr-x 2 jenkins jenkins 4096 Jun 5 14:42 workspace-files

    The log file is indeed not there but there's a log.gz file.

    I extracted the file running the following command: gzip -d /var/lib/jenkins/jobs/iText_7_Java/jobs/itextcore/branches/feature-QA-1892-3.0d3g52/builds/3/log.gz

    And now the console output is no longer empty and the stage log is also correct:

    So it seems there's an issue with packing and unpacking the log.

    Hope this helps....

    dnusbaum@cloudbees.com (JIRA)

    unread,
    Jun 5, 2019, 9:15:03 AM6/5/19
    to jenkinsc...@googlegroups.com

    Ok, looks like a duplicated of JENKINS-54678. Compressed build logs are not supported by new versions of workflow-job. If your pipelines use compressBuildLog, you should remove that from their Jenkinsfile for now.

    dnusbaum@cloudbees.com (JIRA)

    unread,
    Jun 5, 2019, 9:16:04 AM6/5/19
    to jenkinsc...@googlegroups.com
    Devin Nusbaum closed an issue as Duplicate
     
    Change By: Devin Nusbaum
    Status: Open Closed
    Resolution: Duplicate

    dnusbaum@cloudbees.com (JIRA)

    unread,
    Jun 5, 2019, 9:16:04 AM6/5/19
    to jenkinsc...@googlegroups.com
    Devin Nusbaum edited a comment on Bug JENKINS-55024
    Ok, looks like a duplicated duplicate of JENKINS-54678. Compressed build logs are not supported by new versions of workflow-job. If your pipelines use {{compressBuildLog}}, you should remove that from their Jenkinsfile for now. If the Compress Build Logs plugin has some global option that causes all build logs to be compressed, then you'll have to turn that off as well.

    marco.andries@itextpdf.com (JIRA)

    unread,
    Jun 5, 2019, 9:19:03 AM6/5/19
    to jenkinsc...@googlegroups.com

    Devin Nusbaum, thanks for the reply. I will have a look.

    marco.andries@itextpdf.com (JIRA)

    unread,
    Jun 5, 2019, 10:18:03 AM6/5/19
    to jenkinsc...@googlegroups.com

    Devin Nusbaum, it seems to solve the issue with the empty console outputs.

    No idea why we first only had it when an error was thrown.

    But now with the plugins completely updated, the console output is there for failed, successful and aborted jobs when "Compress Build Logs" is turned off.

    Thanks!
    Reply all
    Reply to author
    Forward
    0 new messages