[JIRA] (JENKINS-59311) Poor performance of pipeline build compared to freestyle

0 views
Skip to first unread message

ch.fetzer@gmx.net (JIRA)

unread,
Sep 11, 2019, 3:07:03 AM9/11/19
to jenkinsc...@googlegroups.com
Christoph Fetzer created an issue
 
Jenkins / Bug JENKINS-59311
Poor performance of pipeline build compared to freestyle
Issue Type: Bug Bug
Assignee: Unassigned
Components: pipeline
Created: 2019-09-11 07:06
Environment: jenkins ver 2.176.3
Priority: Major Major
Reporter: Christoph Fetzer

I have a bunch of freestyle jobs, that compile and build an installer for my software on two compilers and in debug/release. Each of them takes aroung half an hour.

Then I tried to switch the build to a pipeline build, integrated build of debug and release to one job and additionally integrated several other stuff (cppcheck,...). This build ended up in taking 3-4 hours for quite a while (which I did not expect and regard it as slow also).

Since some weeks now my pipeline job suddenly started taking 8 hours while the freestyle jobs duration staid the same.

Something about my environment: it's a jenkins-installation as service on a windows server 2016 machine. I am not aware of any changes on the machine installation or job configuration since the job got slower. It's not a single step in the pipeline that eats all the time. It's the overall performance and the time is distributed over all steps. E.g. the compile and build installer step (which is the entire job done by the freestyle job) alone takes 3 hours.

Unfortunately I don't have the build history long enough to be able to identify plugin-versions from before and after the buildtime-increase.

Please contact me in case I can help with any additional information.
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

ch.fetzer@gmx.net (JIRA)

unread,
Sep 12, 2019, 10:17:02 AM9/12/19
to jenkinsc...@googlegroups.com
Christoph Fetzer commented on Bug JENKINS-59311
 
Re: Poor performance of pipeline build compared to freestyle

I maybe got the point:

My build deep inside makes use of a compiler | tee | grep,... and therefore is splitting the output into a file at one point.

Not too far ago I switched from a formerly deprected warnings parser to the warnings next generation plugin and therefore had to insert another | tee - split. I'm not yet 100% sure if this exactly matches the point were buildtimes increased but at least it might be close to. I'll set up a test.

On the other hand this | tee provides a list of other problems and I'd like to avoid it at any level - also for warnings next generation. Is there another way to split build output to a file and in the same step keep the return code?

ch.fetzer@gmx.net (JIRA)

unread,
Sep 12, 2019, 10:19:02 AM9/12/19
to jenkinsc...@googlegroups.com
Christoph Fetzer edited a comment on Bug JENKINS-59311
I maybe got the point:

My build deep inside makes use of a compiler | tee | grep,... and therefore is splitting the output into a file at one point.

Not too far ago I switched from a formerly deprected warnings parser to the warnings next generation plugin and therefore had to insert another | tee - split. I'm not yet 100% sure if this exactly matches the point were buildtimes increased but at least it might be close to. I'll set up a test.
At least my way quicker freestyle build do not make use of the inner  | tee | grep!

On the other hand this | tee provides a list of other problems and I'd like to avoid it at any level - also for warnings next generation. Is there another way to split build output to a file and in the same step keep the return code?

ch.fetzer@gmx.net (JIRA)

unread,
Sep 12, 2019, 10:21:03 AM9/12/19
to jenkinsc...@googlegroups.com
Christoph Fetzer edited a comment on Bug JENKINS-59311
I maybe got the point:

My build deep inside makes use of a compiler | tee | grep,... and therefore is splitting the output into a file at one point.

Not too far ago I switched from a formerly deprected warnings parser to the warnings next generation plugin and therefore had to insert another | tee - split. I'm not yet 100% sure if this exactly matches the point were buildtimes increased but at least it might be close to. I'll set up a test.

At least my way quicker freestyle build do does not make use of the inner  | tee | grep!


On the other hand this | tee provides a list of other problems and I'd like to avoid it at any level - also for warnings next generation. Is there another way to split build output to a file and in the same step keep the return code?

ch.fetzer@gmx.net (JIRA)

unread,
Sep 13, 2019, 5:25:02 AM9/13/19
to jenkinsc...@googlegroups.com

In fact, it is the grep. But that's in there for a longer time. Is there some real relation between a grep (from msys) and some components of jenkins that might explain the significant decrease in the speed?

 

Second issue I observed with warnings-ng - for my tests I changed the use of recordIssued from:

recordIssues sourceCodeEncoding: 'ISO-8859-1',
         tools: [groovyScript(parserId: 'MSVC-Warning', pattern:'**/build-makedep.log', reportEncoding:'ISO-8859-1'),
                        groovyScript(parserId: 'MSVC-Error', pattern:'**/build-makedep.log', reportEncoding:'ISO-8859-1') ]

to

 recordIssues sourceCodeEncoding: 'ISO-8859-1',
                    tools: [groovyScript(parserId: 'MSVC-Warning', reportEncoding:'ISO-8859-1'),
                        groovyScript(parserId: 'MSVC-Error', reportEncoding:'ISO-8859-1') ]

(note the missing "pattern:") but that does not work as documented (scanning the console output without the need of an intermediate file).

ch.fetzer@gmx.net (JIRA)

unread,
Sep 13, 2019, 6:51:04 AM9/13/19
to jenkinsc...@googlegroups.com
Christoph Fetzer edited a comment on Bug JENKINS-59311
In fact, it is the grep. But that's in there for a longer time. Is there some real relation between a grep (from msys) and some components of jenkins that might explain the significant decrease in the speed?

 

Second issue I observed with warnings-ng - for my tests I changed the use of recordIssued from:
{code:java}

recordIssues sourceCodeEncoding: 'ISO-8859-1',
         tools: [groovyScript(parserId: 'MSVC-Warning', pattern:'**/build-makedep.log', reportEncoding:'ISO-8859-1'),
                     groovyScript(parserId: 'MSVC-Error', pattern:'**/build-makedep.log', reportEncoding:'ISO-8859-1') ]
{code}
to
{code:java}

recordIssues sourceCodeEncoding: 'ISO-8859-1',
                    tools: [groovyScript(parserId: 'MSVC-Warning', reportEncoding:'ISO-8859-1'),
                     groovyScript(parserId: 'MSVC-Error', reportEncoding:'ISO-8859-1') ]
{code}

(note the missing "pattern:") but that does not work as documented (scanning the console output without the need of an intermediate file).


 

Ecuse me, that's not an issue, that's documented - but ugly!

ch.fetzer@gmx.net (JIRA)

unread,
Sep 26, 2019, 4:39:04 AM9/26/19
to jenkinsc...@googlegroups.com

The problem seems to have disappeared after the latest LTS update to v2.190.1. At least at the first test build after that update.

ch.fetzer@gmx.net (JIRA)

unread,
Sep 27, 2019, 9:12:02 AM9/27/19
to jenkinsc...@googlegroups.com
Christoph Fetzer updated Bug JENKINS-59311
 

Was solved somewhere between V2.176.2 and v2.190.1.

Relation to any plugin or module not clear.

Solution is released.
Change By: Christoph Fetzer
Status: Open Fixed but Unreleased
Resolution: Fixed

ch.fetzer@gmx.net (JIRA)

unread,
Sep 27, 2019, 9:12:02 AM9/27/19
to jenkinsc...@googlegroups.com
Change By: Christoph Fetzer
Status: Fixed but Unreleased Resolved

ch.fetzer@gmx.net (JIRA)

unread,
Oct 10, 2019, 3:13:02 AM10/10/19
to jenkinsc...@googlegroups.com
Christoph Fetzer reopened an issue
 

This jenkins installation is driving me nuts - the slow behaviour came back.

The last quick run was on September, 27th. After that the build failed for some days due to issues with the code and the libraries used. The first completing build after that on Oct. 9th was slow again (same times - ~45min vs 3:40hours).

I implemented every updates in that time as soon as I realised them. There were a number of them in that time frame. Is there a log that shows the plugins updated and the version change? With that I could restore the state of September 27th and apply all the updates step by step until the build gets slower again.

 
Change By: Christoph Fetzer
Resolution: Fixed
Status: Resolved Reopened
Reply all
Reply to author
Forward
0 new messages