Sporadic MSBuild errors / possible durable task issue?

[Steven Foster]

After updating Jenkins with plugins from early April 2018, I'm seeing inconsistent failures in my pipelines' calls to MSBuild which I cannot reproduce outside of Jenkins. It occurs in both powershell and bat steps.

The error presents as 

 

0>MSBUILD : error MSB4166: Child node "14" exited prematurely. Shutting down. Diagnostic information may be found in files in the temporary files directory named MSBuild_*.failure.txt.

There is no such extra information file. This looks a bit like an interrupt to me, is there a way I can reproduce how Jenkins sends interrupts so I can verify that?

Are there any advanced durable task debugging techniques I could use to find out what could be causing an interrupt or something else?

[Simon Richter]

Hi,

On 31.01.19 17:31, Steven Foster wrote:

> There is no such extra information file. This looks a bit like an
> interrupt to me, is there a way I can reproduce how Jenkins sends
> interrupts so I can verify that?

There are a few things going on:

- MSBuild sticks around after a build if the same project is rebuilt
- Jenkins kills processes belonging to a build after it believes the
build has finished
- MSBuild has serious issues with memory management

All of these are somewhat unpredictable.

I've found that it becomes a bit better if you set an environment variable

MSBUILDDISABLENODEREUSE=1

as suggested[1] by the VS team, disable the Jenkins Process Tree
Killer[2] and try to use the 64 bit version of MSBuild[3] to avoid
running out of virtual memory (which MSBuild sometimes, but not always
reports as the reason for termination in the log files in %APPDATA%\Temp)

Simon

[1]
https://developercommunity.visualstudio.com/content/problem/301840/msbuild-not-terminating.html

[2] https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller

[3] https://gitlab.kitware.com/cmake/cmake/issues/18219

[Steven Foster]

On Thursday, January 31, 2019 at 4:45:40 PM UTC, Simon Richter wrote:

There are a few things going on:

 - MSBuild sticks around after a build if the same project is rebuilt
 - Jenkins kills processes belonging to a build after it believes the
build has finished
 - MSBuild has serious issues with memory management

All of these are somewhat unpredictable.

I've found that it becomes a bit better if you set an environment variable

    MSBUILDDISABLENODEREUSE=1

as suggested[1] by the VS team, disable the Jenkins Process Tree
Killer[2] and try to use the 64 bit version of MSBuild[3] to avoid
running out of virtual memory (which MSBuild sometimes, but not always
reports as the reason for termination in the log files in %APPDATA%\Temp)

Thanks for the leads, I will look into those. Just to note that the issue only appeared after updating Jenkins (identical build agent images) and the builds were running fine before.

[Daniel Becroft]

We had a similar issue, and it ended up being related to MSBuild itself, not Jenkins. The jobs seemed to work correctly when run from a command line, but it took a while and they eventually started failing as well.

Our issue was due to the maximum number of temp files in %TEMP% being exceeded (see the GetTempFileNameA[1] function, it has an internal limit of 65,535). Once we cleared those out (and resolved the bug in our code that wasn't cleaning them up), the builds worked again.

[1] - https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-gettempfilenamea

[Steven Foster]

On Friday, February 1, 2019 at 12:04:26 AM UTC, Daniel Becroft wrote:

We had a similar issue, and it ended up being related to MSBuild itself, not Jenkins. The jobs seemed to work correctly when run from a command line, but it took a while and they eventually started failing as well.

Our issue was due to the maximum number of temp files in %TEMP% being exceeded (see the GetTempFileNameA[1] function, it has an internal limit of 65,535). Once we cleared those out (and resolved the bug in our code that wasn't cleaning them up), the builds worked again.

[1] - https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-gettempfilenamea

That sounds like a weird one. I had a look at %TEMP% and there were only about 50 or so files at a time there. The build agents are on-demand so they usually only run a handful of builds before being torn down.