[JIRA] (JENKINS-46553) Maven surefire timeout treated as successful build

16 views
Skip to first unread message

f.modler@gmx.net (JIRA)

unread,
Sep 9, 2019, 7:32:03 AM9/9/19
to jenkinsc...@googlegroups.com
Falko Modler commented on Bug JENKINS-46553
 
Re: Maven surefire timeout treated as successful build

Same here. In a regular shell (locally, not in Jenkins) the error is manifested as a build failure:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project some-project: There was a timeout or other error in the fork -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project some-project: There was a timeout or other error in the fork
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoFailureException: There was a timeout or other error in the fork
        at org.apache.maven.plugin.surefire.SurefireHelper.throwException(SurefireHelper.java:240)
        at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:112)
        at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:354)
        at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1008)
        at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:854)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
        ... 20 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

f.modler@gmx.net (JIRA)

unread,
Sep 9, 2019, 7:41:02 AM9/9/19
to jenkinsc...@googlegroups.com
Falko Modler edited a comment on Bug JENKINS-46553
Same here. In a regular shell (locally, not in Jenkins) the error is manifested as a build failure:
{noformat}
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 57.572 s
[INFO] Finished at: 2019-09-09T13:36:55+02:00
[INFO] Final Memory: 50M/1024M
[INFO] ------------------------------------------------------------------------
{noformat}
That is with {{mvn -e ...}}
, without . Without it wou you won't see the stacktrace but the return code is 1 either way!

f.modler@gmx.net (JIRA)

unread,
Sep 9, 2019, 7:41:02 AM9/9/19
to jenkinsc...@googlegroups.com

That is with {{mvn -e ...}}, without it wou won't see the stacktrace but the return code is 1 either way!

f.modler@gmx.net (JIRA)

unread,
Sep 10, 2019, 7:30:03 AM9/10/19
to jenkinsc...@googlegroups.com

Plot twist: I don't think anymore that Jenkins is to blame, at least in my case:

I am (and probably most other Jenkins users are) setting -Dmaven.test.failure.ignore=true to get all test results instead of a failing build after the first module with test failures.
This is where https://issues.apache.org/jira/browse/SUREFIRE-953 comes into play!

What we would need is another Maven Surefire property/feature, that differentiates between regular test failures and timeouts.

For now I am using the following pipeline lib function which I just need to call in my pipelines (in some post block) without any parameters:

// Parses the console log file for known errors (that Jenkins does not properly recognize as build failures) and fails the build if any are found.
def call() {
    def known = [
        'There was a timeout or other error in the fork'    // https://issues.jenkins-ci.org/browse/JENKINS-46553
    ]
    def consoleLogFile = "${WORKSPACE}/../builds/${BUILD_NUMBER}/log"
    println "Checking for known errors in console log file that are not properly handled by Jenkins: ${consoleLogFile}"
    // grep the cosole log, notes:
    //   - "set +x" to avoid echoing the grep command which would lead to a false positive
    //   - "set +e" to avoid build failure due to non-zero exit code (otherwise "exit 0" wouldn't be reached)
    def foundErrors = sh(script: "set +xe && grep '${known.join('|')}' ${consoleLogFile}; exit 0", returnStdout: true)
    if (foundErrors != "") {
        error("Found known errors in the log (see previous outputs for more context):\n${foundErrors}")
    }
}

PS: Yes I know, directly "grepping" the log file is not good practise. I also haven't tested this with/on slaves.
I do also know that there are log parsing plugins but they do come with their own litte quirks, bugs and overhead...

f.modler@gmx.net (JIRA)

unread,
Sep 10, 2019, 7:30:05 AM9/10/19
to jenkinsc...@googlegroups.com
Falko Modler edited a comment on Bug JENKINS-46553
*Plot twist*: I don't think anymore that Jenkins is to blame, at least in my case:


I am (and probably most other Jenkins users are) setting {{-Dmaven.test.failure.ignore=true}} to get all test results instead of a failing build after the first module with test failures.
This is where https://issues.apache.org/jira/browse/SUREFIRE-953 comes into play!

What we would need is another Maven Surefire property/feature, that differentiates between regular test failures and timeouts.

For now I am using the following pipeline lib function which I just need to call in my pipelines (in some {{post}} block) without any parameters:
{code:groovy |title=failOnKnownConsoleLogPatterns.groovy }

// Parses the console log file for known errors (that Jenkins does not properly recognize as build failures) and fails the build if any are found.
def call() {
    def known = [
        'There was a timeout or other error in the fork'    // https://issues.jenkins-ci.org/browse/JENKINS-46553
    ]
    def consoleLogFile = "${WORKSPACE}/../builds/${BUILD_NUMBER}/log"
    println "Checking for known errors in console log file that are not properly handled by Jenkins: ${consoleLogFile}"
    // grep the cosole log, notes:
    //   - "set +x" to avoid echoing the grep command which would lead to a false positive
    //   - "set +e" to avoid build failure due to non-zero exit code (otherwise "exit 0" wouldn't be reached)
    def foundErrors = sh(script: "set +xe && grep '${known.join('|')}' ${consoleLogFile}; exit 0", returnStdout: true)
    if (foundErrors != "") {
        error("Found known errors in the log (see previous outputs for more context):\n${foundErrors}")
    }
}
{code}

PS: Yes I know, directly "grepping" the log file is not good practise. I also haven't tested this with/on slaves.
I do also know that there are log parsing plugins but they do come with their own litte quirks, bugs and overhead...

f.modler@gmx.net (JIRA)

unread,
Nov 27, 2019, 5:33:02 PM11/27/19
to jenkinsc...@googlegroups.com

What we would need is another Maven Surefire property/feature, that differentiates between regular test failures and timeouts.

I've just created a ticket for that: https://issues.apache.org/jira/browse/SUREFIRE-1728

tibordigana@apache.org (JIRA)

unread,
Dec 2, 2019, 9:07:06 AM12/2/19
to jenkinsc...@googlegroups.com

Mike Duigou

Falko Modler

We cannot modify the purpose of maven.test.failure.ignore .

Let's explain it: The test set will end up with Result Set (total, failures, errors, flakes, skipped) and the test frameworks (JUnit, TestNG, etc) provide RunListener sending the test events into the final Result Set. All the teminology you can see in maven.test.failure.ignore{{ }}talks abou failures and it means the Result Set. It has nothing to do with any timeouts and vice versa!

If we have to fix this without touching the TXT and XML reports, I am all the ear how.

My proposal is to introduce surefire.failIfTimeout=true and failsafe.failIfTimeout=true. It is complemetary principle to the parameter that we already have with failIfNoTests=true. So we control the behavior on fine grained level.

Setting maven.test.failure.ignore=true would not bypass the failIfNoTests=true and the build fails anyway with not having any tests in the project. The same should be with the timeouts. The only difference are the default values, where failIfNotests has false, and failIfTimeout would have true.

If the user wants to completely avoid build failure, run the cmd mvn test -Dmaven.test.failure.ignore=true -Dsurefire.failIfTimeout=false

 

If the user wants to avoid test failure(s) run the cmd mvn test -Dmaven.test.failure.ignore=true

 

Feel free to comment on this.

 

tibordigana@apache.org (JIRA)

unread,
Dec 2, 2019, 9:09:02 AM12/2/19
to jenkinsc...@googlegroups.com
Tibor Digana edited a comment on Bug JENKINS-46553
[~bondolo]

[~famod]

We cannot modify the purpose of *{{maven.test.failure.ignore}}* .

Let's explain it: The test set will end up with Result Set (total, failures, errors, flakes, skipped) and the test frameworks (JUnit, TestNG, etc) provide RunListener sending the test events into the final Result Set. All the teminology you can see in *{{maven.test.failure.ignore}}*{
\ { }}talks abou failures and it means the Result Set. It has nothing to do with any timeouts and vice versa!


If we have to fix this without touching the TXT and XML reports, I am all the ear how.

My proposal is to introduce *surefire.failIfTimeout=true* and *failsafe.failIfTimeout=true*. It is complemetary principle to the parameter that we already have with *failIfNoTests=true*. So we control the behavior on fine grained level.

Setting *maven.test.failure.ignore**=true* would not bypass the *failIfNoTests=true* and the build fails anyway with not having any tests in the project. The same should be with the timeouts. The only difference are the default values, where _failIfNotests has +false+_, and _failIfTimeout would have +true+_.

If the user wants to
completely _avoid +build+ failure due to the plugin ,_ run the cmd *mvn test -Dmaven.test.failure.ignore=true -Dsurefire.failIfTimeout=false*

 

If the user wants to _avoid +test+ failure(s)_ run the cmd *mvn test -Dmaven.test.failure.ignore=true*


 

Feel free to comment on this.

 

tibordigana@apache.org (JIRA)

unread,
Dec 2, 2019, 9:10:04 AM12/2/19
to jenkinsc...@googlegroups.com
Tibor Digana edited a comment on Bug JENKINS-46553
[~bondolo]

[~famod]

We cannot modify the purpose of *{{maven.test.failure.ignore}}* .

Let's explain it: The test set will end up with Result Set (total, failures, errors, flakes, skipped) and the test frameworks (JUnit, TestNG, etc) provide RunListener sending the test events into the final Result Set. All the teminology you can see in {{ *{{maven.test.failure.ignore}}* {\{ }}talks abou about the failures and it means the Result Set. It has nothing to do with any timeouts and vice versa!


If we have to fix this without touching the TXT and XML reports, I am all the ear how.

My proposal is to introduce *surefire.failIfTimeout=true* and *failsafe.failIfTimeout=true*. It is complemetary principle to the parameter that we already have with *failIfNoTests=true*. So we control the behavior on fine grained level.

Setting *maven.test.failure.ignore**=true* would not bypass the *failIfNoTests=true* and the build fails anyway with not having any tests in the project. The same should be with the timeouts. The only difference are the default values, where _failIfNotests has +false+_, and _failIfTimeout would have +true+_.

If the user wants to _avoid +build+ failure due to the plugin,_ run the cmd *mvn test -Dmaven.test.failure.ignore=true -Dsurefire.failIfTimeout=false*


 

If the user wants to _avoid +test+ failure(s)_ run the cmd *mvn test -Dmaven.test.failure.ignore=true*

 

Feel free to comment on this.

 
Reply all
Reply to author
Forward
0 new messages