time out setting in TestNG doesn't work

[Michael Yang]

Have tried a few approaches of setting global/test level timeout in TestNG, but none has got successful.

- <suite ......... parallel="tests" time-out="5000">

- add to Test annotation, e.g. @Test(timeOut=5000)

- after the suites loaded, explicitly set the time out again, suite.setTimeOut("5000"), or test.setTimeOut(5000)

None of above has the timeout working successfully, the tests keep running to the end.

The way I called test is by creating TestNG() instance, and load the suites from the xml file.

Version being used in 6.8.3, actually have tried 6.9.6 but it's same issue.

Anyone has clues how to get this working correctly?

Thanks a lot.

[Krishnan Mahadevan]

Michael,

You would need to share a sample which when executed can trigger the problem.

I haven't been able to simulate your problem. Here's my code and the output

public class TimeOuts {
    @Test (timeOut = 1000)
    public void myTestMethod() throws InterruptedException {
        System.err.println("Commencing sleep");
        Thread.sleep(1500);
        System.err.println("Sleep completed");
    }

    public static void main(String[] args) {
        TestNG testNG = new TestNG();
        testNG.setVerbose(2);
        testNG.setTestClasses(new Class[] {TimeOuts.class});
        String version = new File(TestNG.class.getProtectionDomain().getCodeSource()
            .getLocation().getFile()).getParentFile().getName();
        System.err.println("TestNG Version : " + version);
        testNG.run();
    }
}

TestNG Version : 6.9.6

[TestNG] Running:

  Command line suite

Commencing sleep

FAILED: myTestMethod

org.testng.internal.thread.ThreadTimeoutException: Method org.testng.internal.TestNGMethod.myTestMethod() didn't finish within the time-out 1000

===============================================

    Command line test

    Tests run: 1, Failures: 1, Skips: 0

===============================================

===============================================

Command line suite

Total tests run: 1, Failures: 1, Skips: 0

===============================================

[TestNG] Time taken by org.testng.reporters.XMLReporter@1997c457: 13 ms

[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@6c9d2223: 4 ms

[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 8 ms

[TestNG] Time taken by org.testng.reporters.EmailableReporter2@622d8a59: 3 ms

[TestNG] Time taken by org.testng.reporters.jq.Main@3a65fca: 36 ms

[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@1dc882db: 9 ms

Process finished with exit code 0

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[Michael Yang]

Thanks Krishnan!

It seems be the case if you catch the exception around your "Thread.sleep" statement. You'll notice that test continues to execute.

Does time out force an exception on the running test to break out? What if all exceptions are handled in the test.

Here are my examples:

import org.testng.annotations.Test;

public class TestTimeout {

    @Test

    public void test1(){

        System.out.println("test1 started.");

        try{

            Thread.sleep(10 * 1000);

        }catch(Exception e){}

        System.out.println("test1 finished.");

    }

    @Test

    public void test2(){

        System.out.println("test2 started.");

        try{

            Thread.sleep(10 * 1000);

        }catch(Exception e){}

        System.out.println("test2 finished.");

    }

}

public class TestTimeoutMain {

    public static void main(String[] args) throws Exception{

        String suiteFile = "test-timeout.xml";

        List<XmlSuite> suites = new Parser(suiteFile).parseToList();

        TestNG testng = new TestNG();

        testng.setXmlSuites(suites);

        testng.run();

    }

}


test-timeout.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Test time out" parallel="tests" time-out="1000">

    <test name="TimeoutTest" preserve-order="true">

        <classes>

            <class name="TestTimeout"/>

        </classes>

    </test>

</suite>

Output:

[ThreadUtil] Starting executor timeOut:1000ms workers:1 threadPoolSize:5

test1 started.

test1 finished.

===============================================

Test time out

Total tests run: 0, Failures: 0, Skips: 0

===============================================

test2 started.

[TestNG] Time taken by org.testng.reporters.EmailableReporter2@f151bf2: 9 ms

[TestNG] Time taken by org.testng.reporters.XMLReporter@1ea7d7b9: 15 ms

[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@43d1068c: 22 ms

[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 1 ms

[TestNG] Time taken by org.testng.reporters.jq.Main@21add353: 34 ms

[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@36fd0d7d: 1 ms

test2 finished.

PASSED: test1

PASSED: test2

===============================================

    TimeoutTest

    Tests run: 2, Failures: 0, Skips: 0

===============================================

Thanks,

Michael. 

[Michael Yang]

If I add the timeout value to the annotation of the test, it just ignores the timeout. 

Look at the default time out value of "2147483647" ms in the output:

public class TestTimeout {

    @Test(timeOut = 1000)

    public void test1(){

        System.out.println("test1 started.");

        try{

            Thread.sleep(10 * 1000);

        }catch(Exception e){}

        System.out.println("test1 finished.");

    }

    @Test(timeOut = 1000)

    public void test2(){

        System.out.println("test2 started.");

        try{

            Thread.sleep(10 * 1000);

        }catch(Exception e){}

        System.out.println("test2 finished.");

    }

}

[ThreadUtil] Starting executor timeOut:2147483647ms workers:1 threadPoolSize:5

test1 started.

test1 finished.

test2 started.

test2 finished.

PASSED: test1

PASSED: test2

===============================================

    TimeoutTest

    Tests run: 2, Failures: 0, Skips: 0

===============================================

===============================================

Test time out

Total tests run: 2, Failures: 0, Skips: 0

===============================================

[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 0 ms

[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@207c5965: 26 ms

[TestNG] Time taken by org.testng.reporters.XMLReporter@144c5230: 8 ms

[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@7e03f9ad: 4 ms

[TestNG] Time taken by org.testng.reporters.EmailableReporter2@5ba295be: 6 ms

[TestNG] Time taken by org.testng.reporters.jq.Main@ff70225: 62 ms

[Krishnan Mahadevan]

Michael,

I still can't seem to simulate your problem. Even if you do gobble up all exceptions within your test a timeout condition would still be satisfied because its not the test that throws the exception, but its TestNG that's doing it.

Here's the modified sample and its output.

public class TimeOuts {
    @Test (timeOut = 1000)

    public void myTestMethod() {
        try {

            System.err.println("Commencing sleep");
            Thread.sleep(1500);
            System.err.println("Sleep completed");

        } catch (Exception e) {
            e.printStackTrace();

        }
    }

    public static void main(String[] args) {
        TestNG testNG = new TestNG();
        testNG.setVerbose(2);
        testNG.setTestClasses(new Class[] {TimeOuts.class});
        String version = new File(TestNG.class.getProtectionDomain().getCodeSource()
            .getLocation().getFile()).getParentFile().getName();
        System.err.println("TestNG Version : " + version);
        testNG.run();
    }
}

TestNG Version : 6.9.6

[TestNG] Running:

  Command line suite

Commencing sleep

java.lang.InterruptedException: sleep interrupted

at java.lang.Thread.sleep(Native Method)

at organized.chaos.testng.TimeOuts.myTestMethod(TimeOuts.java:16)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)

at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)

at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:724)

FAILED: myTestMethod

org.testng.internal.thread.ThreadTimeoutException: Method org.testng.internal.TestNGMethod.myTestMethod() didn't finish within the time-out 1000

at java.lang.Throwable.printStackTrace(Throwable.java:648)

at java.lang.Throwable.printStackTrace(Throwable.java:642)

at java.lang.Throwable.printStackTrace(Throwable.java:633)

at organized.chaos.testng.TimeOuts.myTestMethod(TimeOuts.java:20)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)

at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)

at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:724)

So lets take a step back and review what we have so far.

I am running on TestNG 6.9.6 using IntelliJ (Not that it should matter, but still).

In my case, its a main() method that's triggering the execution.

What does your environment look like ? Can you please help explain a bit more.

Also if possible, check if can remove off the IDE from the equation (Eclipse TestNG plugin uses an embedded TestNG jar within it and if you are using an older plugin version that could also be a reason for the confusion) and see if you can run the main method/the test from the command line.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

[Michael Yang]

Hi Krishnan,

I also uses intelliJ and run the test from main() as well as I showed in the examples earlier. I didn't use any IDE plugins for test execution, but regular way to run main app by creating TestNG instance with a xml suite file loaded.

I'm seeing a difference:

in the xml suite file I defined, I have the "parallel" set to "tests", would this matter? I think TestNG spawns threads for parallel tests, but the timeout seems be ignored as the output shows in the example I provided earlier.

Thanks,

Michael.

[Krishnan Mahadevan]

Michael,

Yes now I am able to recreate your problem. The parallel attribute definitely seems to be messing up with the timeouts here. 

My opinion is please go ahead and create a github issue for this and include the below sample for recreating the problem. I will see if I can try taking a look at it over the weekend.

public class TimeOuts {
    @Test (timeOut = 1000)
    public void myTestMethod() {
        try {
            System.err.println("Commencing sleep");
            Thread.sleep(1500);
            System.err.println("Sleep completed");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) {
        TestNG testNG = new TestNG();
        testNG.setVerbose(2);

//         testNG.setParallel("tests");
        // If setParallel("tests") is commented out then timeout happens else there are
        // no timeouts.

        testNG.setTestClasses(new Class[] {TimeOuts.class});
        String version = new File(TestNG.class.getProtectionDomain().getCodeSource()
            .getLocation().getFile()).getParentFile().getName();
        System.err.println("TestNG Version : " + version);
        testNG.run();
    }
}

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

[Michael Yang]

Thank you very much, Krishnan!

I looked at the github issues, I think there's already one that has presented this issue:

https://github.com/cbeust/testng/issues/779 (status: open)

Can you take a look?

Thanks,

Michael.

[Michael Yang]

Anyway, still opened an issue to correlate this thread:

https://github.com/cbeust/testng/issues/811

Thanks!