TestNG and the Retry function... Updated info... I really need help on this!

Greg Martz

unread,

Jul 23, 2015, 2:19:12 PM7/23/15

to testng-users

This is driving me crazy! Can anyone assist me with this? I'm running a retry routine with the listener. When I run the xml from within Eclipse, it retries just fine. However, when I run the same xml from a command line with a batch file, it never even starts the retrylistener...

Here is the output when run from within Eclipse:

**********Running RetryListener

.... (Other extraneous output)

**********onStart

[Invoker 1636165903] Keeping method InitiateBuyPlan.setup(java.lang.String, java.lang.String, java.lang.String)[pri:0, instance:com.bpms.tests.InitiateBuyPlan@24150b2d] for class [TestClass name=class com.bpms.tests.InitiateBuyPlan]

[Invoker 1636165903] Keeping method InitiateBuyPlan.tearDown(org.testng.ITestResult)[pri:0, instance:com.bpms.tests.InitiateBuyPlan@24150b2d] for class [TestClass name=class com.bpms.tests.InitiateBuyPlan]

[Invoker 1636165903] Keeping method InitiateBuyPlan.setup(java.lang.String, java.lang.String, java.lang.String)[pri:0, instance:com.bpms.tests.InitiateBuyPlan@24150b2d] for class [TestClass name=class com.bpms.tests.InitiateBuyPlan]

[Invoker 1636165903] Invoking @BeforeMethod InitiateBuyPlan.setup(java.lang.String, java.lang.String, java.lang.String)[pri:0, instance:com.bpms.tests.InitiateBuyPlan@24150b2d]

In properties file: 'C:\Users\gmartz\Desktop\BPMSLogs\output.properties', Property: 'log.filename' was set to: 'C:\Users\gmartz\Desktop\BPMSLogs\07232015_1242'

In properties file: 'C:\Users\gmartz\Desktop\BPMSLogs\output.properties', Property: 'dateTime' was set to: '07232015 1242'

2015-07-23 10:42:51,730 INFO [Log] ****************************************************************************************

2015-07-23 10:42:51,740 INFO [Log] ****************************************************************************************

2015-07-23 10:42:51,740 INFO [Log] $$$$$$$$$$$$$$$$$$$$$ com.bpms.tests.InitiateBuyPlan $$$$$$$$$$$$$$$$$$$$$$$$$

2015-07-23 10:42:51,740 INFO [Log] ****************************************************************************************

**********onTestStart

[Invoker 1636165903] Invoking com.bpms.tests.InitiateBuyPlan.testInitiateBuyPlan

2015-07-23 10:43:08,059 INFO [Log] Entering Retry

2015-07-23 10:44:14,288 INFO [Log] Retry

Here is the output when run from the command line:

.... (Other extraneous output)

**********onStart

[Invoker 562850178] Keeping method InitiateBuyPlan.setup(java.lang.String, java.lang.String, java.lang.String)[pri:0, instance:com.bpms.tests.InitiateBuyPlan@62d438fd] for class [TestClass name=class com.bpms.tests.InitiateBuyPlan]

[Invoker 562850178] Keeping method InitiateBuyPlan.tearDown(org.testng.ITestResult)[pri:0, instance:com.bpms.tests.InitiateBuyPlan@62d438fd] for class [TestClass name=class com.bpms.tests.InitiateBuyPlan]

[Invoker 562850178] Keeping method InitiateBuyPlan.setup(java.lang.String, java.lang.String, java.lang.String)[pri:0, instance:com.bpms.tests.InitiateBuyPlan@62d438fd] for class [TestClass name=class com.bpms.tests.InitiateBuyPlan]

[Invoker 562850178] Invoking @BeforeMethod InitiateBuyPlan.setup(java.lang.String, java.lang.String, java.lang.String)[pri:0, instance:com.bpms.tests.InitiateBuyPlan@62d438fd]

In properties file: 'C:\Users\gmartz\Desktop\BPMSLogs\output.properties', Property: 'log.filename' was set to: 'C:\Users\gmartz\Desktop\BPMSLogs\07212015_1135'

In properties file: 'C:\Users\gmartz\Desktop\BPMSLogs\output.properties', Property: 'dateTime' was set to: '07212015 1135'

2015-07-21 09:35:34,490 INFO [Log] ********************************************************************************

2015-07-21 09:35:34,506 INFO [Log] ********************************************************************************

2015-07-21 09:35:34,506 INFO [Log] $$$$$$$$$$$$$$ com.bpms.tests.InitiateBuyPlan $$$$$$$$$$$$$$$$$$

2015-07-21 09:35:34,506 INFO [Log] ********************************************************************************

**********onTestStart

[Invoker 562850178] Invoking com.bpms.tests.InitiateBuyPlan.testInitiateBuyPlan

**********onTestFailure

2015-07-21 09:36:31,282 FATAL [Log] Screenshot can be found at : C:\Users\gmartz\Desktop\BPMSLogs\testInitiateBuyPlan_FAIL_1437496591116.png

2015-07-21 09:36:31,282 FATAL [Log] ****Test Failed****

2015-07-21 09:36:31,282 FATAL [Log] ******Aborting*****

2015-07-21 09:36:31,282 FATAL [Log] com.bpms.tests.InitiateBuyPlan Failed with the following: java.lang.AssertionError: null

2015-07-21 09:36:31,282 INFO [Log] XXXXXXXXXXXXXX -E---N---D- XXXXXXXXXXXXXXXXXX

2015-07-21 09:36:31,282 INFO [Log] X

Notice how when run from command line, Running RetryListener is not happening, and onTestFailure is called instead of the retry code.

Here is the code:

run.bat

set classpath=.\bin;libs\log4j\*;libs\selenium\*;libs\selenium\libs\*

@echo off

java org.testng.TestNG SuiteEndToEndTests.xml -d %HOMEDRIVE%%HOMEPATH%\desktop\BPMSLogs

RetryListener.java

import java.lang.reflect.Constructor;

import java.lang.reflect.Method;

import org.testng.IAnnotationTransformer;

import org.testng.IRetryAnalyzer;

import org.testng.annotations.ITestAnnotation;

public class RetryListener implements IAnnotationTransformer

{

public void transform(ITestAnnotation annotation, Class testClass,

Constructor testConstructor, Method testMethod)

{

System.out.println("**********Running RetryListener");

IRetryAnalyzer retry = annotation.getRetryAnalyzer();

if (retry == null)

{

annotation.setRetryAnalyzer(Retry.class);

}

Retry.java

package com.bpms.utility;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.remote.RemoteWebDriver;

import org.testng.IRetryAnalyzer;

import org.testng.ITestResult;

public class Retry implements IRetryAnalyzer

{

private int retryCount = 0;

private int maxRetryCount = 5; // retry a failed test 5 additional times

@Override

public boolean retry(ITestResult result)

{

System.out.println("**********onTestFailure");

String desktopDirectory = System.getProperty("user.home") + "\\Desktop\\BPMSLogs\\";

String testMethodName = result.getName().toString().trim();

String screenShotName = testMethodName + "_FAILRETRY_" + System.currentTimeMillis() + ".png";

String imagePath;

WebDriver driver = com.bpms.utility.Utility.driver;

if (retryCount <= maxRetryCount)

{

if (((RemoteWebDriver) driver).getSessionId() != null)

{

retryCount++;

//Take a snapshot of the screen for the failure

imagePath = Utility.takeScreenShot(driver, desktopDirectory, screenShotName);

Log.error("Screenshot can be found at : " + imagePath);

Log.error(result.getInstanceName() + " Failed with the following: " + result.getThrowable());

Log.error("****Test Failed - Retrying****");

}

return true;

}

return false;

}

TestListener.java

package com.bpms.utility;

import org.openqa.selenium.WebDriver;

import org.testng.ITestContext;

import org.testng.ITestListener;

import org.testng.ITestResult;

public class TestListener implements ITestListener

{

@Override

public void onFinish(ITestContext context)

{

System.out.println("**********onFinish");

}

// Following are all the method stubs that you do not have to implement

@Override

public void onTestStart(ITestResult result)

{

// TODO Auto-generated method stub

System.out.println("**********onTestStart");

}

@Override

public void onTestSuccess(ITestResult result)

{

// TODO Auto-generated method stub

System.out.println("**********onTestSuccess");

}

@Override

public void onTestFailure(ITestResult result)

{

// TODO Auto-generated method stub

System.out.println("**********onTestFailure");

String desktopDirectory = System.getProperty("user.home") + "\\Desktop\\BPMSLogs\\";

String testMethodName = result.getName().toString().trim();

String screenShotName = testMethodName + "_FAIL_" + System.currentTimeMillis() + ".png";

String imagePath;

WebDriver driver = com.bpms.utility.Utility.driver;

//Take a snapshot of the screen for the failure

imagePath = Utility.takeScreenShot(driver, desktopDirectory, screenShotName);

Log.fatal("Screenshot can be found at : " + imagePath);

Log.fatal("****Test Failed****");

Log.fatal("******Aborting*****");

Log.fatal(result.getInstanceName() + " Failed with the following: " + result.getThrowable());

Log.endTestCase(this.getClass().getName());

driver.quit();

System.exit(0);

}

@Override

public void onTestSkipped(ITestResult result)

{

// TODO Auto-generated method stub

System.out.println("**********onTestSkipped");

}

@Override

public void onTestFailedButWithinSuccessPercentage(ITestResult result)

{

// TODO Auto-generated method stub

System.out.println("**********onTestFailedButWithinSuccessPercentage");

}

@Override

public void onStart(ITestContext context)

{

// TODO Auto-generated method stub

System.out.println("**********onStart");

}

} // ends the class

Suite.xml

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</listeners>

</classes>

</test>

</suite>

Thanks!

Greg

martino turturiello

unread,

Jul 23, 2015, 4:43:00 PM7/23/15

to testng...@googlegroups.com

Hello Greg,

You are not crazy, this is happening to me as well when I run it from maven!

Anyone knows why?

Cheers,

Martino

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

--
-----------------------------------------------------
Chi non ha apprezzato il suo maestro né la sua lezione, un giorno sarà forse colto, ma non sarà mai saggio.

Numero telefonico 09711835500
Skype: turturiellomartino

Mark Derricutt

unread,

Jul 23, 2015, 9:27:51 PM7/23/15

to testng...@googlegroups.com

On 24 Jul 2015, at 8:42, martino turturiello wrote:

> You are not crazy, this is happening to me as well when I run it from maven!
> Anyone knows why?

Doesn't maven surefire generate its own XML file to use, unless you've told maven to use your custom one?

--
Mark Derricutt
http://www.theoryinpractice.net
http://www.chaliceofblood.net
http://plus.google.com/+MarkDerricutt
http://twitter.com/talios
http://facebook.com/mderricutt

signature.asc

Greg Martz

unread,

Jul 24, 2015, 9:32:17 AM7/24/15

to testng-users

I've even tried pulling the listener declaration from the xml and putting it on the command line execution with -listener com.bpms.utility.RetryListener and it still will not work.

Thanks!

Greg

Greg Martz
Test Analyst III | (425)416-5039 | gma...@Costco.com

"Confidentiality Notice: This e-mail message, including any attachments, is for the sole use to the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message."

Krishnan Mahadevan

unread,

Jul 25, 2015, 2:59:34 AM7/25/15

to testng...@googlegroups.com

I can’t seem to be able to recreate the problem when either using the java command or via Maven as well [ Both from the command prompt ]

Here’s my test class, listeners and suite file as below :

public class SimpleTestClass {

private static int count = 0;

@Test

public void testMethod() {

count = count + 1;

Reporter.log("Running iteration " + count, true);

if (count != 2) {

throw new RuntimeException("I was asked to fail");

}

public class Retry implements IRetryAnalyzer {

private int retryCount = 0;

@Override

public boolean retry(ITestResult result) {

if (retryCount == 2) {

return false;

}

retryCount = retryCount + 1;

return true;

}

public class TestListener implements ITestListener {

public static final String DECORATOR = "**********";

@Override

public void onTestStart(ITestResult result) {

Reporter.log(DECORATOR + "onTestStart()" + DECORATOR, true);

}

@Override

public void onTestSuccess(ITestResult result) {

Reporter.log(DECORATOR + "onTestSuccess()" + DECORATOR, true);

}

@Override

public void onTestFailure(ITestResult result) {

Reporter.log(DECORATOR + "onTestFailure()" + DECORATOR, true);

}

@Override

public void onTestSkipped(ITestResult result) {

Reporter.log(DECORATOR + "onTestSkipped()" + DECORATOR, true);

}

@Override

public void onTestFailedButWithinSuccessPercentage(ITestResult result) {

Reporter.log(DECORATOR + "onTestFailedButWithinSuccessPercentage()" + DECORATOR, true);

}

@Override

public void onStart(ITestContext context) {

Reporter.log(DECORATOR + "onStart()" + DECORATOR, true);

}

@Override

public void onFinish(ITestContext context) {

Reporter.log(DECORATOR + "onFinish()" + DECORATOR, true);

}

public class MyTransformer implements IAnnotationTransformer {

@Override

public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {

Reporter.log(DECORATOR + "transform()" + DECORATOR, true);

IRetryAnalyzer retry = annotation.getRetryAnalyzer();

if (retry == null) {

annotation.setRetryAnalyzer(Retry.class);

}

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</listeners>

</classes>

</test>

</suite>

When using the maven command :

krmahadevan@necropolis testbed $ mvn clean test

[INFO] Scanning for projects...

[INFO]

[INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder with a thread count of 1

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building testbed 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ testbed ---

[INFO] Deleting /Users/krmahadevan/githome/PlayGround/testbed/target

[INFO]

[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ testbed ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] Copying 1 resource

[INFO]

[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ testbed ---

[INFO] Compiling 29 source files to /Users/krmahadevan/githome/PlayGround/testbed/target/classes

[INFO]

[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ testbed ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] Copying 12 resources

[INFO]

[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ testbed ---

[INFO] Compiling 32 source files to /Users/krmahadevan/githome/PlayGround/testbed/target/test-classes

[INFO]

[INFO] --- maven-surefire-plugin:2.15:test (default-test) @ testbed ---

[INFO] Surefire report directory: /Users/krmahadevan/githome/PlayGround/testbed/target/surefire-reports

-------------------------------------------------------

T E S T S

-------------------------------------------------------

Running TestSuite

**********transform()**********

**********onStart()**********

**********onTestStart()**********

Hello testMethod

Running iteration 1

**********onTestFailure()**********

**********onTestStart()**********

Hello testMethod

Running iteration 2

**********onTestSuccess()**********

PASSED: testMethod

FAILED: testMethod

java.lang.RuntimeException: I was asked to fail

at organized.chaos.testng.SimpleTestClass.testMethod(SimpleTestClass.java:17)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)

at org.testng.internal.Invoker.invokeMethod(Invoker.java:696)

at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:882)

at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1189)

at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)

at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)

at org.testng.TestRunner.privateRun(TestRunner.java:767)

at org.testng.TestRunner.run(TestRunner.java:617)

at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)

at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)

at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)

at org.testng.SuiteRunner.run(SuiteRunner.java:254)

at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)

at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)

at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)

at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)

at org.testng.TestNG.run(TestNG.java:1057)

at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:170)

at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:84)

at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:92)

at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

===============================================

MyTest

Tests run: 2, Failures: 1, Skips: 0

===============================================

**********onFinish()**********

[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@7fcd296b: 18 ms

[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@43e1542f: 3 ms

[TestNG] Time taken by org.testng.reporters.EmailableReporter2@22c03da9: 4 ms

[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 0 ms

[TestNG] Time taken by org.testng.reporters.XMLReporter@2e8f01b8: 5 ms

[TestNG] Time taken by org.testng.reporters.jq.Main@797c048c: 32 ms

Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.675 sec <<< FAILURE! - in TestSuite

testMethod(organized.chaos.testng.SimpleTestClass) Time elapsed: 0.022 sec <<< FAILURE!

java.lang.RuntimeException: I was asked to fail

at organized.chaos.testng.SimpleTestClass.testMethod(SimpleTestClass.java:17)

Results :

Failed tests:

SimpleTestClass.testMethod:17 Runtime I was asked to fail

Tests run: 2, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 4.307 s

[INFO] Finished at: 2015-07-25T12:25:04+05:30

[INFO] Final Memory: 18M/45M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.15:test (default-test) on project testbed: There are test failures.

When using the java command :

krmahadevan@necropolis testbed $ java -classpath “<JarsInMyClassPathGoesHere>" org.testng.TestNG src/test/resources/sample.xml

**********transform()**********

[TestNG] Running:

/Users/krmahadevan/githome/PlayGround/testbed/src/test/resources/sample.xml

**********onStart()**********

**********onTestStart()**********

Hello testMethod

Running iteration 1

**********onTestFailure()**********

**********onTestStart()**********

Hello testMethod

Running iteration 2

**********onTestSuccess()**********

PASSED: testMethod

FAILED: testMethod

java.lang.RuntimeException: I was asked to fail

at organized.chaos.testng.SimpleTestClass.testMethod(SimpleTestClass.java:17)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)

at org.testng.internal.Invoker.invokeMethod(Invoker.java:696)

at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:882)

at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1189)

at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)

at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)

at org.testng.TestRunner.privateRun(TestRunner.java:767)

at org.testng.TestRunner.run(TestRunner.java:617)

at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)

at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)

at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)

at org.testng.SuiteRunner.run(SuiteRunner.java:254)

at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)

at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)

at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)

at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)

at org.testng.TestNG.run(TestNG.java:1057)

at org.testng.TestNG.privateMain(TestNG.java:1364)

at org.testng.TestNG.main(TestNG.java:1333)

===============================================

MyTest

Tests run: 2, Failures: 1, Skips: 0

===============================================

**********onFinish()**********

===============================================

MasterSuite

Total tests run: 1, Failures: 0, Skips: 0

===============================================

[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@153a8045: 7 ms

[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 0 ms

[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@2860abc5: 13 ms

[TestNG] Time taken by org.testng.reporters.EmailableReporter2@7aed9585: 4 ms

[TestNG] Time taken by org.testng.reporters.jq.Main@6dce154c: 28 ms

[TestNG] Time taken by org.testng.reporters.XMLReporter@7be3f171: 5 ms

Thanks & Regards

Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"

My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

Greg Martz

unread,

Jul 29, 2015, 2:07:04 PM7/29/15

to testng-users

Found the issue. The version of testng that eclipse was using was newer than the version of testng the command line was using (it was using the version from the selenium directory. I updated the jar and it's working.

Thanks!

Greg

Greg Martz
Test Analyst III | (425)416-5039 | gma...@Costco.com

"Confidentiality Notice: This e-mail message, including any attachments, is for the sole use to the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message."

Reply all

Reply to author

Forward