Relating Test Instance Data to TestListenerAdapter's ITestResult

[Jeremy Cd]

I'm trying to correlate test executions to test results.  Method name, even qualified with parameters is not good enough because I may reuse the same inputs multiple times.  I have looked for a thread ID in the ITestResults, but I see none.  Is there a way to get the test's thread ID from the TestListenerAdapter? 

In my example below, assuming I generated hundreds of inputs, including cow and crow, my TestAnalysis object might notice both started with "c" and alert me to that. However, in order to do that I need to know if a given test passed or failed.  I don't want to have in the end of every test a "this test passed" when there is an obvious and easy single place to do that, TestListenerAdapter.  However, the only data I can get is the method name (from the stack).  I need something better to positively ID the test than the name.  For example, if I randomly generated 10 animals, I'd see the same method 10 times.  Even with parameters, I might generate the same animal twice.  Are there any suggestions on how to get this information?  Thanks,

- JCD

public class MyTestCases {

    public static final HashMap<Thread, TestAnalysis> test = new HashMap<Thread, TestAnalysis>();

    public static synchronized TestAnalysis getActiveThread() {
        if(!test.containsKey(Thread.currentThread())) test.put(Thread.currentThread(), new TestAnalysis());
       
        return test.get(Thread.currentThread());
    }
   
    public static synchronized TestAnalysis getThreadById(long id) {
        //TODO: Get Thread by Id
    }

   
    @Test(enabled=true, dataProvider = "dataGenerator", dataProviderClass = MyTestCases.class)
    public void testCase(String animal) { getActiveThread().addData(animal);
       if("cow".equals(animal)) throw new Error("Mooooo");
       if("crow".equals(animal)) throw new Error("Never more!");
    }
    //..Create generator, etc.
   
    //TODO Have a final test run the analysis
}


public class TestAnalysisListener extends TestListenerAdapter {

    @Override
    public void onTestFailure(ITestResult tr) {
        MyTestCases.getActiveThread().markTestFailed();//Wrong thread for test.  This will create a new instance and mark it as a failure.
   
     MyTestCases.getThreadById(tr.getThreadIdMagic()???).markTestFailed();//No magic I see can relate the thread executed for the test to the result.
        // ITestResult does not know the thread ID used for the test, so how can I find out what test did
    }

    //Insert all other required methods ...
}

[Krishnan Mahadevan]

Jeremy,

Would something like this work for you ?

@Listeners (MyTestCases.MyAnalyseListener.class)
public class MyTestCases {

    @Test (enabled = true, dataProvider = "dp")
    public void testCase(String animal) {
        if ("cow".equals(animal)) {
            throw new IllegalArgumentException("Mooooo");
        }
        if ("crow".equals(animal)) {
            throw new IllegalStateException("Never more!");
        }
    }

    @DataProvider (name = "dp")
    public Object[][] getData() {
        return new Object[][] {
            {"cow"},
            {"crow"},
            {"dog"}
        };
    }

    public static class TestAnalysis {
        private Thread thread;
        private Object[] parameters;

        public TestAnalysis(Thread thread, Object[] parameters) {
            this.thread = thread;
            this.parameters = Arrays.copyOf(parameters, parameters.length);
        }

        public Thread getThread() {
            return thread;
        }

        public Object[] getParameters() {
            return Arrays.copyOf(parameters, parameters.length);
        }
    }


    public static class MyAnalyseListener extends TestListenerAdapter implements IInvokedMethodListener {

        @Override
        public void onTestFailure(ITestResult tr) {

            TestAnalysis analysis = (TestAnalysis) tr.getAttribute(TestAnalysis.class.getSimpleName());
            if (analysis != null) {
                System.err.println("Analysis : Thread " + analysis.getThread().getId() + " failed with parameters as "
                    + "" + Arrays.toString(analysis.getParameters()));
            }
        }

        @Override
        public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
            TestAnalysis analysis = new TestAnalysis(Thread.currentThread(), testResult.getParameters());
            testResult.setAttribute(TestAnalysis.class.getSimpleName(), analysis);
        }

        @Override
        public void afterInvocation(IInvokedMethod method, ITestResult testResult) {

        }
    }

}

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[Jeremy Cd]

Doh, I hit private reply rather than reply.  There are two problems with your solution.  The first one is that “beforeInvocation” does not run on the same thread as the test does, so the current thread call could not be correlated to the test case.  The second one was a requirement I did not capture due to me trying to simplify my example.  Even though this will get the test case parameters, it does not allow access to the test case code.  For example:

    @Test (enabled = true, dataProvider = "dp")
    public void testCase(String animal) {
        if ("cow".equals(animal)) {
            throw new IllegalArgumentException("Mooooo");
        }
        if ("crow".equals(animal)) {
            throw new IllegalStateException("Never more!");
        }

        AnimalResponse response = new AnimalApi(animal).run();
        test.getActiveThread().add(response);
        LocationData location = new AnimalLocationService(response);
        test.getActiveThread().add(location);
        //TODO add asserts that might cause failures.        
    }

The ultimate idea is if we generate say 100,000+ tests and perhaps 20 of them fail, we want to know what variables might connect these 20 tests or what theories have been disproved by counter examples in passing tests.  In order to create such data I have to know the state of each of the tests -- did they pass or fail.  In theory I might even add things like "Step 1 completed" or even the entire log per test.  Obviously these variables might be generated mid-way though the test.  Do you have any suggestions around this more complete example?  Thanks,

- JCD

[Krishnan Mahadevan]

To the best of my knowledge beforeInvocation(), @Test method and afterInvocation() are always guaranteed to run on the same thread. If you are NOT seeing this behaviour, then you would need to show us a reproducible that can recreate the problem.

With respect to being able to figure out more information w.r.t test failures, you can start do one of the following :

• Log important information via Reporter.log(). This guarantees that the logs would always be associated with the test that generated them.
• Enhance your TestAnalysis object such that it starts having a Map [ Map<String, Object>] such that the keys are all distinct attributes that track a unique thing about your test and the values are values to that attribute. Since you already have access to the current test method's test result via Reporter.getCurrentTestResult() and since this is the same ITestResult object that was passed to beforeInvocation() and afterInvocation() you should be able to enhance everything in one go.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

--

[Jeremy Cd]

So I have managed to isolate what appears to be a bug in TestNG.  When I created a simplified form of what you suggested, it worked, however when using our framework it broke.  So I started looking into the listeners that I had and started adding them back in one by one.  I discovered it was the IAnnotationTransformer.  After going line by line I found that when I called setTimeOut, it then uses a different thread.   I specifically tested to see if other properties such as annotation.setDescription also caused problems, but they did not.  This vaguely makes sense since TestNG probably uses a thread timeout property in order to kill the thread rather than having a background thread do scans of all running threads to see if they have timed out.  Is there a place I should go to file a bug or is this expected behaviour?  Thanks,

- JCD

public class AnnotationOverrides implements IAnnotationTransformer
   public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod)
   {
     if(annotation.getTimeOut() == 0) {
          annotation.setTimeOut((60 * 20) * 1000);
      }
   }
}

[Krishnan Mahadevan]

Jeremy,

I was able to reproduce the problem as well. Can you please file an issue here : https://github.com/cbeust/testng/issues

Please include a listener (which implements annotation transformer and innvokedmethodlistener) and a test class, along with a testtng suite xml file, so that anyone who wants to work on fixing the problem has a reproducible case to start off with in the same issue.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

[Julien Herr]

You can try to use IHookable instead of IInvokedMethodListener which is supposed to work: https://github.com/cbeust/testng/issues/599

In fact, timeout is a specific case where the test method may be executed into a Executor.

Currently, IInvokedMethod is called even if the test method is not.

For the moment, I don't see how to fix the issue without changing the behavior.

But maybe we should change it: https://github.com/cbeust/testng/pull/915#discussion_r47847468

Cédric, Krishnan, what do you think?

[Jeremy Cd]

That appears that might work.  I've not fully implemented a design but in a quick POC and it appears it would run on the same thread.  I probably won't get to it this year, but I will certainly look at it.  I did file a bug around this issue and it points back here.  Here's the bug: https://github.com/cbeust/testng/issues/914 .  Thanks,

- Jeremy