Create New Test Cases Dynamically From Inside an @Test Method

[Zach Thomas]

My company has adopted a somewhat non-standard regression testing approach using testng and selenium. We call it "generic testing," and the concept is that a single, centrally defined test method is responsible for broad patterns accross our application and takes in more specific  testing instructions from Page Object Models as it goes. Individual test cases are defined via testng.xml files. The downside is that no test has full knowledge of what is under test until runtime. That is we often cannot know at the source code, or xml level, what pages a given test will encounter. There are several cases in which one of these tests will encounter something marked in one of our Page Object Models that necessitate breaking the test into seperate sub-tests to isolate the different paths the test should cover. We have handled these situations by running a programatic TestNG inside the outer TestNG, and then injecting the results into the parent suite via some listeners. This worked fine with reportNG, but does not function in tandem with Report Portal, which we are switching too. The problem lies in the fact that ReportNG builds its output based on the end state of the testNG object, but reportNG builds the test/suite structure via API calls onstart of the suite. I need a way to take a currently executing testContext and make that the context used by a new testNG object to attach its xml tests and suites to.

[Krishnan Mahadevan]

Zach,

It would be good if you could please help explain your use case with some sample code.

Its not clear to me as to what the issue is.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

On 27 Nov 2024, at 20:37, Zach Thomas <mrdis...@gmail.com> wrote:

My company has adopted a somewhat non-standard regression testing approach using testng and selenium. We call it "generic testing," and the concept is that a single, centrally defined test method is responsible for broad patterns accross our application and takes in more specific  testing instructions from Page Object Models as it goes. Individual test cases are defined via testng.xml files. The downside is that no test has full knowledge of what is under test until runtime. That is we often cannot know at the source code, or xml level, what pages a given test will encounter. There are several cases in which one of these tests will encounter something marked in one of our Page Object Models that necessitate breaking the test into seperate sub-tests to isolate the different paths the test should cover. We have handled these situations by running a programatic TestNG inside the outer TestNG, and then injecting the results into the parent suite via some listeners. This worked fine with reportNG, but does not function in tandem with Report Portal, which we are switching too. The problem lies in the fact that ReportNG builds its output based on the end state of the testNG object, but reportNG builds the test/suite structure via API calls onstart of the suite. I need a way to take a currently executing testContext and make that the context used by a new testNG object to attach its xml tests and suites to.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/testng-users/d39dd730-dbba-4a33-bdb5-74ed81bd9442n%40googlegroups.com.

[Zach Thomas]

Krishnan,

Thank you much for the quick reply. Here is a bit of sample code showing our current approach:

@Test
public final void test(ITestContext context) {

if (
//condition requiring test to "split" into 2+ seperate, one for each scenario encountered DURING execution
//this scenario cannot be known in advance
) {


XmlSuite synthSuite = new XmlSuite();
// build up parameters, listeners, settings, and XmlTests...

TestNG tng = new TestNG();
tng.setXmlSuites(Arrays.asList(new XmlSuite[] {synthSuite}));

tng.addListener(new TestListenerAdapter(context){
private ITestContext transferToContext;

public TransferingTestListenerAdapter(ITestContext transferToContext) {
this.transferToContext = transferToContext;
}

@Override
public void onFinish(ITestContext testContext) {
for (ITestResult result : testContext.getPassedTests().getAllResults()) {
transferToContext.getPassedTests().addResult(result, result.getMethod());
}
for (ITestResult result : testContext.getSkippedTests().getAllResults()) {
transferToContext.getSkippedTests().addResult(result, result.getMethod());
}
for (ITestResult result : testContext.getFailedTests().getAllResults()) {
transferToContext.getFailedTests().addResult(result, result.getMethod());
}
for (ITestResult result : testContext.getPassedConfigurations().getAllResults()) {
transferToContext.getPassedConfigurations().addResult(result, result.getMethod());
}
for (ITestResult result : testContext.getSkippedConfigurations().getAllResults()) {
transferToContext.getSkippedConfigurations().addResult(result, result.getMethod());
}
for (ITestResult result : testContext.getFailedConfigurations().getAllResults()) {
transferToContext.getFailedConfigurations().addResult(result, result.getMethod());
}
super.onFinish(testContext);
}
});
tng.initializeSuitesAndJarFile();
tng.run();
}
}

This approach works fine when it comes to ReportNG output, because as long as the "parent" test context ends up with the right result objects int on execution finish, the reporter will interpret the results of the suite correctly, placing the split test results (the ones inside the inner programmatic TestNG execution) inside the parent test result output (that is the one hitting the code branch above). This approach is however, is not compatible with Report Portal as it builds its output test object structure a step at a time, during execution (see the image below, wheer each call to testNGService results in http calls to the Report Portal server that build the test object, in the state described by the listener method name):

I currently have no API supported way to "graft" the inner TestNG run into the outer one's context, which is causing unwanted results in Report Portal. If I could make the ITestContext instance from the executing outer TestNG run the context for the inner programmatic TestNG run, then I could effectively relate the results of the inner run to the outer run regardless of reporting architecture. 

Zach Thomas

[Krishnan Mahadevan]

Zach,

Thanks for sharing that code snippet. It doesnt explain a lot with respect to Report Portal relevance here.

>>>> I currently have no API supported way to "graft" the inner TestNG run into the outer one's context, which is causing unwanted results in Report Portal.

That is intentional because as a user of TestNG (either via the annotations or via the API), you aren’t meant to be mingling two TestNG executions into one.

How are you integrating with Report Portal? Are you using the apis as explained in https://reportportal.io/docs/developers-guides/ReportingDevelopersGuide ?

AFAIK Report Portal is supposed to be agnostic of the runner.

Perhaps you could take a look at https://github.com/reportportal/agent-java-testNG/blob/develop/src/main/java/com/epam/reportportal/testng/BaseTestNGListener.java (I think this is what you are referencing in your screenshot) and just do the same thing that you have done currently in your addListener(). You eventually need to something like below:

Java

import com.epam.reportportal.service.ReportPortal;
import com.epam.reportportal.testng.BaseTestNGListener;
import com.epam.reportportal.testng.ITestNGService;
import com.epam.reportportal.testng.TestNGService;
import org.testng.*;
import org.testng.annotations.Test;
import org.testng.xml.XmlSuite;

import java.util.List;

public class SampleTestCase {

    @Test
    public final void test(ITestContext transferToContext) {

        XmlSuite synthSuite = new XmlSuite();
        // build up parameters, listeners, settings, and XmlTests...

        TestNG tng = new TestNG();

        tng.setXmlSuites(List.of(synthSuite));

        tng.addListener(addContextListener(transferToContext));
        tng.addListener(reportPortalListener(transferToContext));
        tng.initializeSuitesAndJarFile();
        tng.run();
    }

    private static BaseTestNGListener reportPortalListener(ITestContext transferToContext) {
        ITestNGService service = new TestNGService(ReportPortal.builder().build());
        return new BaseTestNGListener(service) {
            public void onStart(ITestContext ignored) {
                service.startTest(transferToContext);
            }

            @Override
            public void onFinish(ITestContext ignored) {
                service.finishTest(transferToContext);
            }
        };
    }

    private static TestListenerAdapter addContextListener(ITestContext transferToContext) {
        return new TestListenerAdapter() {
            @Override
            public void onFinish(ITestContext ignored) {
                add(ignored.getPassedTests(), transferToContext.getPassedTests());
                add(ignored.getSkippedTests(), transferToContext.getSkippedTests());
                add(ignored.getFailedTests(), transferToContext.getFailedTests());
                add(ignored.getPassedConfigurations(), transferToContext.getPassedConfigurations());
                add(ignored.getSkippedConfigurations(), transferToContext.getSkippedConfigurations());
                add(ignored.getFailedConfigurations(), transferToContext.getFailedConfigurations());
            }
        };
    }

    private static void add(IResultMap from, IResultMap to) {
        from.getAllResults().forEach(to::addResult);
    }
}

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

<BaseTestNGListener_snippet.png>

I currently have no API supported way to "graft" the inner TestNG run into the outer one's context, which is causing unwanted results in Report Portal. If I could make the ITestContext instance from the executing outer TestNG run the context for the inner programmatic TestNG run, then I could effectively relate the results of the inner run to the outer run regardless of reporting architecture. 

Zach Thomas

On Wednesday, November 27, 2024 at 10:11:09 AM UTC-5 Krishnan Mahadevan wrote:

Zach,

It would be good if you could please help explain your use case with some sample code.

Its not clear to me as to what the issue is.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

On 27 Nov 2024, at 20:37, Zach Thomas <mrdis...@gmail.com> wrote:

My company has adopted a somewhat non-standard regression testing approach using testng and selenium. We call it "generic testing," and the concept is that a single, centrally defined test method is responsible for broad patterns accross our application and takes in more specific  testing instructions from Page Object Models as it goes. Individual test cases are defined via testng.xml files. The downside is that no test has full knowledge of what is under test until runtime. That is we often cannot know at the source code, or xml level, what pages a given test will encounter. There are several cases in which one of these tests will encounter something marked in one of our Page Object Models that necessitate breaking the test into seperate sub-tests to isolate the different paths the test should cover. We have handled these situations by running a programatic TestNG inside the outer TestNG, and then injecting the results into the parent suite via some listeners. This worked fine with reportNG, but does not function in tandem with Report Portal, which we are switching too. The problem lies in the fact that ReportNG builds its output based on the end state of the testNG object, but reportNG builds the test/suite structure via API calls onstart of the suite. I need a way to take a currently executing testContext and make that the context used by a new testNG object to attach its xml tests and suites to.

-- 
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/testng-users/d39dd730-dbba-4a33-bdb5-74ed81bd9442n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/testng-users/c427347b-1d75-44ea-b7ab-70962e88b912n%40googlegroups.com.
<BaseTestNGListener_snippet.png>

>>>>