Combine/collate multiple testng-results.xml files into one

[Vijay Anand S]

Customer has requested for a single report that speak on weekly achievement and performance of automation suite

Scenario:-

I have multiple (maven) projects which contain tests and all produce
their own testng-results.xml on project level. I also have a parent
project that I execute to invoke all the sub-projects. Ideally I would
like to configure testng in the parent project to take all sub-project
testng-results.xml files and combine them into one which will be
stored on the parent project level.

If anybody has had a similar case I would love to hear how you solved
this problem.

All factual and other information within this e-mail including any attachments, relating to the official business of CodeCraft Technologies (P) Ltd.  It is confidential and legally protected against any unauthorized use. CodeCraft neither owns nor endorses any information which does not relate to its official business. This would include personal mail and/or opinions by senders who may or may not be employed by the CodeCraft.

 

The person addressed in the e-mail is the intended recipient. Please notify the sender immediately if it has unintentionally reached another recipient.  Such recipient must not read, disclose or use the content in any way whatsoever. CodeCraft cannot ensure that this e-mail is free of errors, viruses, interception or interference and will not be held liable for any personal mail and/or opinions expressed herein.

[⇜Krishnan Mahadevan⇝]

TestNG doesn't have this capability.

In simple terms you could build yourself a Java class that can merge "n" xml files into one.

And then have the maven exec plugin just invoke this java class after the "Surefire plugin" test phase.

Would that work ?

Something like this: https://stackoverflow.com/a/20766817 

Or you could have this logic invoked from the implementation of IExecutionListener which gets invoked per TestNG life cycle and only once (even if TestNG is dealing with multiple suite xml files)

Or you could make use of some of the sophisticated test results accumulators such as http://allure.qatools.ru/

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/6227d9a1-ec04-44ed-a495-8905ded9aeacn%40googlegroups.com.

[Vijay Anand S]

Thanks Krishnan. I'm using allure . Generated allure report contains only the last test instance.

Key points about design:-

A. I've implemented iAlterSuiteListener .

B. TestNG report is fine that display the unique test name.

C. Allure contains the last test instance name .

Any help is appreciated. 

[⇜Krishnan Mahadevan⇝]

I didnt quite understand what help you are looking for.

AFAIK, allure was meant to be able to collate test results across the board and still show them as one execution. Are you saying this is broken ? If yes, then you may have to post it on the allure forums to get some traction.

Not sure why the IAlterSuiteListener is present. What are you altering in the suite dynamically ?

Also did you explore the IExecutionListener approach that I called out earlier ? Perhaps that would be simple and effective.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/79241a36-4e33-4914-b9d9-74bc89b44ec3n%40googlegroups.com.

[Vijay Anand S]

Why IAlterSuiteListner?

Utilization of TestNG factory to provide different inputs of tests generates a report displaying all the scenario's under a single test. IAlterSuiteListner solved the problem.

Issue with Allure

Irrespective of the number of iteration or parameters passed, unlike TestNG { test count matches with data (iteration) passed} Allure reports as one test (last test instance or last parameter passed). 

With the unique test name passed, I'm confused with logic considered by Allure to group or sum up the number of test cases.

Any advice is highly appreciated. 

[Vijay Anand S]

Report[html] I always get in, prints the 'test-instance-name' of the last name of the test I ran.  Any solution to this? ...Looks like 

 ITest strategy leaves the name for the test as the name set by the last run

On Tuesday, August 10, 2021 at 1:50:27 PM UTC+5:30 Vijay Anand S wrote:

Report[html] I always get in, prints the 'test-instance-name' of the last name of the test I ran.  Any solution to this ? 

[⇜Krishnan Mahadevan⇝]

Can we please break down the problems into "TestNG related" and "TestNG not related" ?

Allure reports problems, like I said before, are better posted on relevant forums of allure (StackOverFlow maybe?) because TestNG does not own or support Allure reports directly.

The "ITest" implementation AFAIK, kind of goes on a per instance basis to figure out what the test name should be. So if you club it with some test class that has 2 or more methods (does not matter if the test is data driven or not) and trying to use the parameters to construct the test class name, then it would not work.

You may have to explore building your own reporting mechanism to accommodate your requirements.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/38f19061-b889-4de1-a81c-6130db19d1cen%40googlegroups.com.

[Vijay Anand S]

<test-method signature="pageVerification(org.testng.ITestContext)[pri:0, instance:com.web.template.XBase.XBase@263f04ca, instance params:[[class com.web.template.DryRun.DryRun, class com.web.template.SecondDryRun.SecondDryRun]]]" started-at="2021-08-10T13:41:45 IST" name="pageVerification" test-instance-name="User custom testcase name9" finished-at="2021-08-10T13:41:50 IST" duration-ms="5211" status="PASS">

How can I change the name ?. In the above example, the name contains "page verification". 

[⇜Krishnan Mahadevan⇝]

Can you please share a simple sample of what a test class of yours looks like?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/11091958-122c-48a7-9bb6-50daf51c4648n%40googlegroups.com.

[Vijay Anand S]

List<Class<Base>> classes;
StringBuffer contextName = new StringBuffer();
AllureLifecycle lifecycle = Allure.getLifecycle();

protected String testCaseName = "";
AtomicInteger counterA = new AtomicInteger(0); //debug purpose

public XBase(){}

//Method to pick all the classes extending Base
public static List<Class<Base>> identifyClassAtRunTime(){
logger.info("Method : " + Thread.currentThread().getStackTrace()[1].getMethodName());
try {

List<Class<Base>> classLocalVariable;
try (ScanResult scanResult = new ClassGraph().whitelistPackages("com.web.template")
.enableClassInfo().scan()) {
classLocalVariable = scanResult
.getSubclasses(Base.class.getName())
.loadClasses(Base.class);
}
logger.info("No of classes identified :" +classLocalVariable.size());
logger.info("List of classes identified are : "+ classLocalVariable);

return classLocalVariable;
}catch (Exception e) {
e.printStackTrace();
Assert.fail("Identifying class at run time failed" + e);
}
return null;
}

@DataProvider(name="listOfPageClasses")
public Object[] dataProviderMethod(){
return new Object[][]{
{
XBase.identifyClassAtRunTime()
},

};
}
@Factory(dataProvider = "listOfPageClasses")
public XBase(List<Class<Base>> classesObj) {
this.classes = classesObj;
}

@Override
public String getTestName() {
logger.info("TestName");
this.testCaseName = "User custom testcase name" +counterA.incrementAndGet();//dummy name

return this.testCaseName;
}

@Test
public void pageVerification(final ITestContext context)
{
logger.info("Method " + Thread.currentThread().getStackTrace()[1].getMethodName());
//Counter to fetch value from list
AtomicInteger counter = new AtomicInteger(0);
logger.info("Number of classes identified for Test: " +classes.size());
try {

Method getMethod = null;
//ContextName that stores the testName passed in suiteCreationListener
contextName .append(
context.getName().substring(context.getName().indexOf("com"),
context.getName().length()));
logger.info("Context Name-" + contextName);

while(counter.get() < classes.size())
{
//Matching classes gets execution. In another case this loop will execute all classes for each test instance.
if(classes.get(counter.get()).getName().equalsIgnoreCase(String.valueOf(contextName)))
{
getMethod = classes.get(counter.get()).getMethod("executeMethodAtRunTime");
getMethod.invoke(classes.get(counter.get()).getDeclaredConstructor().newInstance());
//dummy validation
Assert.assertTrue(true);
}
counter.getAndIncrement();
}
}catch (Exception e )
{
e.printStackTrace();
logger.error("Page verification failed", e.getMessage());
//Dummy validation for reporting purpose
Assert.assertTrue(false);

}finally {
contextName.setLength(0);

}
}

[Vijay Anand S]

And here is my suitecreation class

public class SuiteCreationListener implements IAlterSuiteListener {

private final static Logger logger = LogManager.getLogger(SuiteCreationListener.class.getName());

@Override
public void alter(List<XmlSuite> suites) {
for (XmlSuite suite : suites) {
List<XmlTest> tests = new ArrayList<>();
List<Class< Base >> classes = XBase.identifyClassAtRunTime();
for (Class<Base> classObj : classes) {
XmlTest test = new XmlTest(suite);
test.setName("Test _" + classObj);
test.getClasses().add(new XmlClass(XBase.class));

[⇜Krishnan Mahadevan⇝]

Like I said, the ITest implementation is perhaps not going to be of help here since TestNG creates one test class instance and all the "@Test" methods in it share the same instance. As such whatever was the last value returned by "ITest" is what TestNG would remember.

You would need to create a custom TestNG reporting mechanism that basically uses the method and its associated parameters to form a proper test method name in the results.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/3741cab2-192f-4e46-9b05-89b66aa69f90n%40googlegroups.com.