Changing the order of tests programmatically

54 views
Skip to first unread message

Baubak

unread,
Oct 14, 2022, 10:07:25 AM10/14/22
to testng-users
Hi there,

I had a question regardign programmatically changing the order of the tests?

I have done the following:
1. Fot he following test :
public class OrderTestsMethod {

public static int value = 100;

@Test
public void testAAA() {
value -= 20;
Assert.assertEquals(value,180);
}

@Test
public void testBBB() {
value *= 2;
Assert.assertEquals(value,200);
}
}

2. I Created a listener

public class ReverseOrderListener implements ITestListener {

@Override
public void onStart(ITestContext context) {

int testMethods = context.getAllTestMethods().length;
ITestNGMethod[] reversedArray = new ITestNGMethod[testMethods];

//Create a reversed array of the tests
for (int i = 0; i < testMethods; i++) {
reversedArray[i] = context.getAllTestMethods()[testMethods-i-1];
}

//Change the order of the tests
for (int i = 0; i < testMethods; i++) {
context.getAllTestMethods()[i] = reversedArray[i];
}

ITestListener.super.onStart(context);
}
}


In this example I have reversed the order of the tests. The tests will work only if I use my listener ReverseOrderListener. (I am well aware of the PRIORITY attribute, but in this excersize and for other reasons I cannot use them ;) ).

Now my question is if there is a better way of programmatically changing the order of my test?

Best regards,

Baubak

⇜Krishnan Mahadevan⇝

unread,
Oct 14, 2022, 10:56:08 PM10/14/22
to testng...@googlegroups.com
You could build a custom method interceptor that looks like below:

public class OrderMethods implements IMethodInterceptor, IInvokedMethodListener {

@Override
public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
Comparator<IMethodInstance> comparator = Comparator.comparing(
o -> o.getMethod().getMethodName());
methods.sort(comparator.reversed());
return methods;
}

@Override
public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
System.err.println(method.getTestMethod().getQualifiedName() + "()");
}
}
and wire this in using the "@Listener" annotation.



Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribblings @ https://rationaleemotions.com/


--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/fb3dc695-4c2f-4f3d-85b5-e16bc5c4d160n%40googlegroups.com.

Baubak

unread,
Oct 17, 2022, 4:36:21 AM10/17/22
to testng-users
Thanks Krishnan,

As always you have great suggestions :)

I will work this in and let you know.

Best regards,

Baubak

Baubak

unread,
Oct 17, 2022, 11:56:58 AM10/17/22
to testng-users
Yes after testing it, it works really nice. Thank you :)

Baubak

unread,
Oct 27, 2022, 8:10:55 AM10/27/22
to testng-users
Ok now here is a followup question:

The IMethodInterceptor works fine, however it intervenes before the dataproviders are set.

Is it possible to reOrder based on the data providers?

Example :
In the example above, If I had a data provider with the values DP1 & DP2 the order would normally be:
testAAA(DP1), testAAA(DP2), testBBB(DP1), testBBB(DP2)

What I would like to acheive is :
testAAA(DP1), testBBB(DP1), testAAA(DP2), testBBB(DP2)

If I perform the Method interceptor it is too early, as it seems that the provider values are set after that.

Any suggestions on this?

⇜Krishnan Mahadevan⇝

unread,
Oct 28, 2022, 12:55:30 AM10/28/22
to testng...@googlegroups.com

The IMethodInterceptor works fine, however it intervenes before the dataproviders are set.

Yes. It's because a data driven method is just multiple iterations of the same method with different parameters.

What I would like to acheive is :testAAA(DP1), testBBB(DP1), testAAA(DP2), testBBB(DP2)

Have you considered coupling a "@Factory" with a data provider to achieve this specific requirement i.e., "Have all methods in a test class run with each pair of test data from a data provider"

Your comparator in that case would look like below:

private static Comparator<IMethodInstance> comparator() {
return (o1, o2) -> {
String m1 = computeMethodName(o1);
String m2 = computeMethodName(o2);
return m1.compareTo(m2);
};
}

private static String computeMethodName(IMethodInstance im) {
Object instance = extractEmbeddedObject(im.getMethod());
boolean considerParameters = instance instanceof IParameterInfo;
if (considerParameters) {
IParameterInfo ip = (IParameterInfo) instance;
return im.getMethod().getMethodName() + "<" +
Arrays.toString(ip.getParameters()) + ">";
}
return im.getMethod().getMethodName();
}

private static Object extractEmbeddedObject(ITestNGMethod tm) {
try {
List<Field> fields = new ArrayList<>();
Class<?> clazz = tm.getClass();
while (!clazz.equals(Object.class)) {
fields.addAll(Arrays.asList(clazz.getDeclaredFields()));
clazz = clazz.getSuperclass();
}
return fields.stream()
.filter(f -> f.getName().equals("m_instance"))
.findFirst()
.map(in -> {
in.setAccessible(true);
try {
return in.get(tm);
} catch (IllegalAccessException e) {
throw new RuntimeException(e);
}
}).orElse(null);
} catch (Exception e) {
throw new RuntimeException(e);
}
}

Note:
One discrepancy that you would see here is that, irrespective of the sorting order I believe TestNG still confines the order of execution (when it actually gets executed) to be grouped by TestClasses (i.e., it orders the methods that belong to a particular class as found in the suite file before moving on to the next one). I don't know the rationale behind that, or if that's even a desired behavior.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/
My Technical Scribblings @ https://rationaleemotions.com/

Baubak

unread,
Oct 30, 2022, 6:51:18 AM10/30/22
to testng-users
Thanks Krishnan,

Wow thanks Krishnan. I got it to work thanks to you :) It took me a while to put it in context, and learn about factories. I'll play around with it an post a final solution, when I am  happy with it.


Baubak
Reply all
Reply to author
Forward
0 new messages