Changing the order of tests programmatically

[Baubak]

Hi there,

I had a question regardign programmatically changing the order of the tests?

I have done the following:

1. Fot he following test :

public class OrderTestsMethod {

public static int value = 100;

@Test
public void testAAA() {
value -= 20;
Assert.assertEquals(value,180);
}

@Test
public void testBBB() {
value *= 2;
Assert.assertEquals(value,200);
}
}

2. I Created a listener

public class ReverseOrderListener implements ITestListener {

@Override
public void onStart(ITestContext context) {

int testMethods = context.getAllTestMethods().length;
ITestNGMethod[] reversedArray = new ITestNGMethod[testMethods];

//Create a reversed array of the tests
for (int i = 0; i < testMethods; i++) {
reversedArray[i] = context.getAllTestMethods()[testMethods-i-1];
}

//Change the order of the tests
for (int i = 0; i < testMethods; i++) {
context.getAllTestMethods()[i] = reversedArray[i];
}

ITestListener.super.onStart(context);
}
}

In this example I have reversed the order of the tests. The tests will work only if I use my listener ReverseOrderListener. (I am well aware of the PRIORITY attribute, but in this excersize and for other reasons I cannot use them ;) ).

Now my question is if there is a better way of programmatically changing the order of my test?

Best regards,

Baubak

[⇜Krishnan Mahadevan⇝]

You could build a custom method interceptor that looks like below:

public class OrderMethods implements IMethodInterceptor, IInvokedMethodListener {

  @Override
  public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
    Comparator<IMethodInstance> comparator = Comparator.comparing(
        o -> o.getMethod().getMethodName());
    methods.sort(comparator.reversed());
    return methods;
  }

  @Override
  public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
    System.err.println(method.getTestMethod().getQualifiedName() + "()");
  }
}

and wire this in using the "@Listener" annotation.

For more information please check https://testng.org/doc/documentation-main.html#methodinterceptors

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/fb3dc695-4c2f-4f3d-85b5-e16bc5c4d160n%40googlegroups.com.

[Baubak]

Thanks Krishnan,

As always you have great suggestions :)

I will work this in and let you know.

Best regards,

Baubak

[Baubak]

Yes after testing it, it works really nice. Thank you :)

[Baubak]

Ok now here is a followup question:

The IMethodInterceptor works fine, however it intervenes before the dataproviders are set.

Is it possible to reOrder based on the data providers?

Example :

In the example above, If I had a data provider with the values DP1 & DP2 the order would normally be:

testAAA(DP1), testAAA(DP2), testBBB(DP1), testBBB(DP2)

What I would like to acheive is :

testAAA(DP1), testBBB(DP1), testAAA(DP2), testBBB(DP2)

If I perform the Method interceptor it is too early, as it seems that the provider values are set after that.

Any suggestions on this?

[⇜Krishnan Mahadevan⇝]

The IMethodInterceptor works fine, however it intervenes before the dataproviders are set.

Yes. It's because a data driven method is just multiple iterations of the same method with different parameters.

What I would like to acheive is :testAAA(DP1), testBBB(DP1), testAAA(DP2), testBBB(DP2)

Have you considered coupling a "@Factory" with a data provider to achieve this specific requirement i.e., "Have all methods in a test class run with each pair of test data from a data provider"

Your comparator in that case would look like below:

private static Comparator<IMethodInstance> comparator() {
  return (o1, o2) -> {
    String m1 = computeMethodName(o1);
    String m2 = computeMethodName(o2);
    return m1.compareTo(m2);
  };
}

private static String computeMethodName(IMethodInstance im) {
  Object instance = extractEmbeddedObject(im.getMethod());
  boolean considerParameters = instance instanceof IParameterInfo;
  if (considerParameters) {
    IParameterInfo ip = (IParameterInfo) instance;
    return im.getMethod().getMethodName() + "<" +
        Arrays.toString(ip.getParameters()) + ">";
  }
  return im.getMethod().getMethodName();
}

private static Object extractEmbeddedObject(ITestNGMethod tm) {
  try {
    List<Field> fields = new ArrayList<>();
    Class<?> clazz = tm.getClass();
    while (!clazz.equals(Object.class)) {
      fields.addAll(Arrays.asList(clazz.getDeclaredFields()));
      clazz = clazz.getSuperclass();
    }
    return fields.stream()
        .filter(f -> f.getName().equals("m_instance"))
        .findFirst()
        .map(in -> {
          in.setAccessible(true);
          try {
            return in.get(tm);
          } catch (IllegalAccessException e) {
            throw new RuntimeException(e);
          }
        }).orElse(null);
  } catch (Exception e) {
    throw new RuntimeException(e);
  }
}

Note:

One discrepancy that you would see here is that, irrespective of the sorting order I believe TestNG still confines the order of execution (when it actually gets executed) to be grouped by TestClasses (i.e., it orders the methods that belong to a particular class as found in the suite file before moving on to the next one). I don't know the rationale behind that, or if that's even a desired behavior.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/53e2b651-27ef-42ff-b6a5-fa78a3d040c3n%40googlegroups.com.

[Baubak]

Thanks Krishnan,

Wow thanks Krishnan. I got it to work thanks to you :) It took me a while to put it in context, and learn about factories. I'll play around with it an post a final solution, when I am  happy with it.

Baubak