Running tests that have multiple groups

[V. Mark Lehky]

Hello.

I am using testng 6.x.

I have multiple groups defined for my tests, probably several dozen.

Is there a way to run only tests that have only two (or more) specific groups?

Something like `mvn test -Dgroups='group1 & group2'`.

After a lot of Googling I think this might not be possible. :( So how do people generally handle this?

[Krishnan Mahadevan]

Mark,

What you ask is definitely possible in TestNG. You just need to connect a few things to get this requirement to work.

First things first. TestNG 6.x series is a very old version. Please upgrade to the latest released version (7.10.2 as of today). This will require you to be on JDK11.

Now to solve your specific use case of wanting to run tests that have only 2 or more specific groups, you can do that using a beanshell.

Steps to follow:

1. Add a dependency on beanshell [ Details on the maven coordinates are available in the documentation at https://testng.org/#_beanshell_and_advanced_group_selection ]

2. Define an implementation of IAlterSuiteListener

3. Within this listener, programmatically add a beanshell that ensures that “and” constraint for groups is adhered to.

4. Wire in this listener as a mandatory listener by referring it via service loading. [ Details about this is available at https://testng.org/#_specifying_listeners_with_serviceloader ]

Here’s a full fledged example.

I created a file called beanshell.txt under “src/test/resources” with its contents as below:

String jvmValue = System.getProperty("groups"); String[] allGroups = jvmValue.split(","); for(String group: allGroups){ if (!groups.containsKey(group)) { print("Exclude " + testngMethod.getQualifiedName() + " because it belongs to " + Arrays.toString(testngMethod.getGroups())); return false; } } return true;

My IAlterSuiteListener implementation, which reads the above mentioned text file, gets its contents and then wires it up as a beanshell expression.

import org.testng.*; import org.testng.xml.XmlMethodSelector; import org.testng.xml.XmlScript; import org.testng.xml.XmlSuite; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; import java.util.List; public class GroupsAddingListener implements IAlterSuiteListener { @Override public void alter(List<XmlSuite> suites) { String groups = System.getProperty("groups", ""); if (!groups.isEmpty()) { System.err.println("Running for the groups " + groups); suites.stream() .flatMap(it -> it.getTests().stream()) .forEach(it -> it.setMethodSelectors(List.of(xmlMethodSelector()))); } } private static XmlMethodSelector xmlMethodSelector() { XmlMethodSelector selector = new XmlMethodSelector(); XmlScript script = new XmlScript(); script.setLanguage("beanshell"); String expression = beanshell(); script.setExpression(expression); selector.setScript(script); return selector; } private static String beanshell() { try { return Files.readString(Paths.get("src/test/resources/beanshell.txt")); } catch (IOException e) { throw new RuntimeException(e); } } }

My test classes look like below:

import org.testng.ITestNGMethod; import org.testng.Reporter; import org.testng.annotations.Test; public class ATestCase { @Test(groups = {"p1", "p2"}) public void testMethod() { ITestNGMethod method = Reporter.getCurrentTestResult().getMethod(); String name = method.getQualifiedName(); String groups = String.join(",", method.getGroups()); System.err.println("Executing " + name + " in " + groups); } }

import org.testng.ITestNGMethod; import org.testng.Reporter; import org.testng.annotations.Test; public class BTestCase { @Test(groups = {"p1", "p3"}) public void testMethod() { ITestNGMethod method = Reporter.getCurrentTestResult().getMethod(); String name = method.getQualifiedName(); String groups = String.join(",", method.getGroups()); System.err.println("Executing " + name + " in " + groups); } }

My execution output looks like below (when I ran mvn clean test -Dgroups=p1,p3

[INFO] --- surefire:3.2.5:test (default-test) @ groups_demo --- [INFO] Using auto detected provider org.apache.maven.surefire.testng.TestNGProvider [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running TestSuite SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Running for the groups p1,p3 Exclude com.rationaleemotions.groups.ATestCase.testMethod because it belongs to [p1, p2] Exclude com.rationaleemotions.groups.ATestCase.testMethod because it belongs to [p1, p2] Executing com.rationaleemotions.groups.BTestCase.testMethod in p1,p3 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.485 s -- in TestSuite [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.104 s [INFO] Finished at: 2024-05-07T10:11:54+05:30 [INFO] ------------------------------------------------------------------------

For more information on beanshell, please refer to my blog https://rationaleemotions.com/beanshell_in_testng/ 

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/a4f25070-aa41-4669-8ced-9066a7df058en%40googlegroups.com.

[V. Mark Lehky]

Thanx Krishnan.

I am actually surprised that this does not already exist.

I saw another thread in this group: "What would you like to see added into TestNG?" How about this! :p

[Krishnan Mahadevan]

Mark,

Yes, that sounds like a good feature. Can you please help log a bug for this issue and we will try to get to it ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/9634e7d4-b00c-4e32-8c10-f6275f09ba2dn%40googlegroups.com.

[V. Mark Lehky]

Did you want me to file a bug? It looks like you guys do not have a "feature request" bug type.

Or did you want me to post it here: https://github.com/testng-team/testng/discussions/3037

[Krishnan Mahadevan]

Mark,

Yes. Please file an issue at https://github.com/testng-team/testng/issues 

I am not sure if GitHub has a way to distinguish  between bugs and feature requests (Yes we can add a label called “Feature Request” (or) “Enhancement”). So we will take care of it internally. You can add [Feature Request] if that’s something that you want to the issue title.

The discussion was more towards capturing the ideas that people have in terms of new stuff that they want in TestNG.

At the end of the day, for tracking purposes (what went into a specific release), everything would be part of the issues. So a discussion idea is also going to be converted into an issue before fixing it. That’s the workflow we have been following in TestNG [ The notion of discussion by itself is something new in TestNG ]

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/b47b3ab2-413b-4fdc-a50d-6f4b5e39dad0n%40googlegroups.com.

[V. Mark Lehky]

For anyone following this thread: https://github.com/testng-team/testng/issues/3125