How to granularly run TestNG cases?

[Rajdeep Biswas]

Hi all.

I have a testSuite.xml and for some conditions, say, depending upon the environment of application deployment, to run or not run validations inside test methods, I am also passing an arg in maven goals in CI. Like

mvn test  -DsuiteXmlFile=src/res/testSuite.xml -DmyCustomProp=true

How do I add more granularity like controlling the classes/methods when to turn on and off?

For example, I may want to run a particular suite in one of the FT jobs, but decide to not run a few classes or methods because they don't need to be based on the application deployment environment. In brief, I have been told to do these:

1. Control the conditions within the test method. (I think the argument I specified above does this)
2. Control the entire test method. (Todo)
3. Control the entire test class. (Todo)
4. Control the entire test suite. (Todo. May be when one suite references other sub suites, and we want to enable/disable some)

How to achieve these?

Thanks

Rajdeep

[Todd Bradley]

I use test groups for this. But that approach only works if you know what test methods and classes should be grouped together in advance. If you need the tests to make that decision solely at runtime, you'll need something more sophisticated.

Todd.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users+unsubscribe@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[Rajdeep Biswas]

Thank you Todd. Do you mean that you have created suites and grouped methods inside them and pass during runtime? Can you give a sample or any link of this where one can turn on/off certain groups in the suite, using arguments, for example from the CI. 

Like, running a suite with certain group once and another group included in different run.

Control should be tunable to levels of suite, class, method, and conditions inside.

Also, to determine things in runtime, with minimal static code like suites and groups, can you suggest me something thats easily maintainable too!

On Thursday, 26 October 2017 20:46:26 UTC+5:30, Todd Bradley wrote:

I use test groups for this. But that approach only works if you know what test methods and classes should be grouped together in advance. If you need the tests to make that decision solely at runtime, you'll need something more sophisticated.

Todd.

On Thu, Oct 26, 2017 at 8:07 AM, Rajdeep Biswas <raj.my...@gmail.com> wrote:

Hi all.

I have a testSuite.xml and for some conditions, say, depending upon the environment of application deployment, to run or not run validations inside test methods, I am also passing an arg in maven goals in CI. Like

mvn test  -DsuiteXmlFile=src/res/testSuite.xml -DmyCustomProp=true

How do I add more granularity like controlling the classes/methods when to turn on and off?

For example, I may want to run a particular suite in one of the FT jobs, but decide to not run a few classes or methods because they don't need to be based on the application deployment environment. In brief, I have been told to do these:

1. Control the conditions within the test method. (I think the argument I specified above does this)
2. Control the entire test method. (Todo)
3. Control the entire test class. (Todo)
4. Control the entire test suite. (Todo. May be when one suite references other sub suites, and we want to enable/disable some)

How to achieve these?

Thanks

Rajdeep

--
You received this message because you are subscribed to the Google Groups "testng-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

[⇜Krishnan Mahadevan⇝]

Here's how you can do the following :

>>>>>> Control the conditions within the test method. (I think the argument I specified above does this)

This is going to be tricky. You can very well control this via JVM arguments, but the problem there is that it would always be one global switch for your entire execution.

>>>>> Control the entire test method. (Todo)

>>>> Control the entire test class. (Todo)

 Use an implementation of org.testng.IAnnotationTransformer, wherein you can enable/disable a particular test method based on some conditions (for e.g., the name pattern of the test ?). This would be a listener which you would wire in using the <listeners> tag or via a Service Loader implementation. To learn about listeners in general take a look at my blog post here.

The other way would be to work with groups and then either use a beanshell (as a method selector) to determine which of your tests should be executed. Take a look at this blog post of mine, to learn how to work with beanshell and how it is used to select groups dynamically.

>>> Control the entire test suite. (Todo. May be when one suite references other sub suites, and we want to enable/disable some)

You can do this via an implementation of org.testng.IAlterSuiteListener (this again is another TestNG listener), wherein you can add/modify/remove <suite> (or) <test> in runtime based on some condition. For more details on this, refer to my blog post here.

There are again much more ways of doing this. The options that I listed are some of the most common and straight forward ones.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribbings @ http://rationaleemotions.wordpress.com/

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users+unsubscribe@googlegroups.com.

[Todd Bradley]

On Thu, Oct 26, 2017 at 11:08 PM, Rajdeep Biswas <raj.my...@gmail.com> wrote:

Thank you Todd. Do you mean that you have created suites and grouped methods inside them and pass during runtime? Can you give a sample or any link of this where one can turn on/off certain groups in the suite, using arguments, for example from the CI. 

Like, running a suite with certain group once and another group included in different run.

Yes, that's what we do. No, I can't share my source code with you. But we use the IAlterSuiteListener that Krishnan talked about in his reply. My approach is to use the IAlterSuiteListener to determine which test groups to exclude at runtime, based on characteristics of the system under test (such as which feature-toggled features are enabled at the moment). So far, this seems pretty maintainable.

So, imagine that my product has features X, Y, and Z that can be independently enabled or disabled. I have test groups called X-enabled, Y-enabled, and Z-enabled. Then, my IAlterSuiteListener determines which features are not enabled right now (let's say X and Z are enabled, and Y is diabled), and then excludes test groups for disabled features (exclude Y-enabled) before any tests start running.

Todd.

[Rajdeep Biswas]

Thanks Todd and Krishna for the prompt help. I have put forward these to team:

1. using listener and groups, and run time disabling of groups with maven arg, for example

-DexcludeGroups=group1,group2

2. or, just use with suites, creating respective suites (static).

Let me see what the opine.

Thanks