A programmatic way to use SuiteRunner or TestRunner directly

[Truong Nguyen]

Hi testng team,

I know that TestNG class is the entry point for testng execution. However, I wonder can we actually bypass this entry point and use the two mentioned classes directly.

My intention is about to reuse part of my existing test classes by create XmlTest programmatically, then by TestRunner directly, I can have some orchestration load test for either by time or target load. My interest is not about result but about generating target load scenarios with:

1. the very good know how to appropriately invoke a test method by getting knowledge about configuration method, grouping, dependency.

2. doesn't need any capability of test listener actually.

Thanks,

Travis

[⇜Krishnan Mahadevan⇝]

I dont think this capability exists as of today.

SuiteRunner and TestRunners are internals of TestNG and AFAIK they are not exposed outside.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/7fe295db-134c-4804-aa7c-06abc35ad715n%40googlegroups.com.

[Truong Nguyen]

Thanks,

Just in case I keen on the purpose I have, do you have any idea for getting that done? Like the flow I need to manually hand-wiring before I can use an instance from the classes?

Regards,

Travis

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/CANikZLknqZgCYCQ%3DmkKGUPS4MC-ikDnU3nwpGYU0fyeEVObHvw%40mail.gmail.com.

[Baubak]

Hi Travis,

I do a lot of tets prrogrammatially, but I always go through the TestNG class.

However if you just want to fetch the test information, you can do it in two passes:

- Pass one : knowledge extraction :  You load the tests in TestNG. In the IAnnotationTansformer, you fetch the data but disable the tests

- Pass two : you run the tests....

If I have misunderstood your question, then please disregard my comment.

Best regards,

Baubak

[⇜Krishnan Mahadevan⇝]

Travis,

Can you please help explain a bit more on the exact details of your use cases so that we can check what can be done at the best here ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/2cd82b34-1591-4e44-815a-4256733e425bn%40googlegroups.com.

[Amr Salem]

Is there way to pass Vm options  in xml testng file?

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/CANikZLknqZgCYCQ%3DmkKGUPS4MC-ikDnU3nwpGYU0fyeEVObHvw%40mail.gmail.com.

[Truong Nguyen]

I can specify more detail about my intention as below. My test project has built with quite a number of classes. Test information is annotated across classes with priority, grouping, dependencies. From those classes, I want to form two test suites represent two user behaviors:

Basic scenarios

<test name="Moderate scenarios">

  <classes>

   <class name="class with action A" />

   <class name="class with action B" />

   <class name="class with action C" />

  </classes>

</test>

Advanced scenarios

<test name="Advanced scenarios">

  <classes>

   <class name="class with action A" />

   <class name="class with action B" />

   <class name="class with action C" />

   <class name="class with action D" />

   <class name="class with action F" />

  </classes>

</test>

Let's say from an entry point from my program, I have a method to control load over Number of concurrent basic and advanced scenarios over a time period.

int basicConcurrentCount= 80;

int advancedConcurrentCount = 20;

while (time constrains) && (basic < basicConcurrentCount) && (advanced < advancedConcurrentCount) {

do stuff here to increase conncurent count for basic and advanced via TestRunner // SuiteRunner ?

}

Regards,

Travis

[⇜Krishnan Mahadevan⇝]

Travis,

I would suggest that you explore the TestNG listener IAlterSuiteListener, which basically gives the ability to dynamically alter suites.

So you could on a high level do the following:

1. Provide a mechanism to discover your test classes (perhaps via a VM argument or via some data source such as xml/json/csv etc)

2. Build an implementation of IAlterSuiteListener that is wired in as a mandatory listener using a Service Provider mechanism.

3. Within this implementation from (2), you start reading the discovery source from (1) and then weave in your suite as you desire. Since you now have the option to construct your suite or your test tag, you literally have the ability to decide your concurrency etc.,

Hope that helps.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/34f47d69-d542-4303-808d-b0d7739484c3n%40googlegroups.com.

[Truong Nguyen]

I used suite altering before and actually it was my go-to way to mark invalid, flaky test from an external source.

There are my questions for your suggestion:

1. Once the testng instance call run() - I don't know which testng API to get some kind of current execution observable metric from the run: Let's say:

 - how many basic flow is running?

2. Just in case the concurrent count is deceased due to finish test, I also don't have any API to pump it up again.

So from my understanding, if I can have an instance of either suite runner or test runner,  I can make orchestration around a swamp of instannses to achieve target concurrent count for time-based scenario. For example: start around 10% target for 1 hour, then pump to 80% for next 2 hours, then keep around 50% for rest of the day.

vutruo...@gmail.com | (+84) 909-037-769

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/CANikZLkh6CDPDHNpoknN4FidAexeWraO%2BfKAfrdRR6F-KmH96w%40mail.gmail.com.

[Truong Nguyen]

With above assumption and target, you can assume that I only need 1 thread per suite runner / test runner. The whole test tag or suite tag is my simulation scenario. The extra sauce here come from my code to orchestrate a vast number of instances to as close as target concurrent level.

[⇜Krishnan Mahadevan⇝]

Travis,

>>>> 1. Once the testng instance call run() - I don't know which testng API to get some kind of current execution observable metric from the run: Let's say:

 - how many basic flow is running?

You can figure this out by having a "round about mechanism of doing this". For e.g., you could club all the basic tests into the same <test> tag. And assuming that your tests are neither data driven nor invocation count driven, it would be a simple problem of finding out the number of "@Test" annotated tests by trying to query the ITestContext.getAllTestMethods().

You could get even a bit more fancier by using an annotation transformer that adds some sort of a custom attribute to the "@Test" annotation via the annotation transformer using org.testng.annotations.ITestAnnotation#setAttributes which will tell you if a particular test belongs to which category.

So now you would have all the tests being added to a category via the annotation transformer,

You would have the count of all the tests from within the ITestContext

Now you can implement a running counter which says "test x/y tests currently running belonging to the category ABC".

>>> 2. Just in case the concurrent count is deceased due to finish test, I also don't have any API to pump it up again.

Under what circumstances is this going to happen ? Do you have an example that can exactly tell me when and how this would be relevant ? I ask this because I fail to understand the actual benefit of you trying to alter your threadpool size based on the tasks that remain to be executed, from a test execution perspective. And also please feel free to elaborate more on what do you mean by this concurrency count and when do you expect them to decrease/increase and what are you trying to do when this state changes ?

You are NOT expected to have any API because concurrency count is supposed to be static in nature per <test> tag and NOT dynamic. TestNG DOES NOT work with a thread pool whose size is dynamic. I am not sure if there's one such thread pool in JDK that supports this except for a CachedThreadPool as explained in this SO question : https://stackoverflow.com/a/11250066 

I have never had a use case to work with this variant so I don't know a lot. But altering the thread pool size of TestNG is meant to be done only via the exposed mechanisms which is the thread-count attribute of a test tag and once that has been set, it stays relevant for the lifetime of that test tag.

>>>> So from my understanding, if I can have an instance of either suite runner or test runner,  I can make orchestration around a swamp of instannses to achieve target concurrent count for time-based scenario. For example: start around 10% target for 1 hour, then pump to 80% for next 2 hours, then keep around 50% for rest of the day.

I am not quite sure if I understand what is the use case that you are trying to solve here. Are you trying to use TestNG to build some sort of a performance test orchestration mechanism?

Like I mentioned before, TestNG in its current state does not expose the creation of a suite runner or a Test runner for an end-user's perspective since they are internals of TestNG which can and will change over a period of time.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/ef38e557-3ce9-4322-b938-3f53a7453795n%40googlegroups.com.

[travis.n...@gmail.com]

"

...

Are you trying to use TestNG to build some sort of a performance test orchestration mechanism?

...

"

This is my intention up to now. Since testng knows how to wire dependency between tests (via annotations), I want to reuse logic by those annotations across test classes. However, I also need a capability to control concurrent count.

Regards,

Travis

[⇜Krishnan Mahadevan⇝]

Travis,

>>>> This is my intention up to now. Since testng knows how to wire dependency between tests (via annotations), I want to reuse logic by those annotations across test classes. However, I also need a capability to control concurrent count.

You should have started with that :)

Either way, I don't think we are going to arrive at a consensus on this on the mailing list.

Lets continue the discussion in this Github issue that I created on your behalf : https://github.com/cbeust/testng/issues/2632

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/7c33ba7a-3f24-4123-b4a8-785e0becbe68n%40googlegroups.com.