Change different parameters for a test for each run

[Eyal]

Hi,

How can I change different parameters in test? not just 1 parameters, but a lot of diff params for each test invocation count.\

Possible for all the class also ?

Thanks.

[⇜Krishnan Mahadevan⇝]

Can you please share a sample which would explain what you are looking for?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/880bac57-571e-456b-8ea5-7039ceb43ab8n%40googlegroups.com.

[Eyal]

Hi  Krishnan ,

actually no need for code, My question is about the way of doing that:

I have tests that cab get some parameters from config.properties file and also using
IAnnotationTransformer to control invocation count / threadpoolsize

My question is:
How can I tell the class/tests to get each iteration(invocation count) different parameters 

ב-יום שני, 29 בנובמבר 2021 בשעה 14:46:26 UTC+2, Krishnan Mahadevan כתב/ה:

[Eyal]

Because data provider is for 1 test with 1 parameter, Here it's lot of parameters for 1 class or at least for 1 test.

ב-יום שני, 29 בנובמבר 2021 בשעה 14:52:14 UTC+2, Eyal כתב/ה:

[⇜Krishnan Mahadevan⇝]

Why not couple your data provider with a "@Factory" annotation ? Then the test class will have multiple instances, one parameter will be bound to one instance and then internally you can have test methods use that data set.

You could take this to the next level, wherein your Factory + data provider would basically work with different data sources and then your data provider within your test class would read data from that data source and have your test methods iterate n number of times ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/c01c072a-0026-42d0-8330-4594279cf81dn%40googlegroups.com.

[Eyal]

Sounds very good, never did that, so It's sound very very complicated :(

ב-יום שני, 29 בנובמבר 2021 בשעה 14:55:34 UTC+2, Krishnan Mahadevan כתב/ה:

[Krishnan Mahadevan]

Here’s a full-fledged sample along with required javadocs to help you understand what I am referring to.

 

import com.google.gson.JsonElement;

import com.google.gson.JsonParser;

import java.io.FileReader;

import java.io.IOException;

import java.util.HashMap;

import java.util.Map;

import java.util.Properties;

import java.util.concurrent.atomic.AtomicInteger;

import org.testng.annotations.DataProvider;

import org.testng.annotations.Factory;

import org.testng.annotations.Test;

 

public class ExampleFactory {

 

  private final Map<String, String> data = new HashMap<>();

  private final String fileName;

 

  /**

   * This constructor accepts the file type and the file name from which test data should be read.

   *

   * @param fileName - The relative location of the file.

   * @param fileType - A {@link FileType} that represents the type of the file.

   * @throws IOException - In case of problems

   */

  @Factory(dataProvider = "files")

  public ExampleFactory(String fileName, FileType fileType) throws IOException {

    this.fileName = fileName;

    data.putAll(fileType.load(fileName));

  }

 

  /**

   * This static data provider basically lets TestNG create Test class instances. It passes the

   * location of the file and its type that a test class instance should use to basically have its

   * own data provider run iterations based off of.

   *

   * @return - A 2D array of objects

   */

  @DataProvider(name = "files")

  public static Object[][] files() {

    return new Object[][]{

        {"src/test/resources/props.json", FileType.JSON},

        {"src/test/resources/props.properties", FileType.PROPERTIES},

    };

  }

 

  /**

   * A regular test method

   */

  @Test(dataProvider = "testData")

  public void testMethod(String k, String v) {

    System.err.println(fileName + " => (" + k + ", " + v + ")");

  }

 

  /**

   * A regular data provider which converts a {@link HashMap} into a 2D array wherein every array

   * consists of key/value pair.

   */

  @DataProvider(name = "testData")

  public Object[][] testData() {

    Object[][] result = new Object[data.size()][2];

    AtomicInteger counter = new AtomicInteger(0);

    data.forEach((k, v) -> {

      result[counter.get()][0] = k;

      result[counter.get()][1] = v;

      counter.incrementAndGet();

    });

    return result;

  }

 

  /**

   * An enum that represents the supported file types.

   */

  enum FileType implements DataLoader {

    JSON {

      @Override

      public Map<String, String> load(String fileName) throws IOException {

        JsonParser parser = new JsonParser();

        JsonElement parsed = parser.parse(new FileReader(fileName));

        Map<String, String> result = new HashMap<>();

        parsed.getAsJsonObject().entrySet()

            .forEach(each -> result.put(each.getKey(), each.getValue().getAsString()));

        return result;

      }

    },

    PROPERTIES {

      @Override

      public Map<String, String> load(String fileName) throws IOException {

        Properties properties = new Properties();

        properties.load(new FileReader(fileName));

        Map<String, String> result = new HashMap<>();

        properties.forEach((key, value) -> result.put(key.toString(), value.toString()));

        return result;

      }

    }

 

  }

 

  /**

   * Represents the capabilities to be possed by a data source loader.

   */

  interface DataLoader {

 

    /**

     * @param fileName - The relative location of the file from the file system from which data is

     *                 to be read from.

     * @return - A {@link Map} of the read data.

     * @throws IOException - In case of problems.

     */

    Map<String, String> load(String fileName) throws IOException;

  }

 

}

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/d2f9b81f-e7be-46d5-be87-78162fb26a60n%40googlegroups.com.

[Eyal]

This is not What I meant,

I will explain:

I want to run the class every time with different invocation count and threal pool size values,

along with it I want to pass diff values for more functions.

Seems that control class run is not possible in testng.

Thanks

ב-יום שלישי, 30 בנובמבר 2021 בשעה 06:13:41 UTC+2, Krishnan Mahadevan כתב/ה:

[Eyal]

I need that the base class will be able to change invocation count, thread pool, parameters, then I can extend from it,
but it doesn't supports the @BeforeClass

ב-יום ראשון, 5 בדצמבר 2021 בשעה 09:42:58 UTC+2, Eyal כתב/ה:

[⇜Krishnan Mahadevan⇝]

For you to be able to change invocation count, thread pool etc you can use an IAnnotationTransformer.

To be able to add parameters to a suite dynamically based on a data source, you can make use of an IAlterSuiteListener implementation.

Here's the order in which you do it.

1. Create a suite file that does not contain any test class information.
2. Define a JVM argument via which you can retrieve the test class package information to be included in the suite file.
3. Create an implementation of IAlterSuiteListener and include it in the suite file from (1).
4. Within the IAlterSuiteListener implementation define the logic that can retrieve data from a data source and add it as parameters to your suite file
5. Define an IAnnotationTransformer implementation that will read the annotations and change whatever you need and include this as well in your suite file.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/e5eb86be-efc6-4bd5-acc9-b33664699754n%40googlegroups.com.

[Eyal]

Since my test is too complicated I have tried diff methods but fails.

I am doing this:
1. Implement AnnotationTransformer:

public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {

if (testMethod.getName().equals("RepetitionLoadTest")) {
if (j == 1) {
invocation = Integer.parseInt(ReadPropertyFile("Invocation1"));
threads = Integer.parseInt(ReadPropertyFile("threads1"));
annotation.setInvocationCount(invocation);
annotation.setThreadPoolSize(threads);
} else if (j == 2) {
invocation = Integer.parseInt(ReadPropertyFile("Invocation2"));
threads = Integer.parseInt(ReadPropertyFile("threads2"));
annotation.setInvocationCount(invocation);
annotation.setThreadPoolSize(threads);
} else if (j == 3) {
invocation = Integer.parseInt(ReadPropertyFile("Invocation3"));
threads = Integer.parseInt(ReadPropertyFile("threads3"));
annotation.setInvocationCount(invocation);
annotation.setThreadPoolSize(threads);
} else if (j == 4) {
invocation = Integer.parseInt(ReadPropertyFile("Invocation4"));
threads = Integer.parseInt(ReadPropertyFile("threads4"));
annotation.setInvocationCount(invocation);
annotation.setThreadPoolSize(threads);
}

this way I can tell each iteration to run with data provider and provide other settings.
Issue is that I can't tell the method name to be change every iteration, so The counter is not usefull.

What else can I do to change method name ?

Thanks

ב-יום שני, 6 בדצמבר 2021 בשעה 05:14:20 UTC+2, Krishnan Mahadevan כתב/ה:

[Eyal]

The needs:

need to get some special identifier to be changed in runtime in the method/test.
Why ? because this test should run x times, every time it should use another invocation count, another threadpool size, another value of some connection to DB,.

I don't want to copy the method x times and change the name mannually for each.

Any relevant help can be life rescue.

Thanks

ב-יום רביעי, 8 בדצמבר 2021 בשעה 21:29:22 UTC+2, Eyal כתב/ה:

[Eyal]

variable to be passed to each test     Invocation count for each test     threadpool size for each test

2000                                                                     20                                                          4

2000                                                                     20                                                          8

2000                                                                     20                                                          10

As you can see, The same test should be run every time with other invocation count, thread pool, other value according to this table above.

I only managed to implement the I transform class to change the invocation count and thread pool, But I can't tell the code: If (method name == x) because method name can't be changed in runtime, therefor I don't have a way to implement this table in any way.

Thanks

ב-יום רביעי, 8 בדצמבר 2021 בשעה 21:36:00 UTC+2, Eyal כתב/ה:

[Eyal]

 If IAnnotationTransformer had some ability to change method name or test name in real time and
get the new method name and do stuff accordingly that could be amazing. 

ב-יום רביעי, 8 בדצמבר 2021 בשעה 22:17:13 UTC+2, Eyal כתב/ה:

[Eyal]

any help will be very appriicate!

ב-יום רביעי, 8 בדצמבר 2021 בשעה 22:19:34 UTC+2, Eyal כתב/ה:

[⇜Krishnan Mahadevan⇝]

I have tried going back to the email thread to read through and understand the expectation, but I am not able to get the gist of what you are trying to do.

It looks like you are trying to use TestNG to build performance tests. This is what I have understood on a high level.

So can you please summarize your problem statement with as much details as possible once again?

Feel free to include pseudo code, test data, diagrams or anything that will help people in this group understand the problem statement.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/e5e74f16-f09e-414d-a647-48ef03b0bd41n%40googlegroups.com.

[Eyal]

Hi.

You are right it is indeed Performance test with Testng.

So I will summarize and will put here more code:

1. I have implemented the IAnnotationTransformer class like this:

@SneakyThrows
@Override

public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {

if (testMethod.getName().equals("RepetitionLoadTest01")) {
throughput = Integer.parseInt(ReadPropertyFile("throughput1"));
throughputList.add(throughput);

invocation = Integer.parseInt(ReadPropertyFile("Invocation1"));

invocationList.add(invocation);

threads = Integer.parseInt(ReadPropertyFile("threads1"));

threadsList.add(threads);
annotation.setInvocationCount(invocation);
annotation.setThreadPoolSize(threads);
SetThroughput(throughput);

} else if (testMethod.getName().equals("RepetitionLoadTest02")) {
throughput = Integer.parseInt(ReadPropertyFile("throughput2"));
throughputList.add(throughput);

invocation = Integer.parseInt(ReadPropertyFile("Invocation2"));

invocationList.add(invocation);

threads = Integer.parseInt(ReadPropertyFile("threads2"));

threadsList.add(threads);
annotation.setInvocationCount(invocation);
annotation.setThreadPoolSize(threads);
SetThroughput(throughput);

}

and so on.... Until 18.

2. The issue: I had to copy the test function 18 times in order it to work, Why? because I didn't have anyway to change the test dynamically that the IAnnotationTransformer can know that same function is now with id x for example.

So basically I want a way that the function will be written only once but I will be able to change the invocation count, thread pool, other values from one place to run it as much as I need and 

for that I guess I must have some unique identifier for the test method that can be set dynamically somehow, and that the IAnnotationTransformer will be fast enough to catch it and change annotation before the run.

Thanks.

ב-יום שלישי, 14 בדצמבר 2021 בשעה 14:08:30 UTC+2, Krishnan Mahadevan כתב/ה:

[⇜Krishnan Mahadevan⇝]

Ok. I think you need to revisit the strategy here. I say this because:

1. AFAIK, there is no way in which one can inject methods into a loaded class at runtime (Not without a lot of byte code manipulation etc.,)
2. AFAIK, TestNG exposes ONLY those methods available in a class, when it is at the phase of annotation transformation.

So you should perhaps look at building something which involves using the TestNG APIs to orchestrate your execution.

By this what I mean is that, 

• Given you have a test class that has 1 or more test methods which you would like to run in performance mode.
• So within a main method, you should create a TestNG object per perf combination but all of them can perhaps share a threadsafe reporting mechanism which can capture the test results.
• Within each TestNG object, you orchestrate your tests such that it fits your perf model.
• The last line in the main method basically collates the test results across the TestNG run executions and presents it properly.
• Leverage an appropriate maven plugin (exec plugin) or a create a custom task (if in Gradle) or a custom maven plugin to orchestrate your tests.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/c944b66a-baba-4ec0-9423-e4b34cb32cban%40googlegroups.com.

[Eyal]

What about any kind of a loop that waits in the IAnnotationTransformer class and will do some manipulation ?

ב-יום שלישי, 14 בדצמבר 2021 בשעה 15:42:59 UTC+2, Krishnan Mahadevan כתב/ה:

[⇜Krishnan Mahadevan⇝]

An IAnnotationTransformer is agnostic of anything related to the TestNG execution parameters when it starts working at changing the attributes of an annotation.

Its instantiation is managed by TestNG if you are NOT using the TestNG APIs. So I am not sure what you are referring to when you say any kind of loop.

Customisation of the IAnnotationTransformer can be done, if you instantiate it with the required parameters that it needs for it to behave dynamically and that an be done ONLY when you use TestNG APIs (programmatic execution) and not when you execute TestNG tests via build tools integration (surefire plugin for e.g.,)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/3e1a2172-0050-44b5-b355-8696901b6ba5n%40googlegroups.com.

[Eyal]

That is a very series problem, do you familiar with framework that can do it ?

ב-יום שלישי, 14 בדצמבר 2021 בשעה 16:00:27 UTC+2, Krishnan Mahadevan כתב/ה:

[Eyal]

if I have  exactly 18 secnarios, can the invocation / threadpool/ other value can be injected with data provider / factory annotation ?

ב-יום שלישי, 14 בדצמבר 2021 בשעה 16:54:44 UTC+2, Eyal כתב/ה:

[⇜Krishnan Mahadevan⇝]

> That is a very serious problem, are you familiar with a framework that can do it ?

Why would it be a serious problem? AFAIK, TestNG was not designed for doubling up as perf tests and that perhaps explains why the annotation parsing/transformation is agnostic of the TestNG execution parameters. 

I don't have personal experience but it looks like you need something like Taurus for running your TestNG tests in performance mode.

See here: https://gettaurus.org/docs/TestNG/

> if I have  exactly 18 scenarios, can the invocation / threadpool/ other value be injected with data provider / factory annotation ?

We will be back to the original problem statement which is, 

The attributes of an annotation can ONLY be altered via an IAnnotationTransformer implementation and this implementation gets kicked off much earlier before the data provider or factories even come into picture. So No, you would not be able to do this with either of them.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/a10ef7c4-f3ec-463d-be23-deaeb2a70dc6n%40googlegroups.com.