Parallel @DataProvider use too much hard memory

Никита Грицкевич

unread,

Nov 16, 2023, 7:28:38 AM11/16/23

to testng-users

Hi testNG group,

I have a situation, our team have one test method and one data provider size of 2048x11, where each element our data provider matrix accept 1 or 0. We run test on the remote kubernetes klaster pod and this test run too many time (more than 24 hours). Okey, we try to use parallel data provider with data-provider-thread-count="2". Our remote pod is a Linux machine with 4 Gib hard memory. Can you explain, why machine loses free hard memory in geometric progression? (The test is UI (use Selenide) with API calls (use Rest Assured). In code, we don't have any synchronization for code blocks).

Grafana graphics:

data-provider-thread-count="2"

data-provider-thread-count="5"

⇜Krishnan Mahadevan⇝

unread,

Nov 16, 2023, 7:31:26 AM11/16/23

to testng...@googlegroups.com

You would need to share a bit more details than what you already have.

What version of TestNG are you using? [ Latest versions of TestNG 7.5.1 (JDK8), 7.8.0(JDK11) ]
The graphs that you shared, is that the memory graph of the JVM wherein the tests are being run?
What does your test code look like?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/70be1a96-465a-48c6-8374-f02eaad738d4n%40googlegroups.com.

Никита Грицкевич

unread,

Nov 16, 2023, 7:53:10 AM11/16/23

to testng-users

TestNg version 7.4.0.
Graph of the common memory use of a pod.
I can't share my code, but I can describe conceptually about my code:

Log in to the page
Send Get request for getting api model
Extract all necessary inputs for filling UI panel
Based on the number of input I generate truth table, where 1 -> need to fill input, 0 -> input empty
Open necessary panel
On passed arguments from the data provider and inputs type, fill inputs and saved.

четверг, 16 ноября 2023 г. в 15:31:26 UTC+3, Krishnan Mahadevan:

⇜Krishnan Mahadevan⇝

unread,

Nov 16, 2023, 8:07:03 AM11/16/23

to testng...@googlegroups.com

You are not on the latest version of TestNG.

Can you please upgrade to TestNG 7.8.0 and see if that helps?

Graph of the common memory use of a pod.

Is your test running in this pod or is this running somewhere else? I am basically trying to find out if you are sharing the memory consumption of your application under test (or) if you are sharing the memory consumption of the JVM from where the test is running (For e.g., Jenkins build).

I can't share my code, but I can describe conceptually about my code:

It's not going to be possible to debug issues with conceptual code because it doesn't show what your code is doing or what your memory footprint looks like.

We would at least need to see what your data provider looks like (That's the only part that is of interest from TestNG perspective)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/7566d977-f71f-4636-8ab7-f29ddd65db47n%40googlegroups.com.

Cédric Beust ♚

unread,

Nov 16, 2023, 12:00:52 PM11/16/23

to testng...@googlegroups.com

Side question: have you tried returning an `Iterator` from your data provider?

--

Cédric

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/7566d977-f71f-4636-8ab7-f29ddd65db47n%40googlegroups.com.

Никита Грицкевич

unread,

Nov 17, 2023, 10:16:58 AM11/17/23

to testng-users

1) After update to the last testNG version change nothing
2) Pod created in test stage and using only for our test
3) I can to share example data provider method generating:
int countInputs = 11;
int combinationsCount = 1 << countInputs; // 2^countInputs
Object[][] truthTable = new Object[combinationsCount][countInputs];
for (int i = 0; i < combinationsCount; i++) {
for (int j = 0; j < countInputs; j++) {
truthTable[i][j] = (i & (1 << j)) != 0; // i /(2^j) % 2;
}
}
return truthTable;

4) Cédric, Thnx, try return Iterator<Object[]>

OPTIONAL: after decrease data-provider matrix to 8^2 x 8 hard memory use became linear

четверг, 16 ноября 2023 г. в 20:00:52 UTC+3, Cedric Beust:

⇜Krishnan Mahadevan⇝

unread,

Nov 19, 2023, 1:54:38 AM11/19/23

to testng...@googlegroups.com

I am guessing that you upgraded to TestNG v7.8.0 and are still experiencing the same problem.

But it's still not clear as to where your memory is bloating. So I created a simple image that should help us clarify the ambiguities.

%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22group%22%20vertex%3D%221%22%20connectable%3D%220%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%2220%22%20y%3D%22240%22%20width%3D%22670%22%20height%3D%22120%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%223%22%20value%3D%22%26lt%3Bfont%20style%3D%26quot%3Bfont-size%3A%2025px%3B%26quot%3B%26gt%3BJVM-1%26lt%3B%2Ffont%26gt%3B%22%20style%3D%22rounded%3D1%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%222%22%3E%3CmxGeometry%20x%3D%2260%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%224%22%20value%3D%22%26lt%3Bfont%20style%3D%26quot%3Bfont-size%3A%2025px%3B%26quot%3B%26gt%3BJVM-2%26lt%3B%2Ffont%26gt%3B%22%20style%3D%22rounded%3D1%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%222%22%3E%3CmxGeometry%20x%3D%22480%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%225%22%20value%3D%22%26lt%3Bfont%20style%3D%26quot%3Bfont-size%3A%2019px%3B%26quot%3B%26gt%3BTest-case%20is%20running%20here%26lt%3B%2Ffont%26gt%3B%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%222%22%3E%3CmxGeometry%20y%3D%2270%22%20width%3D%22230%22%20height%3D%2240%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%226%22%20value%3D%22%26lt%3Bfont%20style%3D%26quot%3Bfont-size%3A%2019px%3B%26quot%3B%26gt%3BApplication%20Under%20Test%20%26lt%3Bbr%26gt%3Bis%20running%20here%26lt%3B%2Ffont%26gt%3B%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3B%22%20vertex%3D%221%22%20parent%3D%222%22%3E%3CmxGeometry%20x%3D%22440%22%20y%3D%2270%22%20width%3D%22230%22%20height%3D%2250%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%227%22%20value%3D%22%22%20style%3D%22shape%3DflexArrow%3BendArrow%3Dclassic%3Bhtml%3D1%3Brounded%3D0%3BexitX%3D1%3BexitY%3D0.5%3BexitDx%3D0%3BexitDy%3D0%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3B%22%20edge%3D%221%22%20parent%3D%222%22%20source%3D%223%22%20target%3D%224%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22370%22%20y%3D%22120%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22420%22%20y%3D%2270%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E

In the above figure, are you experiencing a memory bloat in:

JVM-1 (or)
JVM-2

If you are experiencing bloating in JVM-1 then can try one of the following to see if it would help:

As Cedric mentioned earlier, transform your data provider method to return Iterator<Object[]>. But remember that for this to be truly lazy loading you should maybe create your own implementation of Iterator, which would fetch data from a data source everytime the hasNext()/next() is invoked. I say this because in your example code that you shared, you are pre-creating the large 2D array in one shot and so that memory is already allocated and occupied in the heap. You can refer to this article for some tips on how to do this: https://www.baeldung.com/java-creating-custom-iterator
Run the test again by setting this JVM argument [ -Dtestng.memory.friendly=true ]. Since the number of test methods is large here, TestNG by default remembers the java lang backed Method object for each and every iteration. Setting this argument would prevent TestNG from doing so and can bring down the memory footprint. [ List of all JVM arguments honoured by TestNG is available here ]
If both (1) and (2) still don't help, then you would need to do one of the following:

Increase the max heap memory size via -Xmx (or)
Reduce the number of iterations to a lesser value and maybe spawn multiple instances of your test (i.e., duplicate JVM-1 multiple times)

If you are noticing a bloat in memory for JVM-2, then there's not much that can be done from the TestNG side. You would need to increase the Xmx memory for the JVM of your application under test, or reduce the test volume i.e., apply 3.1 (or) 3.2 on your application under test.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/86c8b033-10a3-493a-a959-40529e8f6bben%40googlegroups.com.

Никита Грицкевич

unread,

Nov 20, 2023, 6:31:13 AM11/20/23

to testng-users

Thanks for your time and advices, Iterator<Object[]> really helps me, but if you have time, can you explain, what difference in using memory between using Object[][] and Iterator<Object[]> in data provider. But after the fix I find suspect situation in console, If I use parallel data-provider, test method index in console duplicated, but data from data-provider are correct.

воскресенье, 19 ноября 2023 г. в 09:54:38 UTC+3, Krishnan Mahadevan:

⇜Krishnan Mahadevan⇝

unread,

Nov 20, 2023, 8:21:15 AM11/20/23

to testng...@googlegroups.com

Can you explain, what difference in using memory between using Object[][] and Iterator<Object[]> in data provider.

When you work with a data provider that returns a 2D Object array ( Object[][] ), TestNG calls the data provider and the data provider ends up creating the entire 2D array in 1 shot. So for example, lets say your data driven test method accepts 1 integer as parameter, and you return 10,000 rows, so here we end up occupying memory worth 10,000 * 4bytes = 40kb even before the test starts running.

If you leverage a data driven test method that is powered by a data provider which returns Iterator<Object[]> then you aren't allocating that much memory in one shot. TestNG gets the Iterator and for each of the 10,000 iterations, you will consume 4bytes incrementally.

Depending upon your reporting requirements, there's a very good chance that you will still end up with a bloated memory, because TestNG remembers all the parameters of all the test methods (along with the corresponding java.lang.reflect.Method objects) for all your 10,000 iterations and so, by the time your test execution is done and you are in your reporting phase, you may still hit an OutOfMemoryError.

The JVM argument that I shared in my previous post, will atleast avoid taking the extra memory occupied by each of the 10k java.lang.reflect.Method references (It might not be an issue because it's the same method reference in all the 10k invocations, but I hope you get the idea)

But after the fix I find suspect situation in console, If I use parallel data-provider, test method index in console duplicated, but data from data-provider are correct.

That may be an issue with the IDE UI (It looks to me that you are using IntelliJ from that screenshot). You may have to file an issue with Jetbrains in that case.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/a5a82a1a-5a50-4c06-9377-24130f81ad54n%40googlegroups.com.

Никита Грицкевич

unread,

Nov 20, 2023, 8:49:14 AM11/20/23

to testng-users

1) Thnx for your explanation, I'll hope that I all correctly catch, very informative answer, really thnx.
2) It's InteljIDE, but, when I run through the git lab pipeline, In gitlab pipeline console the same problem

понедельник, 20 ноября 2023 г. в 16:21:15 UTC+3, Krishnan Mahadevan:

⇜Krishnan Mahadevan⇝

unread,

Nov 21, 2023, 2:19:57 AM11/21/23

to testng...@googlegroups.com

2) It's InteljIDE, but, when I run through the git lab pipeline, In gitlab pipeline console the same problem

Can you please share a simple sample that I can use to reproduce this behaviour ?

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/d7bd30b1-6347-4cfb-8f61-de6677a25aa7n%40googlegroups.com.

Никита Грицкевич

unread,

Nov 21, 2023, 6:30:56 AM11/21/23

to testng-users

Yeah, sure.

@DataProvider(parallel = true)
public Iterator<Object[]> get() {
int count = 20;
int combinationsCount = 1 << count; // 2^countInputs
Object[][] truthTable = new Object[combinationsCount][count];

for (int i = 0; i < combinationsCount; i++) {

for (int j = 0; j < count; j++) {

truthTable[i][j] = (i & (1 << j)) != 0; // i /(2^j) % 2;
}
}

return Arrays.stream(truthTable).iterator();
}

@Test(dataProvider = "get")
public void test(Boolean...args) {
System.out.println(Arrays.toString(args));
}

data-provider-thread-count=3

вторник, 21 ноября 2023 г. в 10:19:57 UTC+3, Krishnan Mahadevan:

⇜Krishnan Mahadevan⇝

unread,

Nov 21, 2023, 8:18:50 AM11/21/23

to testng...@googlegroups.com

I reduced the iterations so that its easier to see whats going on.

Can you please help point me to where the problem is, in the below screenshot? It looks fine to me (I mean the method indexes)

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/f7bfe146-33eb-4846-a87e-2f1ce857dd22n%40googlegroups.com.

Никита Грицкевич

unread,

Nov 21, 2023, 12:00:10 PM11/21/23

to testng-users

Please, try to increase count variable to e.g. 11 or 20 (need more huge data Provider returning value size)

вторник, 21 ноября 2023 г. в 16:18:50 UTC+3, Krishnan Mahadevan:

⇜Krishnan Mahadevan⇝

unread,

Nov 22, 2023, 8:28:05 AM11/22/23

to testng...@googlegroups.com

Here's the output with count as 10.

Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/

My Technical Scribblings @ https://rationaleemotions.com/

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/ba460114-6d8d-4c12-8b7e-2432e66022een%40googlegroups.com.

Никита Грицкевич

unread,

Nov 23, 2023, 4:26:49 AM11/23/23

to testng-users

Hm, can you share your data-provider-thread-count?

среда, 22 ноября 2023 г. в 16:28:05 UTC+3, Krishnan Mahadevan:

Krishnan Mahadevan

unread,

Nov 23, 2023, 6:31:03 AM11/23/23

to testng...@googlegroups.com

I am running with data provider thread count as “3”.

I hope this below screenshot answers all the questions

To view this discussion on the web visit https://groups.google.com/d/msgid/testng-users/c8038999-9446-4e77-a3d1-2728b52ce53bn%40googlegroups.com.

Никита Грицкевич

unread,

Nov 23, 2023, 12:06:18 PM11/23/23

to testng-users

Krishan, okay, thanks for your answers!

четверг, 23 ноября 2023 г. в 14:31:03 UTC+3, Krishnan Mahadevan:

Reply all

Reply to author

Forward