Thread count and invocation count
OR
We are using TestNG to run stress tests on our server. We are
consdering using the thead count attribute with the invocation count.
For example, we are trying to run 100 threads with invocation count set
to 1000 (10 times the thread count) in order to simulate 100
simultaneous users.
However, I may have misunderstood what means the thread count. In the
generated report, I didn't find all threads. 5 threads seem to be
launched by TestNG. However, in the suite we are declaring 100 threads
:
<suite name="AIRS3Performances" parallel="methods" thread-count="100">
...
</suite>
Thanks for your help.
Cédric Beust ♔
I have been thinking to address what you're suggesting in a future version, namely:
<suite thread-pool-size="100">
@Test(threadPoolSize = 5)
and in this case, the 5 threads would be taken out of the the 100 threads created at the suite level.
Is this what you're thinking of?
--
Cédric
OR
I need to be sure that I run X threads simultaneously for a given test.
I thought I could declare 100 threads and be sure to run alll these
threads concurrently. This would have simulated 100 concurrent users
running the same test. Thsi is an important point for me.
What I understand now is that declaring thread-count="100' as an
attibute only prepares 100 threads but doesn't guarantee that 100
threads would run.
Right?
Thanks for your help.
Olivier
On 16 nov, 18:04, "Cédric Beust ♔ " <cbe...@google.com> wrote:
> The way it's implemented right now, @Thread(threadPoolSize) will override
> the pool size defined in testng.xml (hence the different names).
>
> I have been thinking to address what you're suggesting in a future version,
> namely:
>
> <suite thread-pool-size="100">
>
> @Test(threadPoolSize = 5)
>
> and in this case, the 5 threads would be taken out of the the 100 threads
> created at the suite level.
>
> Is this what you're thinking of?
>
> --
> Cédric
>
Alexandru Popescu
>
> Ok I think I missed something and you'll correct :
>
> I need to be sure that I run X threads simultaneously for a given test.
> I thought I could declare 100 threads and be sure to run alll these
> threads concurrently. This would have simulated 100 concurrent users
> running the same test. Thsi is an important point for me.
>
> What I understand now is that declaring thread-count="100' as an
> attibute only prepares 100 threads but doesn't guarantee that 100
> threads would run.
>
It pretty much depends :-). It depends on the number of invocations
and the threadPoolSize.
a) if invocations < threadPoolSize then at most #invocations threads
will be used
b) if invocations >= threadPoolSize then all threads will be used
However one aspect that may be interesting for you is the fact that
for the moment there is no guarantee that all threads will be started
together. I have been thinking about this lately, and I think I have a
solution to guarantee this too (it would be pretty normal behavior
considering that this kind of scenarios will be used for load
testing).
HTH,
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
OR
However, I noticed that we never have more than 5 running threads at
the same time.
I'm wondering if I need to specify the thread pool size at the method
level using the @Test annotation to be sure to have at most the
rquested number of threads. We only declare the thread count as an
attribute of the suite tag. Is it enough?
In fact, our scenarii are only for load testing so the number of active
threads at the same time is pretty important for us.
Olivier.
On 17 nov, 10:57, "Alexandru Popescu"
<the.mindstorm.mailingl...@gmail.com> wrote:
> On 11/17/06, OR <oliv...@richaud.net> wrote:
>
>
>
> > Ok I think I missed something and you'll correct :
>
> > I need to be sure that I run X threads simultaneously for a given test.
> > I thought I could declare 100 threads and be sure to run alll these
> > threads concurrently. This would have simulated 100 concurrent users
> > running the same test. Thsi is an important point for me.
>
> > What I understand now is that declaring thread-count="100' as an
> > attibute only prepares 100 threads but doesn't guarantee that 100
> > threads would run.It pretty much depends :-). It depends on the number of invocations
Alexandru Popescu
> However, I noticed that we never have more than 5 running threads at
> the same time.
>
> I'm wondering if I need to specify the thread pool size at the method
> level using the @Test annotation to be sure to have at most the
> rquested number of threads. We only declare the thread count as an
> attribute of the suite tag. Is it enough?
>
The number specified at suite level definition should be enough (if
you haven't found a bug). What number are you using there?
> In fact, our scenarii are only for load testing so the number of active
> threads at the same time is pretty important for us.
>
Yep, I am using it for similar purposes, that was why I have figured
out this possible problem :-). I will try to fix it soon, but right
now I am quite busy. I will let you know when things are done.
OR
- invocation count : always 500 (fixed at code level).
- thread count : varies from 10 to 100 threads.
We run our test several times in a row, and every time we generate a
new XML suite file using an ant script (we replace a token for the
thread-count). Then, we launch TestNG ant task.
On 17 nov, 11:16, "Alexandru Popescu"
<the.mindstorm.mailingl...@gmail.com> wrote:
> On 11/17/06, OR <oliv...@richaud.net> wrote:
>
>
>
> > Of course we're taking care of having more invocations than threads.
> > However, I noticed that we never have more than 5 running threads at
> > the same time.
>
> > I'm wondering if I need to specify the thread pool size at the method
> > level using the @Test annotation to be sure to have at most the
> > rquested number of threads. We only declare the thread count as an
> > attribute of the suite tag. Is it enough?The number specified at suite level definition should be enough (if
> you haven't found a bug). What number are you using there?
>
> > In fact, our scenarii are only for load testing so the number of active
> > threads at the same time is pretty important for us.Yep, I am using it for similar purposes, that was why I have figured
Alexandru Popescu
> - invocation count : always 500 (fixed at code level).
> - thread count : varies from 10 to 100 threads.
> We run our test several times in a row, and every time we generate a
> new XML suite file using an ant script (we replace a token for the
> thread-count). Then, we launch TestNG ant task.
>
Oke... I will check it. Looks like a bug but I cannot confirm it yet.
Alexandru Popescu
> On 11/17/06, OR <oli...@richaud.net> wrote:
> >
> > OK my numbers:
> > - invocation count : always 500 (fixed at code level).
> > - thread count : varies from 10 to 100 threads.
> > We run our test several times in a row, and every time we generate a
> > new XML suite file using an ant script (we replace a token for the
> > thread-count). Then, we launch TestNG ant task.
> >
>
> Oke... I will check it. Looks like a bug but I cannot confirm it yet.
>
Can you show me your suite definition file? What I think I am seeing
is completely different: a single thread execution (with
@Test(invocationCount=10) and the thread-count specified only at suite
level).
OR
Additionally, I define at method level an @Test annotation where I set
the invocationCount to 500.
Beside I'm trying to see if an AnnotationTransformer helps. I'll let
you know.
<suite name="AIRS3Performances" parallel="methods"
thread-count="10" time-out="2500">
<test name="Creation of documents">
<groups>
<run>
<include name="create" />
</run>
</groups>
<classes>
<class name="com.digitech.airs3perfs.CreateTest" />
</classes>
</test>
<test name="Upload of documents">
<groups>
<run>
<include name="upload" />
</run>
</groups>
<classes>
<class name="com.digitech.airs3perfs.UploadAttachedFilesTest" />
</classes>
</test>
<test name="Folding documents">
<groups>
<run>
<include name="folder" />
</run>
</groups>
<classes>
<class name="com.digitech.airs3perfs.FolderTest" />
</classes>
</test>
<test name="Removal of documents">
<groups>
<run>
<include name="remove" />
</run>
</groups>
<classes>
<class name="com.digitech.airs3perfs.DestroyTest" />
</classes>
</test>
<test name="Search on documents">
<groups>
<run>
<include name="search" />
</run>
</groups>
<classes>
<class name="com.digitech.airs3perfs.SearchTest" />
</classes>
</test>
</suite>
On 17 nov, 12:41, "Alexandru Popescu"
<the.mindstorm.mailingl...@gmail.com> wrote:
> On 11/17/06, Alexandru Popescu <the.mindstorm.mailingl...@gmail.com> wrote:
>
> > On 11/17/06, OR <oliv...@richaud.net> wrote:
>
> > > OK my numbers:
> > > - invocation count : always 500 (fixed at code level).
> > > - thread count : varies from 10 to 100 threads.
> > > We run our test several times in a row, and every time we generate a
> > > new XML suite file using an ant script (we replace a token for the
> > > thread-count). Then, we launch TestNG ant task.
>
> > Oke... I will check it. Looks like a bug but I cannot confirm it yet.Can you show me your suite definition file? What I think I am seeing
Alexandru Popescu
test methods, but for multiple invocation the same thread would be
used. At least this is what I am seeing.
I got to think about it a bit more, because at the moment I don't
think that combining the suite level thread-count and @Test level
invocation would be pretty intuitive (while the @Test level
invocationCount and threadPoolSize is pretty intuitive). In the former
case it would be required that even if you would like to run partially
concurrently you will need to have all your classes thread safe, which
is not really a requirement for all projects.
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
OR
threadPoolSize to 10 and invocationCount to 500.
The report only shows 5 threads.
Looks like a bug.
On 17 nov, 12:41, "Alexandru Popescu"
<the.mindstorm.mailingl...@gmail.com> wrote:
> On 11/17/06, Alexandru Popescu <the.mindstorm.mailingl...@gmail.com> wrote:
>
> > On 11/17/06, OR <oliv...@richaud.net> wrote:
>
> > > OK my numbers:
> > > - invocation count : always 500 (fixed at code level).
> > > - thread count : varies from 10 to 100 threads.
> > > We run our test several times in a row, and every time we generate a
> > > new XML suite file using an ant script (we replace a token for the
> > > thread-count). Then, we launch TestNG ant task.
>
> > Oke... I will check it. Looks like a bug but I cannot confirm it yet.Can you show me your suite definition file? What I think I am seeing
Alexandru Popescu
> threadPoolSize to 10 and invocationCount to 500.
> The report only shows 5 threads.
>
> Looks like a bug.
>
That reporter is really bad. I have fixed it meanwhile. Once I
investigate a bit more the other problem I will probably push another
version that is addressing the report problem.
Meanwhile if interested, I can send you a development version to test it.
OR
stateless.
On 17 nov, 14:39, "Alexandru Popescu"
<the.mindstorm.mailingl...@gmail.com> wrote:
> Thanks. In fact I think what you are seeing is: different threads for
> test methods, but for multiple invocation the same thread would be
> used. At least this is what I am seeing.
>
> I got to think about it a bit more, because at the moment I don't
> think that combining the suite level thread-count and @Test level
> invocation would be pretty intuitive (while the @Test level
> invocationCount and threadPoolSize is pretty intuitive). In the former
> case it would be required that even if you would like to run partially
> concurrently you will need to have all your classes thread safe, which
> is not really a requirement for all projects.
>
> ./alex
> --
> .w( the_mindstorm )p.
> TestNG co-founder
> EclipseTestNG Creator
>
Cédric Beust ♔
I've just written a simple annotation transformer to for the
threadPoolSize to 10 and invocationCount to 500.
The report only shows 5 threads.
Looks like a bug.
I don't think so, please reread what I wrote above: @Test(threadPoolSize) creates its own pool, regardless of what you defined at the <suite> level.
<suite thread-count="100">
@Test(threadPoolSize = 5)
public void t1() {}
@Test
public void t2() {}
@Test
public void t3() {}
t1 will have its own pool of 5 threads, while t2 and t3 will share the suite-level pool of 100 threads.
Makes sense?
--
Cédric
OR
Alex, I would appreciate to test a development version.
My need is to be sure to run exactly the number of the threads I
declare. Each test simulates a client accessing our server.
Olivier.
On 17 nov, 15:59, "Cédric Beust ♔ " <cbe...@google.com> wrote:
> On 11/17/06, OR <oliv...@richaud.net> wrote:
>
>
>
> > I've just written a simple annotation transformer to for the
> > threadPoolSize to 10 and invocationCount to 500.
> > The report only shows 5 threads.
>
> > Looks like a bug.I don't think so, please reread what I wrote above: @Test(threadPoolSize)
Alexandru Popescu
>
> Alex, I would appreciate to test a development version.
>
> My need is to be sure to run exactly the number of the threads I
> declare. Each test simulates a client accessing our server.
>
Oke... first I will have to think a bit about starting all threads
together. I will send it to your privately when ready.
./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator
> Olivier.