Poisson process support in ultimate threadgroup and shaping timer

[Vladimir Sitnikov]

Hi,

I feel threadgroup and shaping timer should be thought to create poisson process for the incoming load.

That would simplify running multiple load scripts in parallel and create more realistic load.

See [1] for additional reasoning of using poisson process for load testing.

I do not like current "rampup" behaviour since it distributes load at even intervals, thus if you have two or more jmx files with 60 requests/hour each it would result in synchronized load from both scripts.

In case a threadgroup spawns threads according to poisson process, the load from both jmx would not suffer such a synchronization.

The implementation is very easy: say you have requests_per_second and all you need is just use delays of -ln(1-Random.nextDouble())/requests_per_second between each starts of threads.

Same goes for shaping timer.

Do you feel such kind of improvement can be merged into JP?

One more point: I would like to have seed for random as an input parameter. That will enable running the same workload multiple times (same delays between threads) and simplify validation if the test is reproducable.

[1]: http://perfdynamics.blogspot.ru/2012/05/load-testing-with-uniform-vs.html

[Andrey Pohilko]

I think this is inapplicable for current UTG and shaper, since actual load will not look like planned. I think you just need to use additional delay timers assigned to samples, this will produce delays distributed by desired formula. Moreover, I think you don't need UTG and the shaper for such tests at all, because their features will be totally masked by poisson delays.

четверг, 4 апреля 2013 г., 23:32:58 UTC+4 пользователь Vladimir Sitnikov написал:

[Vladimir Sitnikov]

>> I think you just need to use additional delay timers assigned to samples,

That will kill the idea that threads should be started only as required.

I do not want to start 100500 threads at the very start and put them to sleep.

>>actual load will not look like planned.

It will.

>>will produce delays distributed by desired formula

That is no-go for open system testing (the case when no loops is used, and each thread executes just a single testcase)

>>I think you don't need UTG and the shaper for such tests at all, because their features will be totally masked by poisson delays.

The load would be changed according to UTG/shaping configuration.

What is the problem with that?

Say one specified 10 requests/second -- it will fire 10 requests/second _on average_.

[Andrey Pohilko]

Well, I'd suggest you to fork on GitHub project sources and play with your changes made into code. Then you'll see if it works as you like. 

My experience in performance testing says that burst load leads to less reproducible tests and adds not much value to performance improvement process.

I think your request is too uncommon to be a part of plugins set.

пятница, 5 апреля 2013 г., 11:31:28 UTC+4 пользователь Vladimir Sitnikov написал:

[mark]

Did you fork out a project in jmeter to deal with poisson arrivals?

[Vladimir Sitnikov]

I just created my own plugin with poisson arrivals timer.

Vladimir

[Vincent Daburon]

Hi,
I don't like random timer, i prefer constant timer.

When i launch 2 load tests with the same parameters i want the same result.
With random you don't know if the difference are because you use random timer or because the application returns different page generation time results.

I use only random timer if i see to much periodic calls on to much synchronization call. (lot of threads are waiting together same page or sql results).

Regards.
Vincent

[Vladimir Sitnikov]

>When i launch 2 load tests with the same parameters i want the same result.

So do I that is why I use seed for my random generator, so my tests
are reproducible (each execution creates exactly the same pattern of
"randomized" delays).

Vladimir

[Vincent Daburon]

Hi,

For me reproducibility is more important than the pseudo reality of users waiting times.

At the end of my load test I look at the number of times I played the script over 1 hour (cadences/hour).

This is the main objective of my load test.

Regards.
Vincent D.

[Vladimir Sitnikov]

So what?

Vladimir

[mark]

For me, testing the system under a more realistic load of arrival is more important than reproducibility (which can be done with a seed).

Thanks for the info Vladimir. I will write my own as well. Hopefully this feature can be part of Jmeter in the future.

Vatche