How to run tests with constant thread count..

[shankar KC]

Hi,

Our product is tweaked to support say 60 concurrent users. 

Say I have 2 maven projects with 5 test suites. each having 10 tests. For convenience lets say all tests except one each takes 1min to run. Long running test takes 10 min to complete.

Lets assume i run these 2 projects in parallel in 2 jenkins job. Each job runs 3 test suites in parallel with 10 tests in parallel( using parallel=methods and thread-count =5)

Now when i start the tests during first minute we have 2 * 3 * 10. After first min all tests in a suite completed execution except the long running test. It means only 6 tests are running. Rest of the threads (2 * 3 * 9) are idle and will not run remaining suites as current thread didnt finish because of long running test. It cannot pick remaining 2 suites though they are idle. How do we solve this and put a constant load on our server. Now 2nd batch of test suite execution happens only after 10 minutes. All the threads were idle expect one for 9 minutes. This is a common issue in product companies i think. But I am not sure how to tackle it properly without wasting lot of time. How you all handle this at your organization?

Thanks

Shankar KC 

[shankar KC]

sorry for the typo. threadcount is -> ( using parallel=methods and thread-count =10)

[Todd Bradley]

Our product is tweaked to support say 100,000 concurrent users. How we approach your problem is to use TestNG as primarily a functional testing framework, and use something different for scalability and load testing.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[shankar KC]

We too use testng for functionality testing.  But wouldn't this behavior of execution increase overall execution time? Threads keeps waiting for other long running test to complete.

You received this message because you are subscribed to a topic in the Google Groups "testng-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/testng-users/mdFYBy7bWk0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to testng-users...@googlegroups.com.

[Todd Bradley]

Yes, and we parallelize tests as much as we reasonably can. But ultimately we have the same issue you do, and just accept that we can't perfectly balance all the test execution threads without a lot more work. And for us, at least, that extra work isn't worth it because all it would gain is a small improvement in test run time. So we do other things to improve test run time - running tests of slow changing features less often, running slow tests less often, etc.

I misunderstood your goal, I think. I thought you are trying to put a constant load on the system-under-test, and I'd say use different tools for that.