JMeter Throughput Limitations?

[Mark Tomlinson]

Hi guys,

This may not be a limitation due to the jmeter-plugins, but this is one of the best forums to post on these complex issues.

We are seeing a limitation with our setup using a combination of the Stepping Thread Group and Throughput Shaping Timer.

On a single load generator running JMeter with 4GB heap...

  - STG is set to ramp up to 1000 threads, constant for about 1.5 hours

  - TST is set to ramp up to 50tps in 900 min and steady state at 50tps for 1800 min.  (this works fine)

  - TST is then configured to ramp up to 100tps and steady state at 100tps for 1800min.  (this fails to reach 100tps...maxes out around 60tps)

   - response times are averaging about 600ms, but a small % of request do come in about 3-4 seconds

So, we try a distributed scenario using 2 load generators with separate scripts:  

      - running 500 threads on one load generator, producing 50tps and response time is < 1 sec on average

      - running 500 threads on a separate load generator, producing 50tps and response time is < 1 sec on average

      - the aggregated total hits 100tps easily, with steady response times...when I combine the .jtl outputs together and upload to Loadosophia

Now I must explain to my customer why JMeter is choking on itself.

Any thoughts or recommendations here?  Perhaps a single script, single load generator with 2 Thread Groups but without the TST?

Thanks in advance for your thoughts,

-mt

[Mark Tomlinson]

One thought was that JMeter might be backing up on its own file I/O - with all the logging, etc.

  - I am using default Batch-Stripped mode from JMeter 2.9

  - we are using CSV format .jtl

  - we are logging to NFS mount...but network does not show latency, perhaps the I/O is still latent?

Still working on it...

-mt 

[Andrey Pohilko]

Are you sure your app is not the bottleneck here?

вторник, 2 июля 2013 г., 11:18:42 UTC+4 пользователь Mark Tomlinson написал:

[Mark Tomlinson]

Hi Andrey,

It doesn't appear to be the app limitation - CPU only 20%, Heap is only about 60% utilized, Connection pools about 50%, network is only about 2% max for these small SOAP requests.  App response times for the SOAP requests are average sub-second (~500ms).  I run 1000 active threads to hit 100tps in the TST - which should give some breathing room in the pacing of about 10 seconds between each call per thread.  Remember, when I ran 50tps distributed across 2 LG's - I did have a successful run hitting 100tps for this script.  It seems when I try to push above 60tps with 1 LG (1 Jmeter instance with 1 thread group), it just can't seem to push more traffic.    

     (note:  if I check the kernel process for java/jmeter, I can see > 90% LCK time using the Solaris prstat command...which means my script logic is binding on itself, I think) 

I've got some time over the next few days (Holiday here in the US) to experiment with separate JMeter configurations, using 4 load generators to run 4 different scripts/transactions:

  - 1 JMeter instance on each LG with 4 x Thread Groups each having 25% of the load 

  - 4 JMeter instances on each LG - with 1 x Thread Group running 25% of the load

   

This will help me to determine if separate logical thread groups can solve the problem - or - if I need to separate the threads physically as well.

I could use some tips on getting logging internal to JMeter and JVM to tell me more about what the threads are waiting on - perhaps DEBUG log?  What kinds of messages would I be looking for?  Or should I just pull out JProfiler/YourKit to profile the Jmeter JVM?

Thanks sincerely,

-mt

[Andrey Pohilko]

Verbose debug messages will give you only the overhead. You should dump java stacktraces for jmeter process and see what all those guys are doing.

среда, 3 июля 2013 г., 21:01:45 UTC+4 пользователь Mark Tomlinson написал:

[vasub...@gmail.com]

Hi Mark,

   Have you managed to solve this puzzle?

I am in a similar situation where I am unable to increase the throughput with increasing concurrent threads. As the threads increases, the response times shown my JMeter increase. The machines runnning JMeter are of high Spec  16CPU, 30GB Mem, the machine is running comfortably with a load average of just 0.3. So definitely some limitation in Jmeter. Any inputs would be very helpful.

Regards,

Vasu