CF App Load Division Between Instances

[Wahaj Ali]

I was doing some benchmarking of our cloud environment for a web application. I start of with one application instance and as the load increases I keep adding instances when avg cpu usage exceeds 70%. At one point I have three instances of the application running. Ideally, I would expect that the load divides well amongst the different instances. However, at some point the initial instance starts taking up the bulk of the load. I am using Jmeter for creating the load and have made sure that http keep alive is set to close. Following is a graph of the CPU usage:
 

The blue line is the first instance, red the second and finally green is the third instance of the application. I have repeated this experiment thrice and noted similar results. In there any way to find out why the load is not being evenly divided?

[James Bayer]

by any chance does the app have a JSessionID that could activate the sticky sessions feature of the router [1] ?

are the app instances on a single DEA or spread across multiple DEAs?

are you using a VM for each DEA? which hypervisor?

[1] http://www.iamjambay.com/2013/10/a-quick-tour-of-cloud-foundry-router.html

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Thank you,

James Bayer

[Wahaj Ali]

Hi, 

The app does not have JSessionID. 

The app instances are initially on a single DEA, as the load increase both DEA and app instances increase, although this is independent of each other (therefore it might happen that an app instance is added followed by a DEA, although the otherway round would have been better). 

Lastly, each DEA has its own VM, running on OpenStack/KVM.

Could this be because the cpu usage is computed pcpu. So for example if there are 3 instances on one DEA and 1 instance on another DEA, would pcpu usage be different since the second one has more available cpu per instance. Is there a way to explicitly rebalance applications instances across the DEAs? 

Thanks!

[James Bayer]

the cloud controller should prefer DEA's with fewer instances of the app [1].

starting/stopping the app would cause a force rebalancing, but it should be balanced as instances start up according to the CC algorithm.

[1] https://github.com/cloudfoundry/cloud_controller_ng/blob/3a1907d2941ac5ff1a4568be9b0068b9cdbff16f/lib/cloud_controller/dea/dea_pool.rb#L38-L52