Is there a way to Rate Limit BOSH's creation of missing VMs?

[daniel....@gmail.com]

When I try to deploy a full cloudfoundry on our openstack, things get messed up because not only are the openstack API requests rate limited, but possibly the infrastructure can't handle 46 VMs spawning at once.  4 are successfully started, 6 fail to start and are not present in openstack instances.  A subsequent "bosh deploy" will still fail.  Is there a way to limit how many VMs are spun up by BOSH at a time?

#bosh deploy......

Creating bound missing VMs
##Many more 413 Request Entity Too Large
 infrastructure/31: Expected([200, 202]) <=> Actual(413 Request Entity Too Large) Response includes the text "Only 10 POST request(s) can be made to * every minute.\"
  infrastructure/0 (00:01:26)                                                   
  infrastructure/4 (00:01:45)                                                   
  infrastructure/3 (00:01:45)                                                   
  infrastructure/7 (00:01:50)                                                   
  infrastructure/8: Timed out pinging to e6f3ef8b-27f7-4b12-945c-0d333e236485 after 300 seconds (00:05:52)
##5 more Timed out pinging

#bosh deploy again....
Preparing deployment
  binding deployment (00:00:00)                                                 
  binding releases (00:00:00)                                                   
  binding existing deployment: VM `ad0077df-736b-4c64-9c03-eb25685c03c2' is out of sync: it reports itself as `debian_nfs_server/0' but there is no instance reference in DB (00:01:30)
Error            3/9 00:01:30

#bosh vms...

+---------------------+--------------------+----------------+---------------+
| Job/index           | State              | Resource Pool  | IPs           |
+---------------------+--------------------+----------------+---------------+
| unknown/unknown     | unresponsive agent |                |               |
..........MORE............
| unknown/unknown     | unresponsive agent |                |               |
| acm/0               | running            | infrastructure | 192.168.13.38 |
| cloud_controller/0  | running            | infrastructure | 192.168.13.35 |
| debian_nfs_server/0 | running            | infrastructure | 192.168.13.36 |
| syslog_aggregator/0 | running            | infrastructure | 192.168.13.37 |
+---------------------+--------------------+----------------+---------------+

VMs total: 26

Thank you.

[shair khan]

The problem is not just limited to Open Stack implementation. We see this problem big time in our implementation with VMware vCenter. The issue is there needs to be either some kind of throttling or retry logic.   

[Martin Englund]

Daniel,

we have a limit - 32 parallel threads creating VMs, but it isn't tunable:
https://github.com/cloudfoundry/bosh/blob/master/director/lib/director/jobs/update_deployment.rb#L85

File an issue and we'll add it to our todo list :)

Cheers,
/Martin

--
cheers,
/Martin
--
Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
"The question is not if you are paranoid, it is if you are paranoid enough."

[shair khan]

Thanks - Files the bug

[daniel....@gmail.com]

While you are working on a fix, what is your suggested workaround:
1. Build custom micro bosh using https://github.com/drnic/bosh-getting-started/blob/master/create-a-bosh/creating-a-micro-bosh-from-stemcell-openstack.md pattern with 4.6 parallel threads
2. 10x micro deployments of 4.6 VMs each, eg deployments CF part 1, CF part 2 .. CF part 10

On Tuesday, February 5, 2013 8:28:54 PM UTC-8, Martin Englund wrote:

[leftleg]

Hi Martin

    I encountered the same problem. But I can not find where CF-release and bosh-release update_deployment.rb file. Could you direct me to where you can modify max_threads?

在 2013年2月6日星期三UTC+8下午12时28分54秒，Martin Englund写道：

[leftleg]

I found the file update_deployment.rb#L85 in inception-vm. I need to redeploy micro-bosh and bosh deploy cf-release?

在 2013年2月7日星期四UTC+8上午3时55分18秒，daniel....@gmail.com写道：

[Ferran Rodenas]

A bypass if you want to test it without recreating a new stemcell: ssh to the microbosh vm, update the /var/vcap/packages/director/bosh/director/lib/director/jobs/update_deployment.rb file and restart director (sudo monit restart director).

- Ferdy

2013/2/19 leftleg <leftl...@gmail.com>

[daniel....@gmail.com]

I can confirm that this works.

I also see that

Config.max_threads

has been implemented

Please pardon my ignorance, but how do I set

Config.max_threads?

[Martin Englund]

Daniel,

it is set through the director properties when you deploy (micro) bosh
(see "director/templates/director.yml.erb)

/M

[K_James]

how many theads cound be suitable？

在 2013年3月2日星期六UTC+8上午6时39分21秒，Martin Englund写道：

[Martin Englund]

I don't know, as it depends on how many api calls you may do per second. Creating a new instance requires about 5 calls, but the are all lumped together in the beginning and then there is a lot of waiting for the VM to be created. 

I'd recommend starting at one, and then try to increase it until you hit the limit. 

We could add a rate limit to the API calls, but that isn't a simple thing to do so it won't happen for a while unless you can't get this change to work. 

/M

Sent from my iPhone

[Ferran Rodenas]

Just for clarification, there're 2 types of rate limits:

1) Number of concurrent tasks (i.e. number of VMs that director spins up or terminates in paralel): Now you can configure this by setting a director.max_threads property at your deployment manifest (defaults to 32 threads), and it's available for all IaaS. This is useful if your infrastructure cannot handle spawning at once a large number of VMs.

2) API calls. As Martin pointed out, creating a new VM requires several API calls (number depends on what IaaS you're using): For OpenStack, now the CPI detects if there's an API overlimit and instead of failing, it waits the amount of time that the OpenStack API indicates, and retries the call.

Both features are not yet available at the latest public stemcells, so you'll need to create your very own stemcells if you want to use them (or use the edge-stemcell option if you're using bosh-bootstrap).

- Ferdy

2013/3/5 Martin Englund <meng...@rbcon.com>

[daniel....@gmail.com]

On my system I set the max_threads to 2.  I'm thinking it could go to 4, but I haven't gotten around to restarting the director.

@Ferdy: API overlimit detection sounds nice