Scalability & latency-sensitivity of BOSH

[behrend...@web.de]

Hi,

I just came across a few BOSH-related questions -- would very much appreciate your insight into these topics:

- How latency-sensitive is BOSH? Let's say my BOSH Director is in one geo and my CF deployment is in another one -- would that work? Does anyone have experience with that? The perf penality would probably depend to some degree on the chattiness of the BOSH protocol -- does someone have insight into that?
- How scalable is BOSH? Let's say I want to manage several CF very large installations with it -- is there any experience wrt scalability limits?
- Based on the questions above -- is there any general (experience-based) guidance on when to use a single BOSH Director to manage multiple CF installations vs. setting up a dedicated one per CF deployment?

Thanks a lot in advance.



Thanks & best regards
Michael

[James Bayer]

inline

On Tue, Jul 8, 2014 at 12:54 AM, <behrend...@web.de> wrote:

Hi,

I just came across a few BOSH-related questions -- would very much appreciate your insight into these topics:

- How latency-sensitive is BOSH? Let's say my BOSH Director is in one geo and my CF deployment is in another one -- would that work? Does anyone have experience with that? The perf penality would probably depend to some degree on the chattiness of the BOSH protocol -- does someone have insight into that?

we only test with a co-located low-latency network between the BOSH Director and BOSH deployment VMs. there are certainly some NATS and agent-director communications that may be affected by using a high-latency network. you are in unchartered territory if you try to use a BOSH Director to deploy VMs to a different geography or with a high latency network.

 

- How scalable is BOSH? Let's say I want to manage several CF very large installations with it -- is there any experience wrt scalability limits?

the current design goal is to manage thousands of VMs. we had done some scalability testing a long time ago with certain components going up to several thousand nodes managed by a single BOSH cluster. practically speaking i only have awareness of people managing several hundreds of VMs with a single BOSH cluster currently. ferdy recently uncovered some scalability limits while provisioning 150+ VMs concurrently which are now known issues and bottlenecks we intend to improve. having tens of thousands of nodes to manage would be a great problem to have in the future.

 

- Based on the questions above -- is there any general (experience-based) guidance on when to use a single BOSH Director to manage multiple CF installations vs. setting up a dedicated one per CF deployment?

we have separate installations of BOSH at Pivotal based on our environments for parts of our overall development pipeline segments. so our pipe segments look like:

development -> integration -> pre-production -> production

each of these pipeline segments has it's own BOSH. development typically uses bosh-lite and other segments use larger BOSH configurations. this way when we make changes for BOSH version or stemcells etc we treat it like we're rolling out new versions of our code. we can propagate those changes through the pipeline incrementally.

in addition to pipeline considerations, geography and operator tenancy would be other considerations for when you might consider multiple BOSH deployments. by operator tenancy, i mean to ask yourself this question: are the BOSH operators that have admin access to a BOSH deployment trusted to have access to the other BOSH deployments managed by the same BOSH instance? if the answer is no, then you need separate BOSHes.

 

Thanks a lot in advance.



Thanks & best regards
Michael

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

--

Thank you,

James Bayer