MicroBOSH, BOSH in an Enterprise Deployment

[Curt]

I've spent the better part of this week reviewing the documentation on cloudfoundry.org with lots of additional Googling and I am still coming up empty on some answers to some pretty fundamental questions regarding managing a BOSH system.

I am confused about the relationship between MicroBOSH and BOSH. I understand that MicroBOSH is a mini-BOSH implementation on a single VM. Some of what I read suggests that if you don't need a multi-node BOSH you can just use a MicroBOSH and you are off to the races. Other things I read seem to indicate that you merely use MicroBOSH to deploy BOSH and then you use BOSH proper to install Cloud Foundry and for general proposes (BOSH and CF being different products). My question revolves around long term maintenance.

On the "MicroBOSH-only" side of the equation, how does one continually keep a MicroBOSH up-to-date? It appears to me that the light-bosh stemcell (and accompanying AWS AMI in my case) contains all of the BOSH code for the MicroBOSH. I don't see much of an upgrade path here. If my MicroBOSH has deployed an instance of Cloud Foundry, if I wanted to update the MicroBOSH I would need to blow away the instance: how do I preserve the Cloud Foundry deployment (and keep it running with zero down-time).

If the MicroBOSH is only used to bootstrap a regular BOSH, the question is largely the same but more complicated. I should be able to update my regular BOSH through the MicroBOSH since the BOSH was created as a deployment by the MicroBOSH but I still need to keep the MicroBOSH up-to-date. In some sense it seems like the MicroBOSH is supposed to be transient, but if that is the case I'm not sure how you maintain/update BOSH.

Assume I am putting together a moderately sized internal Enterprise PaaS running on AWS. I need to understand the model for how to keep BOSH up-to-date and running 24/7 and I don't really have a picture of what the "well worn paths" are for that with BOSH.

Curt

[Dr Nic Williams]

Am in a conference, so here's a picture:

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

[Dr Nic Williams]

And here's another for deployment patterns, including the hypothetical two BOSHes resurrecting each other:

On Fri, Apr 25, 2014 at 2:07 PM, Dr Nic Williams <drnicw...@gmail.com> wrote:

Am in a conference, so here's a picture:

<61C2D82C-66B7-4050-B4EE-4ACFBDFB1A0E.jpg>

[Dr Nic Williams]

Options for persistence of each bosh/microbosh:

On Fri, Apr 25, 2014 at 2:12 PM, Dr Nic Williams <drnicw...@gmail.com> wrote:

And here's another for deployment patterns, including the hypothetical two BOSHes resurrecting each other:

<D06DE7E7-4332-4668-B869-F85509EDC5A5.jpg>

[Dr Nic Williams]

Hopefully these pictures are helpful!

On Fri, Apr 25, 2014 at 2:15 PM, Dr Nic Williams <drnicw...@gmail.com> wrote:

Options for persistence of each bosh/microbosh:

<31453589-8C0E-4530-9C33-3B5402848A16.jpg>

[cbea...@miovision.com]

Thanks for the replies. They are helpful. A few related follow-on questions...

Let's suppose I want to walk down the MicroBOSH -> BOSH(Single) example from your deployment patterns diagram. In addition I want to use the self-contained (EBS-backed) persistence pattern. The existing BOSH (and light-bosh) stemcells for AWS seem to create EC2 instances with ephemeral storage: do I need to create my own stemcells to use EBS-backed EC2 instances?

[Dr Nic Williams]

EBS for persistent disk is very much part of the BOSH feature set.

In a normal deployment (of CF, or a bosh, or anything you might deploy with BOSH), you can specify a "persistent_disk: 1024" to tell bosh to provision (or resize) the EBS to 1024 Mb. The bosh agent always mounts the volume at /var/vcap/store.

If you change the value, when you deploy it will migrate your EBS volume to the new size (creates a new disk, copies contents over, deletes old disk and remounts new disk to /var/vcap/store.

BONUS: if the VM flavor supports ephemeral disk (t1.micro doesn't; many default flavors in openstack don't) then it will be mounted at /var/vcap/data. For this reason, warden containers are stored in /var/vcap/data. Jobs & packages are stored in /var/vcap/data (have a look at the symlinks in /var/vcap/jobs ... they go into /var/vcap/data/jobs...).

MicroBOSH should also be deployed with a persistent disk. In this example manifest http://docs.cloudfoundry.org/bosh/create-micro-manifest.html#ex-aws resources.persistent_disk describes the size of the disk used for postgres, redis and the blobstore. So make it big enough for a few bosh releases :)

--

Dr Nic Williams

Stark & Wayne LLC - consultancy for Cloud Foundry users

http://drnicwilliams.com

http://starkandwayne.com

cell +1 (415) 860-2185

twitter @drnic

[nickdg...@gmail.com]

These are also things that interest me, so rather than start another discussion on this I thought I'd resurrect this one - which I don't feel was entirely answered.

I'm mainly interested in how the upgrade process works (in regard to stemcells mostly) when a MicroBOSH is used to deploy a multi-VM BOSH which is in turn used to deploy everything else (CloudFoundry, plus numerous other BOSH releases).

Does the theory state that this should be a relatively painless process? As it stands I've not had much luck upgrading a MicroBOSH to a later stem cell (going from vsphere 1722 to vcloud 2559, for example) which resulted in its persistent disk being lost in the ether and deployments having to be manually cleaned up (this is whilst the MicroBOSH is the "home" for all deployments).

Thanks,
Nick

[Dmitriy Kalinin]

MicroBOSH is designed to be updated regularly. I update my MicroBOSH
at least once a week. To do so you can run:

cd ~/workspace/my-micro
bosh micro deployment ./my-micro-manifest.tgz
bosh micro deploy ~/Downloads/some-new-stemcell.tgz --update

Let me describe what that command does assuming that you already had
an existing MicroBOSH:

- find existing vm
- connect to existing agent
- stop bosh services via the agent
- detach the persistent disk which contains all of deployment
information (bosh db, etc)
- delete existing vm
- delete existing stemcell
- create new stemcell in iaas
- create new vm based on the new stemcell
- reattach persistent disk
- start bosh services via the agent
- wait until things are running

So to summarize existing persistent data is kept and will be made
available when new microbosh vm is created. All of your deployments
managed by microbosh will be there.

Sometimes problems arise during this procedure. For example IaaS
completely failed and new vm was not created. You should be able to
just re-run bosh micro deploy command (from the same directory where
bosh-deployments.yml is kept).

If that does not work out persistent disk is still recorded and _not_
deleted. You can always log into the IaaS and manually delete the VM.
After that modify bosh-deployments.yml and remove vm_cid field (make
sure to keep disk_cid) and re-run deploy command.

[nickdg...@gmail.com]

Thanks for the info Dmitriy.

When increasing the disk size of a MicroBOSH does a similar process occur as to a deployment managed by BOSH? i.e. it attaches the new disk, moves the existing data, then shuffles the mount point and removes the old disk?

[Dmitriy Kalinin]

Yeap that should happen.