Desired Feature: bosh package release

[daniel....@gmail.com]

Hi,

After having to mess around with using different VMs/laptops for different parts of the create/deploy process, I had the idea that I could package a release into a tarball and distribute that, just like you can distribute a stemcell.  That is, when I do $bosh upload release and I see
Copying packages
...
Building tarball
----------------
Generated /tmp/d20130110-9288-3jg7h0/d20130110-9288-193ciw3/release.tgz
Release size: 1.5G
...
Uploading release
release.tgz:     2% |                        |  32.3MB 825.8KB/s ETA:  00:31:53
...

Then I'd rather do
$bosh package release
Copying packages
...
Building tarball with every "FOUND LOCAL" package and job
----------------
Generated /output_directory/release.tgz
Release size: 1.5G

$bosh upload release /output_directory/release.tgz
Verifying release...
File exists and readable                                     OK
Extract tarball                                              OK
Manifest exists                                              OK
Release name/version                                         OK
...
Uploading release
release.tgz:    20% |oooo                    | 316.6MB 822.5KB/s ETA:  00:26:07


That may open up future things like $bosh download public releases

Just a thought

[Dr Nic Williams]

I like this idea. It wouldn't technically be a lot of dev effort - it's a large package to upload/download like a stemcell. CLI and director changes. Want to have a crack at it? I could help you along if you have questions etc.

A bonus v2 version of it, much more effort I think, is to distribute releases that include precompiled packages (for a given stemcell). Imagine not having to compile 50+ cf-release packages when you're just getting started!

--

Dr Nic Williams

http://drnicwilliams.com

cell +1 (415) 860-2185

[Martin Englund]

Daniel,

we already have that - you get it by running:
bosh create release --with-tarball

cheers,
/Martin
--
Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
"The question is not if you are paranoid, it is if you are paranoid enough."

[Martin Englund]

Nic,

the second part is something I've thought about a lot when testing micro bosh stemcells. The way it works now, every package gets recompiled whenever you build a new stemcell, even if the base OS hasn't changed. You also have excessive compiling when you have multiple bosh directors deploying the same release (using the same stemcell).

If we make compiled packages (for final versions of releases) available for download (keyed on stemcell version) a deploy will become considerably faster!

cheers,
/Martin
--
Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
"The question is not if you are paranoid, it is if you are paranoid enough."

[Matthew Kocher]

Agreed, sharing compiled packages somewhere central makes a lot of sense.  I think they can only be use if the stemcells are exactly the same however, as we can't know if all the shared libraries are the same on an upgraded stemcell.

I'd love if the package cache could just be a blobstore with no server in front of it with well defined file names (eg, stemcell-bosh-stemcell-aws-0.6.4-sha1ofstecell-postgres-9-1-sha1ofpostgres).  I think at the moment bosh doesn't assume it can chose a filename in the blobstore, but I'm not sure where that restriction comes from.

[James Bayer]

Off-topic. Speaking of blob stores, just for fun you can take a look at this next-gen blob store called camilstore under development by Brad Fitzpatrick that Adrian Crockcoft retweeted tonight that I thought was interesting.

http://camlistore.org/talks/2011-05-07-Camlistore-Sao-Paolo/#1

Thank you,

James Bayer

[Martin Englund]

If the stemcell is the same version, we know that the libraries are the same. We never upgrade the OS contents, other than upgrading the stemcell, and then we have a known set of libraries.

The blobstore can be the actual service, we added the blobstore server component to make sure it lives as close to the consumers (VMs that bosh creates) as possible. If you are running on AWS, you can ditch the blobstore server and just configure bosh to use S3.

cheers,
/Martin
--
Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
"The question is not if you are paranoid, it is if you are paranoid enough."

[Martin Englund]

Looks like a very good architecture. Once the blobstore can run on something other than GAE it'll be an interesting option.

cheers,
/Martin
--
Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
"The question is not if you are paranoid, it is if you are paranoid enough."

[Dr Nic Williams]

James, that slide deck only has 3 slides in it for me. Anyone else? If you have a fuller set of slides can you email them?

Nic

[Dr Nic Williams]

From http://camlistore.org/

There is an older presentation with all its slides: https://docs.google.com/presentation/d/1otsEBBmOYoPY6T7p4YtM0ysX1vUG3amZQAykr9jZZgk/present#slide=id.i0