Migrating cf-release blobs to S3

[Mike Heath]

We're trying to fork cf-release and migrate our blobs to S3. We have been able to move all of the blobs in cf-release/blobs over. That part was easy. When we try to create a release doing 'bosh create release' we get errors saying that it can't find packages in the blobstore.

Specifically we're getting:

Building DEV release

---------------------------------

Building packages

-----------------

Building daylimit...

  Final version:   NOT FOUND

  Dev version:     FOUND LOCAL

Building mysql55...

  Final version:   Downloading `mysql55 (4)'...

Final version of `mysql55' not found in blobstore

Is there an easy way to migrate the package blobs to S3?

-Mike

[Dr Nic Williams]

bosh create release --force - should skip the check

or 

bosh upload blobs

bosh create release

--

Dr Nic Williams

http://drnicwilliams.com

cell +1 (415) 860-2185

[Mike Heath]

Adding --force eliminates the warnings about my changes not being committed in git but doesn't prevent bosh from trying to download packages from the blobstore, unfortunately.

[Dr Nic Williams]

What does "bosh upload blobs" do?

[Nicholas Terry]

There is no simple way to do what you want. We dealt with this issue a little while back.The best way to accomplish this is to modify the source to use two blobstores. (Its how we had to do it)

There may be a hacky way to get the blobstore populated...

First, make sure the final.yml and provate.yml files in the config dir are the vanilla copies and do a bosh sync blobs

1. Recreate the non-submodule directories in the src/ folder to the blobs directory (not to be confused with the (dot)blobs directory)

2. For each symlink in each folder in the src dir, copy the referenced blob to its corresponding folder in the blobs directory. For example:

Given: src/debian_nfs_server/whatever.tgz -> /path/to/.blobs/a44b3f8319984bace61f21bb6d15cee1419e5bf3 (THIS IS AN EXAMPLE)

Do the following:

cp /path/to/.blobs/a44b3f8319984bace61f21bb6d15cee1419e5bf3 blobs/debian_nfs_server/whatever.tgz

3. Empty out blobs.yml in the config folder so that is only contains --- {}

4. Then change your private and final yaml files to you appropriate setup for your s3 store
5. Then do a bosh upload blobs. You should be prompted to upload each blob in turn and blobs.yml will be populated with your blobs and their ids and stuff.

If you are unsure of how to format you final and provate yml files, go here:

https://github.com/cloudfoundry/bosh/blob/master/cli/lib/cli/release.rb#L72-L86

make sure you have provisioned your s3 bucket correctly you need to follow these directions:

https://github.com/cloudfoundry/oss-docs/blob/master/bosh/documentation/bosh_releases.md#setting-up-a-new-release-repository-using-s3

(The bucket policy referenced there is incorrect. Use the IAM builder to make it)

Let me know if this works for you

[Martin Englund]

Excellent write-up! Thanks for taking the time to do it

/M

Sent from my iPhone

[Mike Heath]

It uploads all the blobs in my blobs/* directory just fine. I have those uploaded to S3 and it works great. I have a script that keeps the cf-release blobs synced with our blobs in S3. The other devs on my team can checkout our cf-release fork and do a 'bosh sync blobs' and all the blobs in the in the blobs directory get downloaded. No problems there.

The problem we're seeing (that I did not anticipate) is that it appears that jobs and packages get cached in the blobstore as well. The index.yml files in /.final_builds are full of blobstore_id fields that reference what appear to be tarballs in the http://blob.cfblob.com blobstore.

These job and package blobs are not listed in the config/blobs.yml. Which is why I was surprised to find this problem. I like the idea of caching these, especially to ensure consistency for future deployments. I don't like that I can't do a "bosh add blob ..." to get them in S3. :)

I've been chatting with Nick Terry about his patch. We'll likely go down that route. I like the idea of being able to chain blobstores.

-Mike

[Dr Nic Williams]

Chaining/overriding blob stores sounds very nice. I look forward to seeing any finish products/repos/gists.

[nte...@familysearch.org]

YAY! We committed the composite blobstore client to be accepted!!! See it here:

http://reviews.cloudfoundry.org/#/c/13445/


Also whats the process for submitting fixes/additions to the oss-docs project (we need to document the format for the private and final yml files to use the composite blobstore client)? Do I just create a pull request in github?

[Dr Nic Williams]

CONGRATS!

Well done for submitting a (first?) patch!

Nic

[Dr Nic Williams]

AFAIK, oss-docs is also on gerrit at the moment.

[James Bayer]

Actually we're in the middle of transitioning some processes for contributing to be a bit simpler, can you try a standard github pull request and see if it gets auto-closed? I think it's not going to do that any longer.

Thank you,

James Bayer