Private blob mirroring

[Ryan Grenz]

Hi guys,

I thought I'd pose this question to the forum and see what people think:

We are considering having a local/private mirror of all blobs required to do a BOSH or Cloud Foundry deployment. 

Why? 3 reasons: 

1) Because we'd like to not be completely reliant on internet access when doing BOSH release upload/deployments

2) Speed of download from the blob sources can often be a little slow inside of our network and we'd like them to be lightning quick

3) Because our corporate transparent proxy randomly corrupts downloads (although infrequent its very annoying, so it'd be nice not have to deal with that)

As we're using Openstack Grizzly as our IaaS, and Ceph as our storage provider, we have the opportunity to install Rados gateway http://ceph.com/docs/next/radosgw/ to take advantage of its S3-compatible API.

We felt we could use it to provide an exact mirror of all blobs required for BOSH deployments (amongst other things).

I'm keen to hear thoughts from people on this subject, and how we could go about mirroring the blobs from the command line (or another way)

Cheers,

Ryan

[Dr Nic Williams]

Does https://github.com/yudai/simple_blobstore_proxy solve your problem?

--

Dr Nic Williams

Stark & Wayne LLC - consultancy for Cloud Foundry users

http://drnicwilliams.com

http://starkandwayne.com

cell +1 (415) 860-2185

twitter @drnic

[Ryan Grenz]

Sure looks like it might - lemme try it out and i'll report back here in a bit.

Cheers

Ryan

[Iwasaki Yudai]

Ruben's script may help you migrating files to your local blobstore.

https://gist.github.com/rkoster/5705252#file-migrate_blobs-sh

Btw, simple blobstore proxy is a really adhock tool using the local file system.

I recommend you to use reliable storage solutions.

[Ryan Grenz]

Yeah was talking to Dr Nic about this yesterday, and I think I arrived at the decision that I shouldn't need to source blobs that often from the internet, especially for final releases as they'll never change.

Once the BOSH server gets the blobs uploaded, they should be stored on a better blob store than BOSH's default - so Radosgw is still a viable storage backend for BOSH.

I'll adjust my BOSH deploy manifest today.

Thanks,

Ryan