stemcell-builder base image bosh-ubuntu-trusty-os-image.tgz broken?

[Matt Johnson]

Hi All,

Our stemcell build process was working OK and is now bailing out on unpacking the .tgz base image, documented in:

    https://github.com/cloudfoundry/bosh/blob/master/bosh-stemcell/OS_IMAGES.md

It looks like it was updated on S3 last night:

<Contents>
<Key>bosh-ubuntu-trusty-os-image.tgz</Key>
<LastModified>2014-09-24T19:24:05.000Z</LastModified>
<ETag>"548045149f28d222cdf44366b62a5e66"</ETag>
<Size>404201310</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
</ListBucketResult>

Can someone else confirm the issue? (just downloading manually and extracting for me produces an error);

wget http://s3.amazonaws.com/bosh-os-images/bosh-ubuntu-trusty-os-image.tgz
tar -xzvf bosh-ubuntu-trusty-os-image.tgz
 ..... Output Trimmed .....

./etc/grub.d/

./etc/grub.d/20_linux_xen

./etc/grub.d/30_os-prober

./etc/grub.d/README

./etc/grub.d/40_custom

./etc/grub.d/00_header

./etc/grub.d/30_uefi-firmware

./etc/grub.d/41_custom

./etc/grub.d/05_debian_theme

./etc/grub.d/10_linux

./etc/libnl-3/

./etc/libnl-3/pktloc

./etc/libnl-3/classid

./etc/aliases.db

./tmp/

./mnt/

tar: Exiting with failure status due to previous errors

This highlights an issue I raised in a previous thread: https://groups.google.com/a/cloudfoundry.org/forum/#!topic/bosh-dev/RtyTinf6uxs

Take my current situation as a use case;

• We have automation to run stemcell builder with some custom stages to build stemcells for our environments.
  
• The automation at this point is broken because I cannot use this single tarball from the link above.
  
• I also cannot build a previous version of the stemcell (ever... even if the tarball was fine) because although there are tags in the Bosh repo for each stemcell version, I have no way of building against the accompanying snapshot of the base tarball.

For that reason, we have no choice but to have our stemcell builder pointing at the 'develop' branch of bosh, because thats the one that seems most often compatible with the changes that keep being made to the single base image tarball.

Am I missing something with this (please shout if my issue isnt clear and i'll try and explain better).

I think this could be solved in one of two ways:

1. Have the S3 repo contain snapshots of the base image for releases, such as 'bosh-ubuntu-trusty-os-image-<VERSION>.tgz'
   
2. Have a way to build the base image tarball myself, which could be done as a pre-req in our stemcell builder jenkins jobs. If this was/is in the bosh repo, it would then be tagged with the existing release numbers.

Regards,

Matt

[Matt Johnson]

I've found this rake task in bosh/bosh-dev/lib/bosh/dev/tasks/stemcell.rake

 desc 'Build a base OS image for use in stemcells'
 task :build_os_image, [:operating_system_name, :operating_system_version, :os_image_path] do |_, args|

Running now, it looks like it will do what I want for point 2 above, especially since it's within the BOSH repo and so tagged at the right version.

If anyone has any 'here be dragons' concerns with using this instead of the version hosted on S3 (as per the stemcell-builder docs) let me know!

Matt

[Dmitriy Kalinin]

untarring succeeded for me.

What we are planning to do shortly is check in s3 version pointer into the repo, such that you will always get os image referenced for that build. That's going to be coming down the pipeline shortly.

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-dev+u...@cloudfoundry.org.