Oo Diskimage

[Dominik Ransom]

It includes support for building images based on many majordistributions and can produce cloud-images in all common formats(qcow2, vhd, raw, etc), bare metal file-system images andram-disk images. These images are composed from the many includedelements; diskimage-builder acts as a framework to easily addyour own elements for even further customization.

Two elements use the source-repositories element, but use the same filenamefor the source-repositories config file. Files such as these (and indeed thescripts in the various .d directories listed below) should be named such that they are unique. If theyare not unique, when the combined tree is created by disk-image-builder forinjecting into the build environment, one of the files will be overwritten.

If your element mounts anything into the image build tree ($TMP_BUILD_DIR)then it will be automatically unmounted when the build tree is unmounted - andnot remounted into the filesystem image - if the mount point is needed again,your element will need to remount it at that point.

Elements should allow for remote data to be cached. When $DIB_OFFLINE isset, this cached data should be used if possible.See the Global image-build variables section of this document formore information.

Elements in the upstream diskimage-builder elements should not createexecutables which run before 10- or after 90- in any of the phases ifpossible. This is to give downstream elements the ability to easily makeexecutables which run after our upstream ones.

Make as many of the following subdirectories as you need, depending on whatpart of the process you need to customise. The subdirectories are executed inthe order given here. Scripts within the subdirectories should be named with atwo-digit numeric prefix, and are executed in numeric order.

Only files which are marked executable (+x) will be run, so other files can bestored in these directories if needed. As a convention, we try to only storeexecutable scripts in the phase subdirectories and store data files elsewhere inthe element.

Final tuning of the root filesystem, outside the chroot. Filesystemcontent has been copied into the final file system which is rootedat $TMP_BUILD_DIR/mnt. You might do things like re-mount acache directory that was used during the build in this phase (withsubsequent unmount in cleanup.d).

Perform final tuning of the root filesystem. Runs in a chroot after the rootfilesystem content has been copied into the mounted filesystem: this is anappropriate place to reset SELinux metadata, install grub bootloaders and soon.

Because this happens inside the final image, it is important to limitoperations here to only those necessary to affect the filesystem metadata andimage itself. For most operations, post-install.d is preferred.

To set environment variables for other hooks, add a file to yourelement environment.d. This directory contains bash scriptsnippets that are sourced before running scripts in each phase. Notethat because environment includes are sourced together, they shouldnot set global flags like set -x because they will affect allpreceeding imports.

This is always set. When not empty, any operations that perform remote dataaccess should avoid it if possible. If not possible the operation should stillbe attempted as the user may have an external cache able to keep the operationfunctional.

In the general case, configuration should probably be handled either by themeta-data service (eg, o-r-c) or via normal CM tools(eg, salt). That being said, it may occasionally be desirable to create aset of elements which express a distinct configuration of the same softwarecomponents.

diskimage-builder has the ability to retrieve source code for an element andplace it into a directory on the target image during the extra-data phase. Thedefault location/branch can then be overridden by the process runningdiskimage-builder, making it possible to use the same element to track morethen one branch of a git repository or to get source for a local cache. Seesource-repositories for more information.

DIB exposes an internal $IMAGE_ELEMENT_YAML variable whichprovides elements access to the full set of included elements andtheir paths. This can be used to process local in-element filesacross all the elements (pkg-map for example).

The manifests element will make a range ofmanifest information generated by other elements available forinspection inside and outside the built image. Environment andcommand line arguments are captured as described in the documentationand can be useful for debugging.

Tests are run with tools/run_functests.sh. Runningrun_functests.sh -l will show available tests (the example abovewould be called apt-sources/test-case-1, for example). Specifyyour test (or a series of tests as separate arguments) on the commandline to run it. If it should not be run as part of the default CIrun, you can submit a change with it added to DEFAULT_SKIP_TESTSin that file.

Running the functional tests is time consuming. Multiple paralleljobs can be started by specifying -j . Each of thejobs uses a lot resources (CPU, disk space, RAM) - therefore the jobcount must carefully be chosen.

Using sudo outside the chroot environment can cause breakoutissues where you accidentally modify parts of the hostsystem. dib-lint will warn if it sees sudo calls that do notuse the path arguments given to elements running outside the chroot.

For the past year I have been responsible for redesigning and keeping a multi-region OpenStack deployment up to date. I will eventually go in-depth on what that entailed, but for nowI will focus on my first challenge on that infrastructure, keeping the cloud images provided to the users up to date andcustomized according to defined local policies.

To achieve this, and after researching my choices, I chose diskimage-builder.It may seem overkill at first glance, but because of its modular architecture it called to my OCD nature and in the endyou can make your own modules that can depend on existing modules, so yes it seemed perfect for the job. Besides, it ismade by the Openstack team, so it is more than trustworthy.

Before I continue let me reword myself, this is used for generating stripped down Linux bootable cloud images, you canuse them wherever you would like that supports booting the resulting filesystem. These images are cloud-ready and shouldbe able to retrieve any metadata you configure it to fetch from your cloud provider.

I recommend having some background prior to delving in this use case, as it can get very complex when you need to debugmore advance generation scenarios (i.e: CI). The cloud images provided by your favorite distribution should be enough tosatisfy most cloud computing needs. This process helps in reducing the cloud image footprint to your bare needs, andallows you to customize it beyond the distribution's provided cloud images.

It includes support for building images based on many major distributions and can produce cloud-images in all commonformats (qcow2, vhd, raw, etc), bare metal file-system images and ram-disk images. These images are composed from themany included elements; diskimage-builder acts as a framework to easily add your own elements for even furthercustomization.

One of the main reasons I chose to use diskimage-builder was for its modular design, but I quickly discovered that it isalso used extensively by the TripleO project and within OpenStackInfrastructure, which fit perfectly in our Openstack infrastructure.

"Images are built using a chroot and bind mounted /proc /sys and /dev. The goal of the image building process is toproduce blank slate machines that have all the necessary bits to fulfill a specific purpose in the running of anOpenStack cloud. Images produce either a filesystem image with a label of cloudimg-rootfs, or can be customised toproduce whole disk images (but will still contain a filesystem labelled cloudimg-rootfs). Once the file system tree isassembled a loopback device with filesystem (or partition table and file system) is created and the tree copied into it.The file system created is an ext4 filesystem just large enough to hold the file system tree and can be resized up to1PB in size." [2]

Final tuning of the root filesystem, outside the chroot. Filesystem content has been copied into the final file systemwhich is rooted at $TMP_BUILD_DIR/mnt. You might do things like re-mount a cache directory that was used during thebuild in this phase (with subsequent unmount in cleanup.d).

Perform final tuning of the root filesystem. Runs in a chroot after the root filesystem content has been copied into themounted filesystem: this is an appropriate place to reset SELinux metadata, install grub bootloaders and so on.

Because this happens inside the final image, it is important to limit operations here to only those necessary to affectthe filesystem metadata and image itself. For most operations, post-install.d is preferred.

Cloud-init is a method developed by Canonical for cross-platform cloud instanceinitialization. It is supported across major public cloud providers, provisioning systems for private cloudinfrastructure, and bare-metal installations.

Cloud-init will identify the cloud it is running on during boot, read any provided metadata from the cloud andinitialize the system accordingly. This may involve setting up the network and storage devices to configuring SSH accesskey and many other aspects of a system. [4]

We should note that there are two main metadata types, EC2 and Openstack. They both have different API specificationsbut you can usually find support for Openstack metadata links across different public clouds, or even both, be sure toknow your cloud provider!

CentOS has released its version 8 cloud images last month, conveniently the diskimage-builder developers already addedthe much needed changes to their centos-minimal element. Fortunately I noticed that the cloud-init package was missingfrom the generated image (I don't know if it is also missing from the upstream), so we should also add that package toour image, or else we will not have our precious ssh-key added to access any launched instances.

c80f0f1006