Aminator vs Bakery performance and tradeoffs

[Richard Gooch]

  Hi, all. I'm building a high performance AMI construction and distribution system (currently focused on EBS-backed AMIs). While working on this I came across this blog post:
http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html

While I was already aware of Aminator, what drew my attention was the description of the Bakery and how Aminator dropped one of it's core techniques: using a volume pool. The system I'm building is based on a similar concept to improve performance. The blog post states that volume pools were dropped for agility and manageability. I have two basic questions for the Aminator team:

1) what were the agility and manageability issues experienced with Bakery?

2) what was the degree of performance benefit that was sacrificed by giving up volume pools?

Similar to the Bakery, the system I'm building can produce encrypted AMIs which are globally available in different accounts and regions from a (optionally compressed) tarfile of the OS in under 5 minutes. Unlike the Bakery, however, content construction is performed just once rather than in each account/region, which yields guaranteed identical image contents across accounts/regions and also supports building content on-premises (I work at Symantec where we're more comfortable with keeping source code and sensitive key materials on-premises). It also ties in more closely with a system to perform image-based live patching of running systems, which avoids some of the issues with frequent re-launching of instances.

BTW: based on my experience, the time to create snapshots from volumes is the dominant factor in how long it takes to build AMIs, and using volume pools can halve that time from ~6 minutes to ~3 minutes.

Regards,

Richard....

[Brian Moyles]

Actually we do use pools again, but do so internally and haven't released the software to manage that. The big advantage of pools is it helps take away one more bit of uncertainty (create volume/attach volume) leaving the sole long-running piece (aside from the users' package installations) to be the snapshot (which we too have found to be the big limiting factor).  It also took the cleanup of resources out of aminator's hands.

Our pool manager is a separate service that simply maintains a fixed number of volumes (possibly from more than one frequently used base AMI) and uses the aminator argument for specifying an existing block device (which signals to aminator to simply do the mount/chroot/unmount but leave the cleanup to the pool manager).

--
You received this message because you are subscribed to the Google Groups "Aminator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aminator+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Moyles]

To add to something else you mentioned -- ideally we too would create an AMI once and then use something like ec2-copy-image to copy images from region-to-region but have found that doing bakes in parallel per-region gets people to a launchable state much quicker than copy-image as the copy-image task is also quite slow -- often far slower than the install/snapshot process.

I realized I didn't address your first question re: agility/management. Aminator is still part of a suite of services we refer to as "the bakery" so the bakery as-described in that article hasn't really gone away per se, but has evolved.

Our previous bakery iteration was a set of organically-grown shell scripts that were well past the point of "we should probably move to a real programming language" :) The interface to a bake was strictly via command line tool with no way for a user to request a bake and query status later, required that users kept their shell open or use screen/tmux or otherwise capture output, etc.

The current bakery consists of aminator (the only component we've released so far), an API service that provides a http-based interface to starting bakes and controlling jobs, and worker processes that receive instructions from the API service and coordinates with both poolman and aminator on instances.

I don't suspect we'll be releasing the current bakery services (we'd need to do a fair bit of cleanup and de-netflixifcation) but instead you might see something a la aminator better integrated into http://spinnaker.io/ in the future with Spinnaker handling the scheduling and queueing.

[Richard Gooch]

Cool that you're using pools again. Hard to give up the performance advantage. I'm actually allocating one volume per image stream (as "stream" is a series of images with the same software stack where each image is basically a new revision/release). Even with hundreds of streams, this is still pretty cheap and is a small fraction of the instance costs of the stream unpackers.

To unsubscribe from this group and stop receiving emails from it, send an email to aminator+u...@googlegroups.com.

[Richard Gooch]

Oh, yeah. Copying AMIs between regions is slow. 10 minutes on a good day from our experience. However, I don't ship AMIs around. I'm leveraging the system used for doing live image-based patching (the Dominator) to copy image content around: I only move new objects. It's typically 3 seconds to copy fresh content from one region to another. Since copying the content is now basically free, it makes it a viable model to have a central content builder.

I hear you about evolving from scripts :-) We started out with Spinnaker, Jenkins jobs and home-brewed scripts, but more of the components are being implemented in Go and are proper services. Spinnaker is very nice for developing image build pipelines. All the meaty code is public on GitHub and has nothing proprietary/magic that's tied to the company.

To unsubscribe from this group and stop receiving emails from it, send an email to aminator+u...@googlegroups.com.

[Richard Gooch]

FYI: Here is a design document describing the AMI publishing system:

https://docs.google.com/document/d/1UL20UZuYvjyA1mVzU2jy9QFvflTaxJz5qOf643JG1EQ/pub