> I think the problem with OpenPKG and the like is that people keep wanting
> one package system.  They want uniformity, either by adopting one technology
> or abstracting them all away.  Translating:
> install_package(type=gem,name=rails,version=3.0)

> Into:

> gem install rails-3.0

> Is practically a weekend project for a college student.  Tracking the three
> pieces of metadata (gem, rails, 3.0) is pretty easy too.

> Why nobody has approached it this way is beyond me.  Plugins that turn name
> + version into a shell command just aren't rocket science.  No need to
> overcomplicate it (i.e. OpenPKG).

> The reason that this is important is that it means that you won't have to
> have "cloud packages".  Insisting on "cloud packages" basically takes the
> hardest thing solved by OS vendors (getting software to build and install)
> and makes you duplicate it.  It's why people end up standardizing on RedHat
> or something (just to avoid the problem).

> Now, I readily admit that this has various gotchas (most notably in the
> realm of accidentally installing tons of dependencies or uninstalling
> something that is conflicting).  However, in the Cloud, most systems are
> generated once, then thrown away (EC2 for example).  In this realm, you can
> afford to have an imperfect packaging interface.  If you need it to be
> smarter, write a new plug-in.  Meanwhile, everyone else gets work done.

> Similarly, code like Java, Python, and Ruby have special packaging needs
> that we really don't have the resources to track.  If we want to keep the
> leading edge of software in the Cloud (and we want to realize the benefit of
> the Cloud for staging, testing, and continuous integration), then we need to
> offer an interface for the code monkeys to work their magic.

> I think that "standardizing" on a few technologies won't really work in the
> Cloud.  If we look at the development going on, it's clear that "Windows",
> "Red Hat Linux", "OpenSolaris", "Ruby", "Java", etc. aren't going to
> individually win the Cloud.  It defies exclusive standardization because the
> whole point of the Cloud is mashing those sorts of resources together.
>  Rather than trying to standardize on one technology or abstract them all
> behind a perfect API, I think we're going to get the most traction out of
> putting up a functional but imperfect interface that can leverage the
> strengths of each player through plugins.

> Look at the EC2 interface.  That's exactly what it is.  They don't try to
> be generic.  They just try to work with their abstractions, and let you
> monkey with yours.  Incremental development of many, simple, imperfect
> interfaces will get us there faster than working on abstractions or
> excluding wide swaths of the market.  I'd rather offer them a place to
> integrate, and you'll be surprised what will happen.

> <shameless>Mind you, we should all just adopt Vertebra (my company's
> project), and the world will be a better place.</shameless>  :)

> On Mar 10, 2009, at 4:16 AM, Sam Johnston wrote:

> G'day Jayson,

> You'll see that I left you off the list because I didn't hear back from you
> straight away, but I appreciate this feedback. I'm all over packaging
> (having been a sysadmin at various university departments, a windows
> developer and a debian developer) but I don't think it fits well here, and
> apparently most of the existing implementations agree. Packaging adds
> another layer of complexity for developers and previous attempts to "herd
> cats" in this area (e.g. OpenPKG) have not been particularly successful.
> It's also not immediately clear to me which layer (IaaS, PaaS, SaaS) this
> belongs, or even whether it belongs on the user facing side or behind the
> scenes (where it is more of a "best practice" than a "standard").
> User-facing standards and APIs are the most important ones to implement, and
> then you can leave the details to the vendor (which is the whole idea behind
> cloud) who is free to innovate - for example Google use BigTable for most of
> their stuff and that's definitely not compatible with most packaging
> systems.

> An API call to deploy an arbitrary package on a given machine would however
> be interesting (where supported packages depends on the platform - no point
> trying to get every machine to support everything... e.g MSI on Debian).
> Importing/exporting existing applications via such an API could be cool too,
> though in reality it's just uploading/download a file or specifying a URL in
> much the same way that salesforce's API allows you to upload/download a ZIP
> file of everything if you don't want to use CRUD calls.

> Your comment about data stores is interesting - this was another thing I
> figured would be a user-facing API implemented however the vendor saw fit
> (backing onto a JBOD/RAID, SAN or something more elegant like GFS - details
> only visible to the user in the price and performance).

> You can copy in cloud-standards@googlegroups.com in your reply if you
> don't mind this conversation being public - I'd like to hear what others
> think.

> Sam

> On Tue, Mar 10, 2009 at 8:28 AM, Jayson Vantuyl <jvant...@engineyard.com>wrote:

>> Honestly, I don't really know that we CAN endorse getting the code out
>> there in that simple of a way.  I think we should really support the ability
>> to delegate this to a plug-in.  The problem is that if we really want to be
>> "the Cloud", we have to embrace the differing distribution methods
>> available.  I think we'll niche ourselves.  To put this in perspective, I'd
>> suggest we come up with a packaging API, with the following plugins:
>> * rubygems for Ruby code
>> * eggs for Python code
>> * msinstaller for Windows Binary Distribution
>> * portage for Gentoo Distribution
>> * apt for Debian / Ubuntu Distribution
>> * tarZ for random old Unixes
>> * etc.

>> The "packaging" space is already very crowded and very mature.  Each of
>> the above systems is much better to work with than to work against.  Writing
>> a plug-in to wrap these is also probably superior to attempting to
>> reimplement them without the resources to really do it well.  Also, the
>> above creates a wonderful opportunity for people to "get involved" and
>> contribute to the stack.  That kind of ownership and community building is
>> probably what the CCIF is really lacking the most.

>> For the configuration metadata, Git or HTTPS PUT could be workable,
>> although I really suspect that we should talk about Cloud data stores, and
>> endorse the right one for the stack.  I'm not yet sure which one that is.

>> On Mar 9, 2009, at 2:26 PM, Sam Johnston wrote:

>> Jayson,

>> Ok, so you'd be comfortable with something like the attached stack then?
>> WebDAV layer would be optional (e.g. raw HTTPS PUT is fine). Forget about
>> SSH key management - that's well outside of the realms of simplicity.

>> Sam

>> On Mon, Mar 9, 2009 at 9:01 PM, Jayson Vantuyl <jvant...@engineyard.com>wrote:

>>> Our answer for new projects is Git.
>>> We have some legacy SVN, some Puppet, and some Chef recipes that pull
>>> down stuff and build it.  We also pull some of our stuff in through our own
>>> portage overlay (using Gentoo), and I've done similar things at other
>>> companies with Debian.  Since we're Ruby-heavy, we also cheat by
>>> distributing a lot of code as Ruby gems.  I'd imagine some Python shops are
>>> the same way with easy_install.  Ditto for Erlang with their tarball based
>>> "releases".  We're not really a very "pure" example internally.  Lots of
>>> developers made lots of decisions.  :(
>>> Git does everything you want generally.  It also is great for
>>> decentralized configuration systems.  That said, secure Git requires SSH key
>>> management.  Branch management can also be a pain when you start building
>>> tiers of configuration with common history (i.e. one slip and you merge two
>>> branches on accident).

>>> The SSH key management is probably the most contentious thing.
>>>  Especially if Windows or a weird Unix is in the mix.  HTTP PUSH would
>>> probably be a better bet for raw accessibility and when SSH isn't enough of
>>> a security model.

>>> On Mar 9, 2009, at 12:54 PM, Sam Johnston wrote:

>>> On Mon, Mar 9, 2009 at 8:36 PM, Jayson Vantuyl <jvant...@engineyard.com>wrote:

>>>> No problem.  I think he's about to learn what it's like to be on the
>>>> receiving end of "democracy".

>>> Yup, and it's about time too.

>>> So just quickly, I'm looking at platform standards - specifically how
>>> people get their code in and out of the platform[1]. ReasonablySmart uses
>>> Git, App Engine uses HTTP PUSH - what do you guys use?

>>> Sam

>>> 1. http://wiki.cloudcommunity.org/wiki/Standards#SCM

>>>> --
>>>>     Jayson Vantuyl
>>>> Founder and Architect
>>>> *Engine Yard <http://www.engineyard.com>*
>>>> jvant...@engineyard.com
>>>> 1 866 518 9275 ext 204
>>>> IRC (freenode): kagato

>>>     --
>>> Jayson Vantuyl
>>> Founder and Architect
>>> *Engine Yard <http://www.engineyard.com>*
>>> jvant...@engineyard.com
>>> 1 866 518 9275 ext 204
>>> IRC (freenode): kagato

>> <cloud-platform.png><cloud-platform.svg>

>>     --
>> Jayson Vantuyl
>> Founder and Architect
>> *Engine Yard <http://www.engineyard.com>*
>> jvant...@engineyard.com
>> 1 866 518 9275 ext 204
>> IRC (freenode): kagato

> --
> Jayson Vantuyl
> Founder and Architect
> *Engine Yard <http://www.engineyard.com>*
> jvant...@engineyard.com
> 1 866 518 9275 ext 204
> IRC (freenode): kagato