Making OpenStack more delightful (or: managing OpenStack at scale)

[Samuel Cassiba]

Ohai!

I wanted to bring a certain hairy yak that I feel needs shaving to the group, to generate some conversation on how it might be approached conceptually: managing OpenStack using the cookbooks

Currently, as I understand it, in order to "properly" do an upgrade of OpenStack using the cookbooks intra-release, the convention is to build out a new infrastructure, forklift the tenants, rebuild the old. As a reformed sysadmin, I'm perfectly okay with that being the current answer, but there should be a better way going down the road.

As far as I understand, there is no safe way to move from Juno to Kilo, or Kilo to Liberty, etc., without doing what I said above. Of course, this is not the first time this has come up. I've found https://blueprints.launchpad.net/openstack-chef/+spec/openstack-upgrade that began to outline the then-current state of things, and some old ML posts. I've done some poking around and things don't seem much different today.

It's still risky business if remotely feasible for some bits, but from my understanding of how the cookbooks work, they just can't allow for it at present. That, my friends, is not only deeply concerning but downright frightening, as it takes Real People Time to rebuild and test fleets of gear, and it's not without potential for error and downtime. The fewer manual steps we can have in the lifecycle, the more delightful of an experience can be had for all, operators and stakeholders alike.

From a high level, compute would probably be the least risky part to tackle first, given how few moving parts it has compared to a controller and how naturally compute nodes outnumber controllers in a deployment. The most risky would be the database, rabbit and nova-network/SDN.

I don't want to stray to deep in the weeds yet if it doesn't make sense, but want to put it out there to get some perspective from others.

Best,

Samuel

[cmlu...@dragoon.io]

I agree that this is not very delightful. I suppose the reason that we
have not dedicated a lot of time to this is because the steps that are
normally outlined are pretty hairy.

Upgrading openstack seems to slightly differ everytime, so I think each
release would need a separate customized set of recipes for upgrading.

I suppose the hardest problems would be: how do I make sure that custom
sql queries are only run once, should I create a separate recipe for
each component called upgrade and how do I remove that from my runlist
immediately after performed, should this be it's own upgrade cookbook?

We also need to consider how we will support upgrading since things can
and probably will fail. In this case how can we get to a good state if
the upgrade is unsuccessful.

I think we can definitely bring this up during a weekly meeting in order
to "trioratize" if properly. That said I am interested how other CM
systems deal with upgrades and I have a couple of contacts that I can
reach out to.

I may also run icehouse and then try to run Junoon top of it to see what
happens just for a POC start.

> --
> You received this message because you are subscribed to the Google Groups
> "opscode-chef-openstack" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opscode-chef-open...@googlegroups.com.
> To post to this group, send email to
> opscode-che...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/opscode-chef-openstack/CAOqvHK_m%3DKFhii7XY2HoNB9SyLBOk-%2Bi%2BSxVbMYr%2B4QgGVX7qA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Samuel Cassiba]

On Sun, Apr 5, 2015 at 9:04 AM, <cmlu...@dragoon.io> wrote:

I agree that this is not very delightful. I suppose the reason that we
have not dedicated a lot of time to this is because the steps that are
normally outlined are pretty hairy.

Yup. I've done Grizzly to Havana, and that was about as painful as hand-rolling an OpenStack deploy soup to nuts, and took about as long.

 

Upgrading openstack seems to slightly differ everytime, so I think each
release would need a separate customized set of recipes for upgrading.

I suppose the hardest problems would be: how do I make sure that custom
sql queries are only run once, should I create a separate recipe for
each component called upgrade and how do I remove that from my runlist
immediately after performed, should this be it's own upgrade cookbook?

That's what I was thinking. The necessary moving parts seem like they're too much to really put them in the individual cookbooks, but a centralized shim could make it easier, since the process is a moving target from release to release. Over time, that could be refined into what really makes sense so we can get some lifecycle up in here.

We also need to consider how we will support upgrading since things can
and probably will fail. In this case how can we get to a good state if
the upgrade is unsuccessful.

I don't have a good suggestion off the top of my head, but getting to a good state in the light of failure is an important one to me. We can definitely break things down as time goes on.

 

I think we can definitely bring this up during a weekly meeting in order
to "trioratize" if properly. That said I am interested how other CM
systems deal with upgrades and I have a couple of contacts that I can
reach out to.

Cool. Will definitely be good to get some more conversation about this going and see how others are dealing with it with !Chef.

I may also run icehouse and then try to run Junoon top of it to see what
happens just for a POC start.

I look forward to seeing the fireworks.

To view this discussion on the web visit https://groups.google.com/d/msgid/opscode-chef-openstack/1428249876.3659749.249521097.0DEAED4F%40webmail.messagingengine.com.