ganeti plus pacemaker?

[Miles Fidelman]

Hi Folks,

Last time I looked at Ganeti, it didn't do automatic failover of VMs
(e.g., after a hardware failure). Has this changed recently? Or, has
anybody tried using pacemaker w/ Ganeti?

Thanks,

Miles Fidelman

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

[Michael Hanselmann]

2012/10/9 Miles Fidelman <mfid...@meetinghouse.net>:

> Last time I looked at Ganeti, it didn't do automatic failover of VMs (e.g.,
> after a hardware failure). Has this changed recently?

No, it hasn't, sorry.

Michael

[Guido Trotter]

For version 2.6.2 or 2.7 we plan an autorepair tool (see doc/design-autorepair.rst) that should solve this, although at non pacemaker speed and only if the node is set offline (eg by pacemaker itself) . I played with pacemaker for Ganeti but haven't finished the work. I can check again what I did and send it, and if you run it in production and modify/finish it maybe we can integrate it!

Thanks!

Guido

[tschend]

Hi Guido,

i am also intersted in this and want to take a look.

This is also on our list but we had not time to get to this for now.

Regards
Thomas

On Oct 11, 8:21 pm, Guido Trotter <ultrot...@google.com> wrote:
> For version 2.6.2 or 2.7 we plan an autorepair tool (see
> doc/design-autorepair.rst) that should solve this, although at non
> pacemaker speed and only if the node is set offline (eg by pacemaker
> itself) . I played with pacemaker for Ganeti but haven't finished the work.
> I can check again what I did and send it, and if you run it in production
> and modify/finish it maybe we can integrate it!
>
> Thanks!
>
> Guido

[Gregory]

Count myself for testing too. I'm highly interested in such a setup
because today I've to separate my systems in two types clusters -
Ganeti based and HA/automatic fail-over ones.

I'd really love to merge the two, even at a cost of slightly degraded
fail-over time.

Thanks!

Gregory.
==

On Oct 11, 2:21 pm, Guido Trotter <ultrot...@google.com> wrote:
> For version 2.6.2 or 2.7 we plan an autorepair tool (see
> doc/design-autorepair.rst) that should solve this, although at non
> pacemaker speed and only if the node is set offline (eg by pacemaker
> itself) . I played with pacemaker for Ganeti but haven't finished the work.
> I can check again what I did and send it, and if you run it in production
> and modify/finish it maybe we can integrate it!
>
> Thanks!
>
> Guido

[Miles Fidelman]

That's about where I am as well. On our list, but no time. I was
hoping someone had done it already.

In the short term, I'm wondering if anyone has simply installed
Pacemaker on a Ganeti installation, and if there are any incompatibilities.

[Guido Trotter]

I had both installed with no incompatibility. I'll look for my config and try to share it tomorrow.

Thanks

Guido

[Miles Fidelman]

That would be great. Thanks!

Guido Trotter wrote:
>
> I had both installed with no incompatibility. I'll look for my config
> and try to share it tomorrow.
>
> Thanks
>
> Guido
>
> On 18 Oct 2012 17:57, "Miles Fidelman" <mfid...@meetinghouse.net

> <mailto:mfid...@meetinghouse.net>> wrote:
>
> That's about where I am as well. On our list, but no time. I was
> hoping someone had done it already.
>
> In the short term, I'm wondering if anyone has simply installed
> Pacemaker on a Ganeti installation, and if there are any
> incompatibilities.
>
> tschend wrote:
>
> Hi Guido,
>
> i am also intersted in this and want to take a look.
>
> This is also on our list but we had not time to get to this
> for now.
>
> Regards
> Thomas
>
> On Oct 11, 8:21 pm, Guido Trotter <ultrot...@google.com

> <mailto:ultrot...@google.com>> wrote:
>
> For version 2.6.2 or 2.7 we plan an autorepair tool (see
> doc/design-autorepair.rst) that should solve this,
> although at non
> pacemaker speed and only if the node is set offline (eg by
> pacemaker
> itself) . I played with pacemaker for Ganeti but haven't
> finished the work.
> I can check again what I did and send it, and if you run
> it in production
> and modify/finish it maybe we can integrate it!
>
> Thanks!
>
> Guido
> On 9 Oct 2012 01:17, "Miles Fidelman"
> <mfidel...@meetinghouse.net

[Guido Trotter]

I've found my experiments, and in particular they were about
integrating Ganeti with corosync.
I posted them to the dev list, and will try to polish them a bit more
and make sure they can be submitted to the code base. Feel free to
test with them and contribute changes, and keep in mind that it's
purely experimental code, that I've only run on a test cluster.

Thanks,

Guido

[Jean-François Maeyhieux]

Our cluster is ready to try this since we are looking at this kind of solution.

We have started our cluster based on libvirt/kvm/drbd and pacemaker/corosync but now we have switched for ganeti since the management is really more easy especially at drbd/lvm level. But the lack of node monitoring to launch specific action for migrate/rebuild disk/... is really a lack to have a high availability cluster.

So use pacemaker/corosync at the top of ganeti seems to be a solution.

That make me wonder about drbd specific problems like the lost of integrity on the primary drbd volume.

So we are OK to test such a solution with specific crash scenario like lost of a node, of network, disks...

But I don't know what is the ganeti limitations for such scenarios on a 2 node cluster.

It seems that a 2 node cluster could brings some specific problem.

If anyone have some explanation about this point, you're welcome.

Jean-François.

[Guido Trotter]

Hi,

The problem on a "pure" 2 node cluster is that you don't have a way to
decide whether the other node is down or you're shut off from the
network, since there is no quorum to ask to. This applies both in
ganeti (when we do master-failover this is why on a 2 node cluster
with an offline node you need to pass --no-voting) and in corosync. I
am not sure there is any good solution, short of having specialized
hardware to arbitrate between the two conditions (a stonith method,
for example, which incurs in the risk of both nodes trying to kill
each other, or an extra communication system, which could increase
your confidence although still not give you certainty).

thanks,

Guido

[Jérémie Tarot]

Hi,

I'd too be interested in testing HA for Ganeti. I'm currently using Proxmox mainly because of VM failover support, but I don't know Perl and really like Python, design docs...

Under Proxmox, and RHCS generally, the answer to the 2 node quorum problem seems to be the Quorum Disk:

http://magazine.redhat.com/2007/12/19/enhancing-cluster-quorum-with-qdisk/

Bests

2012/10/23 Guido Trotter <ultr...@gmail.com>

--
--
Jérémie Tarot
http://about.me/silopolis

[Guido Trotter]

Hi Jérémie,

Feel free to experiment with the .ocf files that I committed to the
"master" tree (there's no reason for them to be 2.7 specific so you
can try them on a 2.6 cluster as well). Of course we welcome any
improvement you may have after running them in production.

Possible improvements:
- example linux-ha configuration for ganeti (right now just the ocf
files are provided)
- hooks to handle cluster initi, and node add and remove
- better use of fencing (ip fencing, ganeti based ipmi, gnt-node poweroff)

Note that ganeti doesn't require shared storage, so we can't depend on
solving the quorum problem as the article mentions, but I'm sure it
can be added optionally, if you'd like to contribute something about
it.

Thanks,

Guido

--
Guido Trotter
Ganeti engineering
Google Germany