Replication network for cluster with more than 2 nodes
Simon Deziel
Hi all,
We'd like to use Ganeti to manage a cluster of 4 hypervisors. Each of
the hypervisor has a few 1Gbps and 2x 10Gbps ports. The 10Gbps were
initially planned for a direct connection between node pairs for the
DRBD replication. Now that we have tried to join a 3rd node to the
cluster, the master complains it can't reach the secondary IP of the
new node.
So the question is, is it possible to have a cluster with >2 nodes and
still use a direct connection between node pairs (no switch involved
for the DRBD replication) ? We can't afford a 10Gbps switch and the
DRBD folks are also recommending to using avoid a switch.
Are we better off with 2 clusters of 2 nodes each ? How about cluster
to cluster VM (live) migration in that case ?
Thanks in advance,
Simon
Iustin Pop
> Replication network for cluster with more than 2 nodes
>
> Hi all,
>
> We'd like to use Ganeti to manage a cluster of 4 hypervisors. Each of
> the hypervisor has a few 1Gbps and 2x 10Gbps ports. The 10Gbps were
> initially planned for a direct connection between node pairs for the
> DRBD replication. Now that we have tried to join a 3rd node to the
> cluster, the master complains it can't reach the secondary IP of the
> new node.
>
> So the question is, is it possible to have a cluster with >2 nodes and
> still use a direct connection between node pairs (no switch involved
> for the DRBD replication) ? We can't afford a 10Gbps switch and the
> DRBD folks are also recommending to using avoid a switch.
Hmm, I haven't heard of this before. Is the switch introducing that much
delay?
> Are we better off with 2 clusters of 2 nodes each ? How about cluster
> to cluster VM (live) migration in that case ?
In that case, you won't get live migration.
The problem is that live migration requires DRBD on both ends; if you
can't get DRBD working all-to-all, then you can't have arbitrary
migrations.
Given that you only have 2x10Gbps ports, it means that any given node
cannot connect directly to the other three; as such, I don't think you
can make it work without a switch… Maybe with 3x10Gbps you could setup a
mesh yourself, and do smart networking layout at Linux level, but I'm
not entirely sure it would work.
iustin
Micha Kersloot
----- Oorspronkelijk bericht -----
> Van: "Simon Deziel" <simon....@gmail.com>
> Aan: "ganeti" <gan...@googlegroups.com>
> Verzonden: Dinsdag 28 februari 2012 15:17:45
> Onderwerp: Replication network for cluster with more than 2 nodes
>
> Replication network for cluster with more than 2 nodes
>
> Hi all,
>
> We'd like to use Ganeti to manage a cluster of 4 hypervisors. Each of
> the hypervisor has a few 1Gbps and 2x 10Gbps ports. The 10Gbps were
> initially planned for a direct connection between node pairs for the
> DRBD replication. Now that we have tried to join a 3rd node to the
> cluster, the master complains it can't reach the secondary IP of the
> new node.
>
> So the question is, is it possible to have a cluster with >2 nodes
> and
> still use a direct connection between node pairs (no switch involved
> for the DRBD replication) ? We can't afford a 10Gbps switch and the
> DRBD folks are also recommending to using avoid a switch.
We've done such a thing with some routing tricks. We didn't use a dual homed setup, but used the routing table. We used the following setup:
ServerA
10.8.1.6 (eth0) (switch)
192.168.1.6 (eth1) (cross cable)
route add -host 10.8.1.14 gw 192.168.1.14
ServerB
10.8.1.14 (eth0) (switch)
192.168.1.6 (eth1) (cross cable)
route add -host 10.8.1.6 gw 192.168.1.6
ServerC (master)
10.8.1.1
You can duplicate the ServerA/B setup many times if you want. Ganeti only knows the 10.8.1.x ipnumbers, 192.168.1.x is just used for routing.
Met vriendelijke groet,
Micha Kersloot
Altijd en overal bij je belangrijkste gegevens?
http://www.exchange-alternatief.com/
KovoKs B.V. is ingeschreven onder KvK nummer: 11033334
Simon Deziel
> On Tue, Feb 28, 2012 at 06:17:45AM -0800, Simon Deziel wrote:
>> Replication network for cluster with more than 2 nodes
>>
>> Hi all,
>>
>> We'd like to use Ganeti to manage a cluster of 4 hypervisors. Each of
>> the hypervisor has a few 1Gbps and 2x 10Gbps ports. The 10Gbps were
>> initially planned for a direct connection between node pairs for the
>> DRBD replication. Now that we have tried to join a 3rd node to the
>> cluster, the master complains it can't reach the secondary IP of the
>> new node.
>>
>> So the question is, is it possible to have a cluster with >2 nodes and
>> still use a direct connection between node pairs (no switch involved
>> for the DRBD replication) ? We can't afford a 10Gbps switch and the
>> DRBD folks are also recommending to using avoid a switch.
>
> Hmm, I haven't heard of this before. Is the switch introducing that much
> delay?
That kind of switch is prohibitively expansive so we won't be able to
test the delay, unfortunately.
>> Are we better off with 2 clusters of 2 nodes each ? How about cluster
>> to cluster VM (live) migration in that case ?
>
> In that case, you won't get live migration.
Hmm, I don't know where I got that impression but I taught that Ganeti
support this kind of migration. Maybe that was just my imagination but
here is what I taught happened in those situation :
1) Break the DRBD mirror
2) Find a new secondary on the remote cluster
3) Sync the primary with the new secondary
4) Promote the new secondary to "primary"
5) Break the DRBD mirror that spans between 2 clusters
6) Find a new secondary in the remote cluster
> The problem is that live migration requires DRBD on both ends; if you
> can't get DRBD working all-to-all, then you can't have arbitrary
> migrations.
>
> Given that you only have 2x10Gbps ports, it means that any given node
> cannot connect directly to the other three; as such, I don't think you
> can make it work without a switch… Maybe with 3x10Gbps you could setup a
> mesh yourself, and do smart networking layout at Linux level, but I'm
> not entirely sure it would work.
I was thinking about a ring topology with the existing 2x 10Gbps ports
on each hypervisors. The idea is to have the 2 local 10Gbps ports
bridged with STP (to avoid loops) and connect each hypervisors with the
one next to it, creating a true circle. Does that sound crazy ?
Regards,
Simon
Simon Deziel
Very interesting idea. I will test this.
Thank you Micha,
Simon
Iustin Pop
> On 12-02-28 09:25 AM, Iustin Pop wrote:
> > On Tue, Feb 28, 2012 at 06:17:45AM -0800, Simon Deziel wrote:
> >> Replication network for cluster with more than 2 nodes
> >>
> >> Hi all,
> >>
> >> We'd like to use Ganeti to manage a cluster of 4 hypervisors. Each of
> >> the hypervisor has a few 1Gbps and 2x 10Gbps ports. The 10Gbps were
> >> initially planned for a direct connection between node pairs for the
> >> DRBD replication. Now that we have tried to join a 3rd node to the
> >> cluster, the master complains it can't reach the secondary IP of the
> >> new node.
> >>
> >> So the question is, is it possible to have a cluster with >2 nodes and
> >> still use a direct connection between node pairs (no switch involved
> >> for the DRBD replication) ? We can't afford a 10Gbps switch and the
> >> DRBD folks are also recommending to using avoid a switch.
> >
> > Hmm, I haven't heard of this before. Is the switch introducing that much
> > delay?
>
> That kind of switch is prohibitively expansive so we won't be able to
> test the delay, unfortunately.
I see. But are there docs where you read this? I'm interested to learn
about it, that's all.
> >> Are we better off with 2 clusters of 2 nodes each ? How about cluster
> >> to cluster VM (live) migration in that case ?
> >
> > In that case, you won't get live migration.
>
> Hmm, I don't know where I got that impression but I taught that Ganeti
> support this kind of migration.
Not across clusters, nope.
> Maybe that was just my imagination but
> here is what I taught happened in those situation :
>
> 1) Break the DRBD mirror
> 2) Find a new secondary on the remote cluster
> 3) Sync the primary with the new secondary
> 4) Promote the new secondary to "primary"
> 5) Break the DRBD mirror that spans between 2 clusters
> 6) Find a new secondary in the remote cluster
Roughly yes (not across clusters, but if you are inside the cluster),
but it means for any (A, B) old primary/old secondary, and new node C,
you need connectivity both A to B and A to C. If you want to do this
'equally' across all nodes, it results, that you will need full mesh.
Or I might misunderstand how you proposed this to work, sorry.
> > The problem is that live migration requires DRBD on both ends; if you
> > can't get DRBD working all-to-all, then you can't have arbitrary
> > migrations.
> >
> > Given that you only have 2x10Gbps ports, it means that any given node
> > cannot connect directly to the other three; as such, I don't think you
> > can make it work without a switch… Maybe with 3x10Gbps you could setup a
> > mesh yourself, and do smart networking layout at Linux level, but I'm
> > not entirely sure it would work.
>
> I was thinking about a ring topology with the existing 2x 10Gbps ports
> on each hypervisors. The idea is to have the 2 local 10Gbps ports
> bridged with STP (to avoid loops) and connect each hypervisors with the
> one next to it, creating a true circle. Does that sound crazy ?
Crazy, but doable :)
If you can make it at Linux level so that ping from any node to all
other nodes work, then yes, Ganeti will be happy. The only downside
could be that the bridging traffic will create load on many machines,
and thus diagnosing performance issues might be more difficult.
iustin
Simon Deziel
> On Tue, Feb 28, 2012 at 09:51:48AM -0500, Simon Deziel wrote:
>> On 12-02-28 09:25 AM, Iustin Pop wrote:
>>> On Tue, Feb 28, 2012 at 06:17:45AM -0800, Simon Deziel wrote:
>>>> Replication network for cluster with more than 2 nodes
>>>>
>>>> Hi all,
>>>>
>>>> We'd like to use Ganeti to manage a cluster of 4 hypervisors. Each of
>>>> the hypervisor has a few 1Gbps and 2x 10Gbps ports. The 10Gbps were
>>>> initially planned for a direct connection between node pairs for the
>>>> DRBD replication. Now that we have tried to join a 3rd node to the
>>>> cluster, the master complains it can't reach the secondary IP of the
>>>> new node.
>>>>
>>>> So the question is, is it possible to have a cluster with >2 nodes and
>>>> still use a direct connection between node pairs (no switch involved
>>>> for the DRBD replication) ? We can't afford a 10Gbps switch and the
>>>> DRBD folks are also recommending to using avoid a switch.
>>>
>>> Hmm, I haven't heard of this before. Is the switch introducing that much
>>> delay?
>>
>> That kind of switch is prohibitively expansive so we won't be able to
>> test the delay, unfortunately.
>
> I see. But are there docs where you read this? I'm interested to learn
> about it, that's all.
http://www.drbd.org/users-guide/s-prepare-network.html :
"It is recommended, though not strictly required, that you run your DRBD
replication over a dedicated connection. At the time of this writing,
the most reasonable choice for this is a direct, back-to-back, Gigabit
Ethernet connection. When DRBD is run over switches, use of redundant
components and the bonding driver (in active-backup mode) is recommended."
>>>> Are we better off with 2 clusters of 2 nodes each ? How about cluster
>>>> to cluster VM (live) migration in that case ?
>>>
>>> In that case, you won't get live migration.
>>
>> Hmm, I don't know where I got that impression but I taught that Ganeti
>> support this kind of migration.
>
> Not across clusters, nope.
>
>> Maybe that was just my imagination but
>> here is what I taught happened in those situation :
>>
>> 1) Break the DRBD mirror
>> 2) Find a new secondary on the remote cluster
>> 3) Sync the primary with the new secondary
>> 4) Promote the new secondary to "primary"
>> 5) Break the DRBD mirror that spans between 2 clusters
>> 6) Find a new secondary in the remote cluster
>
> Roughly yes (not across clusters, but if you are inside the cluster),
> but it means for any (A, B) old primary/old secondary, and new node C,
> you need connectivity both A to B and A to C. If you want to do this
> 'equally' across all nodes, it results, that you will need full mesh.
>
> Or I might misunderstand how you proposed this to work, sorry.
You understood what I was saying. I was simply not aware of the
requirement about direct node communications. I taught that for cluster
to cluster migration that could happen over the primary network (and not
the secondary one).
My confusion comes from the fact that I always assumed that DRBD was
meant for direct connection and would be usable between 2 nodes of a
pair. Now with your explanation I understand how migration relies on
switched DRBD replication network to migrate instance to a different
node pair. Thanks.
>>> The problem is that live migration requires DRBD on both ends; if you
>>> can't get DRBD working all-to-all, then you can't have arbitrary
>>> migrations.
>>>
>>> Given that you only have 2x10Gbps ports, it means that any given node
>>> cannot connect directly to the other three; as such, I don't think you
>>> can make it work without a switch… Maybe with 3x10Gbps you could setup a
>>> mesh yourself, and do smart networking layout at Linux level, but I'm
>>> not entirely sure it would work.
>>
>> I was thinking about a ring topology with the existing 2x 10Gbps ports
>> on each hypervisors. The idea is to have the 2 local 10Gbps ports
>> bridged with STP (to avoid loops) and connect each hypervisors with the
>> one next to it, creating a true circle. Does that sound crazy ?
>
> Crazy, but doable :)
Yeah, the crazy part makes me nervous :)
> If you can make it at Linux level so that ping from any node to all
> other nodes work, then yes, Ganeti will be happy. The only downside
> could be that the bridging traffic will create load on many machines,
> and thus diagnosing performance issues might be more difficult.
Technically, if the instances are properly assigned to node pair
"sitting side by side" the ring wouldn't be involved for normal DRBD
replication. Under normal operation, the only bridging traffic I would
expect is the STP and ARP.
When migrating an instance to another node pair the DRBD traffic would
use the ring temporarily. Such migration should be very rare in our case
and I think the extra load wouldn't hurt too much.
Thanks for you help, I appreciate it,
Simon
Martin Beauchamp
Martín
Martín Beauchamp
Systems Engineer
Park Slope Food Coop, Inc.
To: gan...@googlegroups.com
Sent: Tuesday, February 28, 2012 10:18:27 AM
Subject: Re: Replication network for cluster with more than 2 nodes
Iustin Pop
Ah, I see. Yes, dedicated connection is important, but I don't read in
that quote as switches being bad, just that you need to be careful about
redundancy.
Indeed. During live migration, DRBD needs to be in primary-primary mode,
so…
This makes sense. I would recommend then, as an extra precaution, to
create two node groups:
- node group A, with the 2 nodes that are directly connected
- node group B, with the 2 nodes that are directly connected
This way, automated Ganeti operations won't mistakenly put an instance
on the "wrong" node pairs.
iustin
Simon Deziel
> Hi Simon,
Hi Martín,
> Interesting discussion going on. I would add that in your proposed ring
> topology, you're not following the "spirit" of the advice given in the
> DRBD docs:
>
> "When DRBD is run over switches, use of redundant components and the
> bonding driver (in active-backup mode) is recommended."
I totally agree that the proposed solution is not optimal.
> Though you're not involving a switch, each of the nodes in your ring
> will be connected to its neighbor on either side by one link. Should a
> link die (NIC port failure on either end, cable failure, hand of God,
> etc.) the nodes on either side of the failure will lose DRBD sync.
If the failure does not remove the IP used by DRBD, then I would expect
the STP to find the new path to get to it. That means if a cable is
unplugged, the optimal path involving only 2 nodes would be broken and
new one involving more nodes would take over.
If the failure results in a loss of IP (NIC failure or even hand of
god), then I'm out of luck as STP won't be able to find another path.
The DRBD mirror would be broken.
> And depending on how the traffic is flowing in this ring (I have never used
> such a setup, so I have no clue how this would work), arbitrary nodes
> may become unreachable to other nodes.
I have never used it myself either but can't figure why other nodes
would become unreachable.
> Assuming that the link failure
> just caused your primary and secondary to lose DRBD sync, this would be
> a bad time to have to first reconfigure the network topology in your
> attempt to get things running again.
>
> I run DRBD over crossover cables between nodes (also because a 10GB
> switch is too expensive for the time being), but I have a 2 node cluster
> and use the bonding driver (active-backup) to create NIC teams of 2
> ports each. While this limits my migration options, it is also a much
> simpler configuration to troubleshoot - K.I.S.S.
Your solution is definitely my plan B. I will still give a shot at the
crazy ring topology though, just for the sake of it.
Regards,
Simon
> Best,
> Martín
> --
> Martín Beauchamp
> Systems Engineer
> Park Slope Food Coop, Inc.
>
>
> ------------------------------------------------------------------------
> *From: *"Simon Deziel" <simon....@gmail.com>
> *To: *gan...@googlegroups.com
> *Sent: *Tuesday, February 28, 2012 10:18:27 AM
> *Subject: *Re: Replication network for cluster with more than 2 nodes
Simon Deziel
My understanding is that having a common DRBD network is required for
live migrations. This is what makes cluster to cluster migration
incompatible with a secondary network that only belongs to one cluster.
When not using a secondary network, I don't see why it would be
impossible to do cluster to cluster migrations (assuming DRBD is able to
connect to the new node pair).
I understand the primary-primary requirement but I don't see how that
relates to cluster to cluster migrations ?
Very good point, thanks.
Simon
Iustin Pop
There are no cluster to cluster migrations. There are instance moves,
which involve copying data and work over the primary IP, but they are
not live.
As to why no cluster-to-cluster live migrations, it's simply a matter of
complexity. We decided to introduce node groups as a method to have
heterogeneous clusters, as opposed to having the feature of live
migrations between clusters. It's definitely doable, just complex.
iustin
Jake Anderson
perhaps set up as 2 pairs of nodes with 10gbe,
but bridge that with your normal 1gbe network
so
| Node A |
Node C |
|
| 10G |
1Gbps |
10G |
| Node B |
Node D |
kind of thing
put the 10G networks in their own ip ranges and then add a route to them over the 1gbps link
so flipping from A to B is fast, and A to C is possible (just slower).
you can then have everything bonded everywhere.
Also you don't need to have it active backup when you have them linked by crossover, you can do roundrobin for that because the packets will be rx-ed in order. Any of the both active modes will work for the 1gbps links except for the roundrobin mode as the switch will often make the packets out of order at the other end :-< but the rest of the traffic should still be ok.
Simon Deziel
Just a follow-up to confirm that the ring + STP setup works perfectly
and is reliable. It also offers the same observable performance as the
plain 10GbE direct connection.
Since my ring only has 4 hypervisors, misplacing an instance (not having
the primary and secondary as closest neighbors) does not impact the
performance noticeably. Those misplacements should be fairly rare and
only happen during maintenance if ever.
For those interested, here is the relevant /etc/network/interfaces :
# === DRBD ring configuration ===
# node1_eth4 (192.168.170.11) <-> node4_eth5 (192.168.170.14)
# node4_eth4 (192.168.170.14) <-> node3_eth5 (192.168.170.13)
# node3_eth4 (192.168.170.13) <-> node2_eth5 (192.168.170.12)
# node2_eth4 (192.168.170.12) <-> node1_eth5 (192.168.170.11)
auto drbd0
iface drbd0 inet static
address 192.168.170.13
netmask 255.255.255.0
bridge_ports eth4 eth5
bridge_stp on
bridge_fd 5
bridge_maxwait 10
# Broadcom NetXtreme II BCM57711 XGb supports a MTU of 9000
post-up ifconfig eth4 mtu 9000
post-up ifconfig eth5 mtu 9000
> This makes sense. I would recommend then, as an extra precaution, to
> create two node groups:
>
> - node group A, with the 2 nodes that are directly connected
> - node group B, with the 2 nodes that are directly connected
>
> This way, automated Ganeti operations won't mistakenly put an instance
> on the "wrong" node pairs.
Since a node can only be part of a single group, I don't see how
grouping my nodes 2 by 2 would help in case of a node failure.
Losing a primary or secondary, means that a replacement must be elected.
Grouping would make sure the replacement is part of that group but a
group of 2 with 1 failed node is not going to help here.
The only way I see to benefit from grouping would be to have the ability
to make a node member of 2 groups at the same time. That dual membership
would then account for "close" members.
For example with a 4 nodes cluster made of node1, node2, node3 and node4
connected like this (sorry, I failed my ASCII art class) :
node1 --> node2
|-> node4
node2 --> node1
|-> node3
node3 --> node2
|-> node4
node4 --> node1
|-> node3
With that setup, I think the following membership would be desirable
(even if not possible ATM) :
groupA: node1,node2,node4
groupB: node2,node1,node3
groupC: node3,node4,node2
groupD: node4,node1,node3
It's quite possible that I've missed the point about grouping since I'm
very new to Ganeti.
Regards,
Simon
Simon Deziel
> just a quick thaught
> perhaps set up as 2 pairs of nodes with 10gbe,
> but bridge that with your normal 1gbe network
>
> so
> Node A
>
> Node C
> 10G
> 1Gbps
> 10G
> Node B
>
> Node D
>
>
> kind of thing
> put the 10G networks in their own ip ranges and then add a route to them
> over the 1gbps link
All my attempts to use this routing trick made the cluster to fail
verification. It was always complaining about the directly connected
nodes being unable to connect to the master using the secondary IP. The
other nodes not directly connected to the master had no such problem.
This appeared to be related to the wrong source IP being used when the
node connect to the master using the secondary network.
I'm curious on how you achieved this setup (or the one proposed by
Micha). Could you detail a bit more ?
Regards,
Simon
Micha Kersloot
Simon Deziel <simon....@gmail.com>schreef:
Guido Trotter
You could make that work via policy routing, if needed.
Regards,
Guido