Load Balancing using 2 NICS (same network) with (iproute2) multipath route
Christopher Markov
I惴 trying to set up load balancing on a Linux (Trustix) box, which is acting as
a firewall + HTTP/FTP proxy (Squid) + NAT gateway for an internal network.
It has 3 network interfaces: two that connect to the external world (Internet)
and one that connects to the internal network. The external connection is
currently going through two Cisco routers, each with a 1Mbps link. The external
network interfaces are on the same network, together with the two Ciscos.
I惴 currently balancing outgoing traffic using iproute2愀 ip util and a
multipath route with equal weights. However, I lack the confidence that this is
indeed the proper way of doing such thing, specially because since this
configuration has been adopted, the internal network users have been
experiencing some odd network behavior. Some regular web sites just became hard
to access (usually returning 500 Internal Server Errors) and, in particular,
on-line banking sites started failing miserably.
We concluded that, perhaps, since the banking sites use secure connections (SSL)
and some kind of application validation, it might get confused if some packets
arrive with a source address of the first external NIC and others with a source
address of the second external NIC. We added static routes to those sites, but
we still get trouble. Still, I thought routes were cached and thus, all packets
flowing to a specific site would (at least for some time) flow out using the
same external NIC.
Does iproute2愀 multipath route balance each packet as they come or does it work
based on connections? Additionaly, is this the proper way of doing such
balancing? Are there other alternatives, perhaps channel bonding or such?
I惴 including relevant config information below (IPs slightly changed to protect
the innocent, netmasks preserved). I悲 appreciate any help, suggestion or report
on the subject.
--
eth0 (external 1) -> 209.X.X.19 / 255.255.255.240
eth1 (external 2) -> 209.X.X.22 / 255.255.255.240
eth2 (internal) -> 192.168.1.1 / 255.255.255.0
Cisco 1 (1Mbps) -> 209.X.X.17 / 255.255.255.240
Cisco 2 (1Mbps) -> 209.X.X.20 / 255.255.255.240
--
root@hal ~# ip route show
209.X.X.16/28 dev eth0 scope link
209.X.X.16/28 dev eth1 proto kernel scope link src 209.X.X.22
192.168.1.1/24 dev eth2 scope link
127.0.0.0/8 dev lo scope link
default
nexthop via 200.X.X.17 dev eth0 weight 1
nexthop via 200.X.X.20 dev eth1 weight 1
root@hal ~#
--
root@hal ~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
209.X.X.192 0.0.0.0 255.255.255.240 U 0 0 0 eth0
209.X.X.192 0.0.0.0 255.255.255.240 U 0 0 0 eth1
192.168.1.1 0.0.0.0 255.255.255.0 U 0 0 0 eth2
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 209.X.X.19 0.0.0.0 UG 0 0 0 eth0
root@hal ~#
--
Here愀 how I惴 adding the multipath route:
# to add a second route, to the same network, through a different interface.
otherwise we get two routes using eth1.
ip route replace 209.X.X.16/28 dev eth0
#
ip route add 209.X.X.16/28 dev eth0 src 209.X.X.19 table firstlink
ip route add default via 209.X.X.17 table firstlink
ip route add 209.X.X.16/28 dev eth1 src 209.X.X.22 table secondlink
ip route add default via 209.X.X.20 table secondlink
ip rule add from 209.X.X.19 table firstlink
ip rule add from 209.X.X.22 table secondlink
ip route add default scope global nexthop via 200.X.X.17 dev eth0 weight 1
nexthop via 200.X.X.20 dev eth1 weight 1
--
And here are some relevant iptables rules:
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j SNAT --to-source
209.X.X.19
iptables -t nat -A POSTROUTING -o eth1 -s 192.168.1.0/24 -j SNAT --to-source
209.X.X.22
--
Regards,
Christopher Markov
P Gentry
Excuse the top posting -- seems more appropriate.
I do think your setup will route packets out alternate interfaces,
ie., packetA1 out eth0, packetA2 out eth1, packetA3 out eth0, etc. I
know that's the default behavior on Ciscos.
On Ciscos you can configure load balancing by alternating by
_connection_ rather than alternating by _packet_, and I assume you can
do the same with Linux.
However, you do have some complicating factors.
-- Eth0 and eth1 are on the same subnet which can cause arp problems.
-- You have no way to tell if _apps_ on the other end of the
connection are "tracking" connection/transaction sources -- would
assume that a banking app would. Are these apps running SET rather
than/with SSL?
-- Some protocols are very touchy/picky re: NATing so you'll have to
sniff the wire to see if any protocols you're using are one of these,
eg., FTP, IPSec.
-- Some platform IP stacks are upset with what appear to be random
arrival order of packets, ie., you have no way to assure the
traversal/arrival order at the far end of outgoing packets.
I'm sure there are other possible effects as well that could cause
problems. At the least I would try to have packets alternating routes
by connection (rather than by arrival order -- you should be able to
sniff if this is the case to confirm) and would place eth0 and eth1 on
separate subnets if possible. Also double check that the Ciscos
aren't adding to the possbile confusion on their end.
I'm only just now digging into the deeper details of iputils and
netfilter combinations/interactions so I won't presume to offer any
advice beyond the above. You're likely going to need to implement
some additional ip routing tables and play with the PRDB rules. And
get used to sniffing the wire.
There are a number of info sources. The two I'm working through at
the moment are:
Guide to IP Layer Network Administration with Linux
http://linux-ip.net/
and
Linux Advanced Routing & Traffic Control HOWTO
http://lartc.org/#download
I would be interested to know how you solve this, but I'm sure you
will have your hands full sorting this out, so ...
Good Luck,
prg
email above disabled
> Greetings,
>
> I惴 trying to set up load balancing on a Linux (Trustix) box, which is acting as
> a firewall + HTTP/FTP proxy (Squid) + NAT gateway for an internal network.
>
> It has 3 network interfaces: two that connect to the external world (Internet)
> and one that connects to the internal network. The external connection is
> currently going through two Cisco routers, each with a 1Mbps link. The external
> network interfaces are on the same network, together with the two Ciscos.
>
> I惴 currently balancing outgoing traffic using iproute2愀 ip util and a
> multipath route with equal weights. However, I lack the confidence that this is
> indeed the proper way of doing such thing, specially because since this
> configuration has been adopted, the internal network users have been
> experiencing some odd network behavior. Some regular web sites just became hard
> to access (usually returning 500 Internal Server Errors) and, in particular,
> on-line banking sites started failing miserably.
>
> We concluded that, perhaps, since the banking sites use secure connections (SSL)
> and some kind of application validation, it might get confused if some packets
> arrive with a source address of the first external NIC and others with a source
> address of the second external NIC. We added static routes to those sites, but
> we still get trouble. Still, I thought routes were cached and thus, all packets
> flowing to a specific site would (at least for some time) flow out using the
> same external NIC.
>
> Does iproute2愀 multipath route balance each packet as they come or does it work
> based on connections? Additionaly, is this the proper way of doing such
> balancing? Are there other alternatives, perhaps channel bonding or such?
[setup rules/data snipped]
Cameron Kerr
> -- Some protocols are very touchy/picky re: NATing so you'll have to
> sniff the wire to see if any protocols you're using are one of these,
> eg., FTP, IPSec.
The new iptable modules SAME can be used to fix this. It sends reply
packets back out the same public IP address it came in on.
--
Cameron Kerr
camero...@paradise.net.nz : http://nzgeeks.org/cameron/
Empowered by Perl!
P Gentry
> P Gentry <rdge...@cablelynx.com> wrote:
>
> > -- Some protocols are very touchy/picky re: NATing so you'll have to
> > sniff the wire to see if any protocols you're using are one of these,
> > eg., FTP, IPSec.
>
> The new iptable modules SAME can be used to fix this. It sends reply
> packets back out the same public IP address it came in on.
Thanks for the info about SAME -- I was rather dreading trying this
with iputils tables and policy routing.
regards,
prg
email above disabled