equallogic and double connections for multipathing with Debian 5

[Laurent]

Hello,
Thank you for the work. This is my 1st post here.

Under a Debian OS, i try to acces by two different paths an Equallogic
PS 6000.

On the server i configured two NICs

eth1 Link encap:Ethernet HWaddr b8:ac:6f:12:38:c9
inet adr:192.168.15.11 Bcast:192.168.15.255 Masque:
255.255.255.0

eth2 Link encap:Ethernet HWaddr b8:ac:6f:12:38:cb
inet adr:192.168.15.12 Bcast:192.168.15.255 Masque:
255.255.255.0
On the same vlan than the SAN

First, the discovery works fine:

iscsiadm -m discovery -t st -I eth1 -I eth2 -p 192.168.15.1 -P 1 -o
delete -o new
Target: iqn.2001-05.com.equallogic:
0-8a0906-0dd652407-00e000e7b254c345-testdeb
Portal: 192.168.15.1:3260,1
Iface Name: eth2
Iface Name: eth1

With EQL, there is only one IP address for the portal, regardless of
how many times i have to access it.

I try to connect via the two nics:

First one:
#iscsiadm -m node --targetname iqn.2001-05.com.equallogic:
0-8a0906-0dd652407-00e000e7b254c345-testdeb -I eth1 --portal
192.168.15.1:3260,13 --login
Logging in to [iface: eth1, target: iqn.2001-05.com.equallogic:
0-8a0906-0dd652407-00e000e7b254c345-testdeb, portal:
192.168.15.1,3260]
Logging in to [iface: eth1, target: iqn.2001-05.com.equallogic:
0-8a0906-0dd652407-00e000e7b254c345-testdeb, portal:
192.168.15.1,3260]
Login to [iface: eth1, target: iqn.2001-05.com.equallogic:
0-8a0906-0dd652407-00e000e7b254c345-testdeb, portal:
192.168.15.1,3260]: successful
#

All is fine with the first one.
ls /dev/sdb
/dev/sdb
mount /dev/sdb /media/testDB/

Jul 9 00:42:10 sx1 kernel: [ 3318.053517] scsi 9:0:0:0: Direct-
Access EQLOGIC 100E-00 4.3 PQ: 0 ANSI: 5
Jul 9 00:42:10 sx1 kernel: [ 3318.057042] sd 9:0:0:0: [sdb] 209725440
512-byte hardware sectors (107379 MB)
Jul 9 00:42:10 sx1 kernel: [ 3318.057042] sd 9:0:0:0: [sdb] Write
Protect is off
Jul 9 00:42:10 sx1 kernel: [ 3318.057379] sd 9:0:0:0: [sdb] Write
cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jul 9 00:42:10 sx1 kernel: [ 3318.057734] sd 9:0:0:0: [sdb] 209725440
512-byte hardware sectors (107379 MB)
Jul 9 00:42:10 sx1 kernel: [ 3318.058035] sd 9:0:0:0: [sdb] Write
Protect is off
Jul 9 00:42:10 sx1 kernel: [ 3318.059480] sd 9:0:0:0: [sdb] Write
cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jul 9 00:42:10 sx1 kernel: [ 3318.059527] sdb: unknown partition
table
Jul 9 00:42:10 sx1 kernel: [ 3318.060635] sd 9:0:0:0: [sdb] Attached
SCSI disk
Jul 9 00:42:10 sx1 kernel: [ 3318.060686] sd 9:0:0:0: Attached scsi
generic sg3 type 0
Jul 9 00:43:51 sx1 kernel: [ 3419.896300] kjournald starting. Commit
interval 5 seconds
Jul 9 00:43:51 sx1 kernel: [ 3419.915116] EXT3 FS on sdb, internal
journal
Jul 9 00:43:51 sx1 kernel: [ 3419.915116] EXT3-fs: recovery complete.
Jul 9 00:43:51 sx1 kernel: [ 3419.915396] EXT3-fs: mounted filesystem
with ordered data mode.



With eth2,, nothing happens.
iscsiadm -m node --targetname iqn.2001-05.com.equallogic:
0-8a0906-0dd652407-00e000e7b254c345-testdeb -I eth2 --portal
192.168.15.1:3260,13 --login
Logging in to [iface: eth2, target: iqn.2001-05.com.equallogic:
0-8a0906-0dd652407-00e000e7b254c345-testdeb, portal:
192.168.15.1,3260]
Logging in to [iface: eth2, target: iqn.2001-05.com.equallogic:
0-8a0906-0dd652407-00e000e7b254c345-testdeb, portal:
192.168.15.1,3260]
iscsiadm: Could not login to [iface: eth2, target: iqn.
2001-05.com.equallogic:0-8a0906-0dd652407-00e000e7b254c345-testdeb,
portal: 192.168.15.1,3260]:
iscsiadm: initiator reported error (8 - connection timed out)

I don't have any log line

Is anyone an idea of what could happen ?

Thank you very much, i am stuck

[Shyam Iyer]

Can you try changing the following /etc/sysctl.conf parameter

net.ipv4.conf.default.rp_filter = 0

Run "sysctl -p" after changing this to have the change reflected in the
kernel.

Mike Christie elucidated in an earlier post that the strict source
routing requirement that some distributions impose can cause the login
through multiple ifaces to fail.

-Shyam

[Shyam Iyer]

To be more clear you would need to change the rp_filter value to 0

[Don Williams]

One quick test. Can ping the group ip with eth2. Recent versions of debian/ubuntu fail that test. So I could nvr get that path 2 work. If u down eth1 eth2 usually starts 2 work. So it's likely something with standard routing.

Also Equallogic has a white paper on setting up OIS with debian. Might want to use current tar ball vs dkpg version. Might b a little stale.

-don
Sent from my iPhone

> --
> You received this message because you are subscribed to the Google Groups "open-iscsi" group.
> To post to this group, send email to open-...@googlegroups.com.
> To unsubscribe from this group, send email to open-iscsi+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
>

[Laurent HENRY]

Hi Don, you are right.
i can't ping from eth2 interface:

#ping -I eth1 192.168.15.1
PING 192.168.15.1 (192.168.15.1) from 192.168.15.11 eth1: 56(84) bytes of
data.
64 bytes from 192.168.15.1: icmp_seq=1 ttl=255 time=0.192 ms
64 bytes from 192.168.15.1: icmp_seq=2 ttl=255 time=0.118 ms
64 bytes from 192.168.15.1: icmp_seq=3 ttl=255 time=0.114 ms
c^C
--- 192.168.15.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.114/0.141/0.192/0.037 ms
# ping -I eth2 192.168.15.1
PING 192.168.15.1 (192.168.15.1) from 192.168.15.12 eth2: 56(84) bytes of
data.
From 192.168.15.12 icmp_seq=1 Destination Host Unreachable
From 192.168.15.12 icmp_seq=2 Destination Host Unreachable
From 192.168.15.12 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.15.1 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3018ms

What is the problem with Debian ? Something wrong in the routing table ?

Do you know where this paper can be found ?

--
Laurent HENRY
Administrateur Systèmes & Réseaux
Responsable du CRI/RSSI
EHESS - CRI
54 Bd Raspail
75006 Paris
Secrétariat du CRI: 01 49 54 23 08
Tel: 01 49 54 23 61

[Laurent HENRY]

thank you very much for your help Shyam. Unfortunately, this parameter doesn't
help my eth2 to ping

# sysctl -p
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0

# ping -I eth2 192.168.15.1
PING 192.168.15.1 (192.168.15.1) from 192.168.15.12 eth2: 56(84) bytes of
data.

From 192.168.15.12 icmp_seq=2 Destination Host Unreachable
From 192.168.15.12 icmp_seq=3 Destination Host Unreachable

From 192.168.15.12 icmp_seq=4 Destination Host Unreachable

[Don Williams]

You hv to ask for it. It's not posted on website. It won't fix this issue. It just shows u how 2 set it up including mpio and persistent device names.

-don
Sent from my iPhone

[Mike Vallaly]

Sorry for the lateness in my reply. Just stumbled across this
thread.. ;)

Part of the problem with MPIO in linux with two (or more) interfaces
connected to the same Ethernet segment is "arp flux". Essentially all
traffic will by default only exit out one path (mac address) on a
multi-homed network. The fix for this is to explicitly tie the
interface to a route rule which ensures traffic leaves via the
interface the application intended.

Here is the script we use in Debian to properly set the interface
routes for MPIO with equallogic. (IE: /etc/scripts/san-interface)

<snip>
#!/bin/bash

# Michael Vallaly (Feb 2008)

# This script configures local interfaces for use CNU's MP-IO iSCSI
SAN Network
# NOTE: This script may be called from /etc/network/interfaces without
parameters

SAN_NETWORKS="10.99.99.0/24"
SAN_MTU="9000"

IP_BIN="/bin/ip"
ETHTOOL_BIN="/usr/sbin/ethtool"

########################################################################################

# Check for required binaries
for req_bin in $IP_BIN $ETHTOOL_BIN; do
if [ ! -x "$req_bin" ]; then
echo "Can't execute ${req_bin}! Aborting.."
exit 1
fi
done

usage="Usage: $0 -i <interface> -m <add/del>"

while getopts "i:m:" options; do
case $options in
i ) interfaces+=" $OPTARG";;
m ) action=$OPTARG;;
\? ) echo $usage
exit 1;;
* ) echo $usage
exit 1;;
esac
done

# Check for ifup/down enviornment variables
if [[ -n $MODE && -n $IFACE ]]; then
interfaces=$IFACE
action=$MODE
fi

# Figure out what we are doing
case $action in
start ) action="add";;
add ) action="add";;
stop ) action="del";;
del ) action="del";;
* ) echo $usage
exit 1;;
esac

for interface in $interfaces; do

# Check that the interface exists before we go playing with it
if ! ($IP_BIN addr |egrep -nqe "inet.*$interface" && $IP_BIN link |
egrep -nqe "$interface.*,UP"); then
continue
fi
table_num=$((`echo ${interface} |tr -d [[:alpha:]]` + 10))
interface_ip=`$IP_BIN route show scope link proto kernel dev $
{interface} |awk '{print $3}'`

if [ $table_num -gt 252 ]; then
echo "Invalid SAN interface (${table_num}) specified!"
exit 1
fi

for network in $SAN_NETWORKS; do

# Configure our remote SAN networks

localnet=`$IP_BIN route |grep "${interface} proto kernel" |cut -
d" " -f1`
existing_san_iface=`$IP_BIN route show ${network} |grep -we "via" |
awk '{print $5}'`

# Don't add networks if they are locally connected
if [[ "$localnet" == "$network" ]]; then
continue
else

# Set our default gateway for remote networks
local=`echo $localnet |cut -d. -f1-3`

if [[ "$action" == "add" ]]; then

# Create a unique route table
$IP_BIN route add ${localnet} dev ${interface} table $
{table_num}
$IP_BIN route add ${network} via ${local}.1 dev ${interface}
table ${table_num}
$IP_BIN rule add from ${interface_ip}/32 lookup ${table_num}
route_match=`echo $existing_san_iface $interface |tr -t ' '
'\n'|sort -u`

else

# Delete the route table
$IP_BIN rule del from ${interface_ip}/32 lookup ${table_num}
2> /dev/null
route_match=`echo $existing_san_iface $interface |tr -t ' '
'\n'|sort -u |grep -v ${interface}`

fi

# Generate required next hops
route_opt=""
for dev in $route_match; do
route_opt="$route_opt nexthop via ${local}.1 dev ${dev}"
done

# Cleanup default route
$IP_BIN route del ${network} via ${local}.1 2> /dev/null

# Add/ReAdd the default route
if [ "${route_opt}" != "" ]; then
eval $IP_BIN route add ${network} scope global ${route_opt}
fi

fi

done

# Flush the routing cache
$IP_BIN route flush cache

# Configure our local network interfaces
# Configure our local network interfaces

if [[ "$action" == "add" ]]; then

# Set the proper MTU for the network interface (note this may take
the interface offline!)
if [ "$($IP_BIN link show $interface |grep "mtu" |cut -d" " -f
5)" != "9000" ]; then
$IP_BIN link set $interface mtu $SAN_MTU
fi

# Force flowcontrol on
$ETHTOOL_BIN --pause $interface autoneg off rx on tx on

# Only ARP for local interface
echo "1" > /proc/sys/net/ipv4/conf/${interface}/arp_ignore

else

# Set the proper MTU for the network interface (note this may take
the interface offline!)
if [ "$($IP_BIN link show $interface |grep "mtu" |cut -d" " -f 5)"
== "9000" ]; then
$IP_BIN link set $interface mtu 1500
fi

# Force flowcontrol autoneg
$ETHTOOL_BIN --pause $interface autoneg on

# ARP for any interface
echo "0" > /proc/sys/net/ipv4/conf/${interface}/arp_ignore

fi

done

</snip>

You can then add something similar to this in your /etc/network/
interface file:

<snip>
# The First SAN interface
iface eth1 inet static
address 10.99.99.100
netmask 255.255.255.0
post-up /etc/scripts/san-interface
pre-down /etc/scripts/san-interface

# The Second SAN interface
iface eth2 inet static
address 10.99.99.101
netmask 255.255.255.0
post-up /etc/scripts/san-interface
pre-down /etc/scripts/san-interface

</snip>

Cheers..

-Mike

[Ulrich Windl]

>>> Mike Vallaly <mval...@cashnetusa.com> schrieb am 04.08.2010 um 19:54 in
Nachricht
<a5c39a25-6de0-4f00...@f33g2000yqe.googlegroups.com>:

> Sorry for the lateness in my reply. Just stumbled across this
> thread.. ;)
>
> Part of the problem with MPIO in linux with two (or more) interfaces
> connected to the same Ethernet segment is "arp flux". Essentially all
> traffic will by default only exit out one path (mac address) on a
> multi-homed network. The fix for this is to explicitly tie the
> interface to a route rule which ensures traffic leaves via the
> interface the application intended.
>

Mike,

looking at the script I tried to figure out what the script actually does. Can you describe in more detail what the problem is, and how the script fixes it?

Regards,
Ulrich
[...]

[Pasi Kärkkäinen]

On Wed, Aug 04, 2010 at 10:54:48AM -0700, Mike Vallaly wrote:
> Sorry for the lateness in my reply. Just stumbled across this
> thread.. ;)
>
> Part of the problem with MPIO in linux with two (or more) interfaces
> connected to the same Ethernet segment is "arp flux". Essentially all
> traffic will by default only exit out one path (mac address) on a
> multi-homed network. The fix for this is to explicitly tie the
> interface to a route rule which ensures traffic leaves via the
> interface the application intended.
>
> Here is the script we use in Debian to properly set the interface
> routes for MPIO with equallogic. (IE: /etc/scripts/san-interface)
>

Uhm, why not just use open-iscsi 'ifaces' feature to bind each
iscsi iface/session to specific ethernet interface?

-- Pasi

[Mike Vallaly]

I should have better explained why this script is useful. Tardiness
should be expected with me ;)

We have a setup something like this:

LunX <-> Openiscsi <-> eth2 (10.99.99.101) <-> IscsiTarget
(10.99.99.10)
and
LunX <-> Openiscsi <-> eth3 (10.99.99.102) <-> IscsiTarget
(10.99.99.10)

Both (eth2) 10.99.99.101 and (eth3) 10.99.99.102 are on a single Linux
Host and we have a single IscsiTarget on the same network
(10.99.99.0/24) which we create two iscsi TCP sessions to
(10.99.99.10).

What happens is you get a linux routing table which looks something
like this:

<snip>
mvallaly@darkstar:~$ ip ro sh
10.99.99.0/24 dev eth2 proto kernel scope link src 10.99.99.101
10.99.99.0/24 dev eth3 proto kernel scope link src 10.99.99.102
default via 10.99.99.1 dev eth2 proto static
</snip>

Connectivity _WILL_ work, but not likely as one would expect. I will
attempt to enumerate the unexpected (land-mine) behavior below:

#1. Arp Flux - In Linux the MAC address by default is tied to the HOST
and not the NIC (counter intuitive I know), meaning _ANY_ active
network interface on a host _MAY_ reply to an ARP request for _ANY_
MAC address the host has NICs for. What this means is that you have a
race condition when two NICs are placed on the same network, as either
of them CAN and WILL respond to ARP requests for the other, resulting
in (my personal experience) unbalanced inbound traffic across two NICs
on the same network.

#2. Routing Table - In the above routing table snippet, you will
notice we have two network routes for 10.99.99.0/24 (one attached to
eth2 and one attached to eth3) the HOST can use when contacting the
IscsiTarget (10.99.99.10). The land-mine here is that even with two
separate processes (separate open-iscsi sessions), both sessions will
use the most specific match in the routing table (IE: "10.99.99.0/24
dev eth2 proto kernel scope link src 10.99.99.101") and traffic
will exit only via eth2. To make matters even worse we have a standard
default gateway of 10.99.99.1, which works until the eth2 network link
is severed, at which point we lose all ability to get packets off the
10.99.99.0/24 network despite sill having a working eth3.

So the script I attached earlier fixes these issues,

First it changes the ARP function on the SAN interfaces. Specifically
'echo "1" > /proc/sys/net/ipv4/conf/${interface}/arp_ignore' on the
SAN interfaces to prevent the Linux kernel from responding to ARPs on
interfaces other than via the NIC containing the MAC used in the ARP
response.

Second it configures iproute2 route rules, used in conjunction to the
standard linux routing table, which forces traffic leaving from the
HOST, to exit using the interface the IP packets are sourced sourced
from.

Third it configures a multi-hop default route for the SAN networks,
which provides dead gateway detection and fail-over should one of the
interfaces on the 10.99.99.0/24 network fail.

Hopefully this helps.. (or explains my maddness)

-Mike




On Aug 5, 1:32 am, "Ulrich Windl" <Ulrich.Wi...@rz.uni-regensburg.de>
wrote:
> >>> Mike Vallaly <mvall...@cashnetusa.com> schrieb am 04.08.2010 um 19:54 in
>
> Nachricht
> <a5c39a25-6de0-4f00-8c0b-79186c6f8...@f33g2000yqe.googlegroups.com>: