Building a multipath setup with DRBD in dual-Primary

[Wido]

Hello,

Last night i had the idea of building a multipath iSCSI setup with
DRBD in Primary/Primary.

My idea was/is:

I have two machines with a 30GB partition wich is synced via DRBD (i
assume everybody here is familiar with DRBD).

My DRBD is running fine both machines are UpToDate and they are both
primary.

On top of /dev/drbd0 i made a LVM VG with the name "data", in this VG
i created 3 LV's with a size of 5GB.

For this e-mail i'll call the machines target1, target2 and iscsi-
initiator

target1: 192.168.5.199
target2: 192.168.5.200
iscsi-initiator: 192.168.5.198

On both the targets i have created the folling ietd.conf:

Target iqn.2007-04.nl.pcextreme:ha
Lun 0 Path=/dev/data/lun0,Type=fileio
Lun 1 Path=/dev/data/lun1,Type=fileio
Lun 2 Path=/dev/data/lun2,Type=fileio

So both targets have the same IQN and i tried to create a multipath
from the client to both tartgets:

iscsiadm -m discovery -t sendtargets -p 192.168.5.199
iscsiadm -m discovery -t sendtargets -p 192.168.5.200
iscsiadm -m node -T iqn.2007-04.nl.pcextreme:ha -p 192.168.5.199 --
login
iscsiadm -m node -T iqn.2007-04.nl.pcextreme:ha -p 192.168.5.200 --
login

This works and i get a total of 6 new SCSI devices, but multipath
won't recognize them as a redundant path:

root@iscsi-client:~# multipath -ll
149455400000000000000000001000000595800000f000000dm-3 IET ,VIRTUAL-
DISK
[size=5.0G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ 5:0:0:0 sde 8:64 [active][ready]
149455400000000000000000001000000dd3c00000e000000dm-0 IET ,VIRTUAL-
DISK
[size=5.0G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ 4:0:0:1 sdc 8:32 [active][ready]
149455400000000000000000001000000925800000f000000dm-5 IET ,VIRTUAL-
DISK
[size=5.0G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ 5:0:0:2 sdg 8:96 [active][ready]
149455400000000000000000001000000655800000f000000dm-4 IET ,VIRTUAL-
DISK
[size=5.0G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ 5:0:0:1 sdf 8:80 [active][ready]
149455400000000000000000001000000a03c00000e000000dm-1 IET ,VIRTUAL-
DISK
[size=5.0G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ 4:0:0:0 sdb 8:16 [active][ready]
149455400000000000000000001000000193d00000e000000dm-2 IET ,VIRTUAL-
DISK
[size=5.0G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ 4:0:0:2 sdd 8:48 [active][ready]
root@iscsi-client:~#

Is this a target issue because the "sessions" are not synchronized
between both targets? Or is this related to Open-iSCSI and multipath?

Is my setup a possibility or should i stay with the old regular setup
of a Primary/Standy with heartbeat?

Thank you in advance,

Wido den Hollander

[Mike Christie]

multipath does not use any iscsi info like targetname or portal to build
the multipath device. It just uses something like scsi vpd page info.

I think in your ietd.conf you will want to use the ScsiId or ScsiSN
setting in the Path line.

> Target iqn.2007-04.nl.pcextreme:ha
> Lun 0 Path=/dev/data/lun0,Type=fileio,scsi_id=some-uuid1
> Lun 1 Path=/dev/data/lun1,Type=fileio,scsi_id=some-uuid2
> Lun 2 Path=/dev/data/lun2,Type=fileio,scsi_id=some-uuid3

Then in the other target box have the scsi_id uuids match up.

[PCextreme B.V. - Wido den Hollander]

Hi Mike,

Yes indeed, that worked!

For the record, my ietd.conf

Target iqn.2007-04.nl.pcextreme:ha
Lun 0 Path=/dev/data/lun0,Type=fileio,ScsiId=nWFKhBCMq56JfSzq
Lun 1 Path=/dev/data/lun1,Type=fileio,ScsiId=wrOhZ1hVFANuQOJM
Lun 2 Path=/dev/data/lun2,Type=fileio,ScsiId=pW8e21PNf7WR2FDh

-
Met vriendelijke groet,

Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: sup...@pcextreme.nl
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl

> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "open-iscsi" group.
> To post to this group, send email to open-...@googlegroups.com
> To unsubscribe from this group, send email to open-iscsi+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/open-iscsi
> -~----------~----~----~----~------~----~------~--~---
>

[Bart Van Assche]

On Fri, May 29, 2009 at 5:21 PM, Wido <wid...@gmail.com> wrote:
> Last night i had the idea of building a multipath iSCSI setup with DRBD
> in Primary/Primary.

> [ ... ]

> Is my setup a possibility or should i stay with the old regular setup of
> a Primary/Standy with heartbeat?

I strongly recommend you to use a primary/standby setup + heartbeat
when using dm-multipath instead of a primary/primary setup. With the
latter setup you risk to run into data corruption. See also
http://lkml.org/lkml/2009/4/13/80.

Bart.

[PCextreme B.V. - Wido den Hollander]

Hi Bart,

At the moment this is only a proof of concept wich seems to be working
fine.

I had some thoughts about it:

* The iSCSI target should not run in Write-Back mode
* DRBD should run in synchronus mode (protocol C)
* I should use the failover policy with dm-multipath instead of
round-robin.

I got this setup working last night and i started bonnie++ on a lun wich
is still running with no errors yet, but ofcourse this should be tested
me thurough.

As far as my knowlegde goes, this setup should be working as long as i
don't start using clustered filesystems since these is only one
initiator writing to the LUN?

And again, i am only testing this setup, my production envirioment will
stick to the Primary/Standby situation with heartbeat.

-
Met vriendelijke groet,

Wido den Hollander
Hoofd Systeembeheer / CSO
Telefoon Support Nederland: 0900 9633 (45 cpm)
Telefoon Support België: 0900 70312 (45 cpm)
Telefoon Direct: (+31) (0)20 50 60 104
Fax: +31 (0)20 50 60 111
E-mail: sup...@pcextreme.nl
Website: http://www.pcextreme.nl
Kennisbank: http://support.pcextreme.nl/
Netwerkstatus: http://nmc.pcextreme.nl