RE: What happens if an iSCSI target resets.

[netz-haut - stephan seitz]

We're running a 2 node DRBD active/passive target. During pre-production tests I've found, that kicking the primary immediately off by heartbeat/sys_req or STONITH works much better than shutdown.
I've configured dm-multipath on top of bonding devices (mode 0) utilizing two switches. The bonding eats some performance, but target availability is 100%.

The heartbeat is configured to
1. move the target ip's to the other node
2. sleep 1 second
3. move a drbd device with only configurations for iscsitarget, etc..
4. another 1 second delay
5. move (say: primary stop and secondary start) lvm VG (iscsitarget uses this VG on top of a drbd)
6. move the big data drbd
7. move iscsitarget

> -----Original Message-----
> From: open-...@googlegroups.com [mailto:open-...@googlegroups.com]
> On Behalf Of Alvin Starr
> Sent: Friday, March 19, 2010 3:13 PM
> To: open-...@googlegroups.com
> Subject: What happens if an iSCSI target resets.
>
>
> I am thinking of a 2 node cluster that sync the disks via DRBD.
>
> I have seen some articles about using multipath and 2 independent
> targets but have also seen generic problem descriptions of using
> multipath in the face of failures.
>
> I was thinking of using a floating IP address for the target and using
> the clustering/HA glue to make it move between servers.
>
> This would be the equivalent of resetting the target software.
>
> Does anyone have a handle on how the initiators will deal with this.
> Ideally in theory and practice.
> --
>
> Alvin Starr || voice: (416)585-9971x690
> Interlink Connectivity || fax: (416)585-9974
> al...@iplink.net ||
>
> --
> You received this message because you are subscribed to the Google
> Groups "open-iscsi" group.
> To post to this group, send email to open-...@googlegroups.com.
> To unsubscribe from this group, send email to open-
> iscsi+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/open-iscsi?hl=en.
>

[Mike Christie]

On 03/25/2010 03:50 PM, netz-haut - stephan seitz wrote:
>> -----Original Message-----
>> From: open-...@googlegroups.com [mailto:open-...@googlegroups.com]
>> On Behalf Of Alvin Starr
>> Sent: Friday, March 19, 2010 3:13 PM
>> To: open-...@googlegroups.com
>> Subject: What happens if an iSCSI target resets.
>>
>>
>> I am thinking of a 2 node cluster that sync the disks via DRBD.
>>
>> I have seen some articles about using multipath and 2 independent
>> targets but have also seen generic problem descriptions of using
>> multipath in the face of failures.
>>
>> I was thinking of using a floating IP address for the target and using
>> the clustering/HA glue to make it move between servers.

When you say floating IP's does that mean they will not be static? That
might not work very well with open-iscsi. If we lose the connection or
run the error recovery and have to relogin we will try to use the IP
that we started with.

Some targets like from Equallogic will have a static IP that the
initiator logs into, then the target will tell us about a different IP
to use. We then login there and do IO. If we get disconnected though, we
go back the first IP and it redirects us again.

[Dheeraj Sangamkar]

You may want to check out 'target moved temporarily' in the following document:

http://ietfreport.isoc.org/all-ids/draft-gilligan-iscsi-fault-tolerance-00.txt

Dheeraj

> --
> You received this message because you are subscribed to the Google Groups
> "open-iscsi" group.
> To post to this group, send email to open-...@googlegroups.com.
> To unsubscribe from this group, send email to

> open-iscsi+...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/open-iscsi?hl=en.
>
>

--
Simplicity of character is the natural result of profound thought.

[Mike Christie]

On 03/26/2010 01:42 PM, Dheeraj Sangamkar wrote:
> You may want to check out 'target moved temporarily' in the following document:
>
> http://ietfreport.isoc.org/all-ids/draft-gilligan-iscsi-fault-tolerance-00.txt
>

Neat. Thanks.

In the open-iscsi.git code (working on making a release here
http://www.kernel.org/pub/linux/kernel/people/mnc/open-iscsi/releases/open-iscsi-2.0-872-rc1.tar.gz)

we sort of do this. Instead of running the iscsiadm discovery command
"iscsiadm -m discovery ...." you can set

discovery.daemon.sendtargets.addresses = $IP

(for isns replace sendtargets with isns)
When iscsid starts up it will do discovery, and try to login to the
portals found. It will then do discovery every poll_interval seconds (or
for isns it will react to isns SCNs), and log in/out of portals that are
found or no longer found.

To implement what is in that spec we could just add some code to fire an
event to iscsid to force it to do discovery right away instead of
poll_interval secs when it cannot log into the portal.

[Alvin Starr]

I am thinking more that if server A has the ip address 1.2.3.4 and
server B has 1.2.3.5
if server A breaks server B takes over 1.2.3.4 along with 1.2.3.5.

Another scenario would be that either server A or B has 1.2.3.4 and if
running server dies the other server takes over the address.

[Mike Christie]

This would be easier and doable now. On the initiator box you will want
to use dm-multipath. dm-multipath has a concept called priority groups.
You could put the target at 1.2.3.4 in one group, then put the other
target at 1.2.3.5 in a different group. dm-multipath would use paths
found in group1 and when that was inaccessible it would switch to group2.

This is a common setup and dm-multipath should work well. However, I
think setting it up might require some pain because for your setup there
probably is no priority group callouts that are going to do what you
wanted automagically. It would probably be best to post to the dm-devel
list to get info on that though. I am not 100% sure.

> Another scenario would be that either server A or B has 1.2.3.4 and if
> running server dies the other server takes over the address.
>

This would be fine, and easier to setup than the first one. For this you
do not need dm-multipath. It might be better though. For the most basic
setup you would just use the iscsi/scsi disk /dev/sdX. Then for the
iscsi settings you could use the normal iscsi noop settings, but for
node.session.timeo.replacement_timeout you want a longer value.

When the iscsi layer detects a problem it internally queues IO, for up
to replacement_timeout seconds. Once that times out the iscsi layer will
fail IO. So if you are not using dm-multipath you want this value to be
long enough to switch from serverA to serverB.

If you are using dm-multipath, you could just use one priority group,
and then you would just dm-multipath for its
no_path_retry/queue_if_no_path settings. For these you would set them
higher to allow a switch from serverA to serverB. The difference with
those settings and replacement_timeout is that the dm-multipath
queue_if_no_path setting will allow you to queue forever.