iSCSI and SRP with single target?

[Dan Swartzendruber]

(I'm pretty sure this isn't possible, but I may as well ask...)  So I am serving up storage via SRP, since the performance is far better than IPOIB.  I'd like to be able to have an iSCSI target serving up the same underlying storage.  I tried this and it didn't work due to lack of permissions (IIRC).  I assumed there is no good way to have the two target SW access the same block device without causing issues?

[Marc Smith]

Hi Dan,

I think I know what you're asking, but want to clarify:

- Lets say your initiator is a VMware ESXi host, and you have an iSCSI initiator (offloaded card, software initiator, whatever) AND you have a SRP initiator configured.

- You then have a SCST device pointing to the same back-end storage, and mapped to a LUN in a security group on an iSCSI target AND a SRP target.

- You want your VMFS datastore to have two different paths: One via iSCSI, and one via SRP.

Is that right? If so, I've had the same thought before, but never actually tried it. The idea was to have two ESOS disk arrays, one local, and one at a DR site or something similar. You could then replicate the storage between both with DRBD, and use Fibre Channel for the local disk array and hosts, and then iSCSI for a path to the same storage on the remote box.

Never tried, but I always wondered the same. I think the issue would come to vSphere not allowing you to have different paths to the same LUN on different mediums (iSCSI vs. FC/SRP/etc.). Their path selector policy may not be aware of that. But I bet if you had a Linux initiator host, and you used multipath-tools on the initiator, it would work. Just a guess.

--Marc

On Wed, Apr 8, 2015 at 6:47 PM, Dan Swartzendruber <dsw...@druber.com> wrote:

(I'm pretty sure this isn't possible, but I may as well ask...)  So I am serving up storage via SRP, since the performance is far better than IPOIB.  I'd like to be able to have an iSCSI target serving up the same underlying storage.  I tried this and it didn't work due to lack of permissions (IIRC).  I assumed there is no good way to have the two target SW access the same block device without causing issues?

--
You received this message because you are subscribed to the Google Groups "esos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Swartzendruber]

On 2015-04-09 09:14, 'Marc Smith' via esos-users wrote:
> Hi Dan,
>
> I think I know what you're asking, but want to clarify:
> - Lets say your initiator is a VMware ESXi host, and you have an iSCSI
> initiator (offloaded card, software initiator, whatever) AND you have
> a SRP initiator configured.
> - You then have a SCST device pointing to the same back-end storage,
> and mapped to a LUN in a security group on an iSCSI target AND a SRP
> target.
> - You want your VMFS datastore to have two different paths: One via
> iSCSI, and one via SRP.
>
> Is that right? If so, I've had the same thought before, but never
> actually tried it. The idea was to have two ESOS disk arrays, one
> local, and one at a DR site or something similar. You could then
> replicate the storage between both with DRBD, and use Fibre Channel
> for the local disk array and hosts, and then iSCSI for a path to the
> same storage on the remote box.
>
> Never tried, but I always wondered the same. I think the issue would
> come to vSphere not allowing you to have different paths to the same
> LUN on different mediums (iSCSI vs. FC/SRP/etc.). Their path selector
> policy may not be aware of that. But I bet if you had a Linux
> initiator host, and you used multipath-tools on the initiator, it
> would work. Just a guess.

Correct. Although I didn't want replication with drbd - just two
different paths to the storage via SRP and iSCSI. When I tried it, esos
wouldn't let the second target use the same underlying storage. My main
concern with a drbd approach would be loss of write performance.

[Marc Smith]

I just tried this and was able to do the configuration on the ESOS side. You don't want to make another SCST device, just map the SCST device as a LUN to another group for another target.

So create the SCST device, then (example):

- Map as LUN 0 to security group for SRP target.

- Map as LUN 0 to security group for iSCSI target.

I tested with an iSCSI target and FC target.

--Marc

--
You received this message because you are subscribed to the Google Groups "esos-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+unsubscribe@googlegroups.com.

[Dan Swartzendruber]

On 2015-04-09 10:35, 'Marc Smith' via esos-users wrote:
> I just tried this and was able to do the configuration on the ESOS
> side. You don't want to make another SCST device, just map the SCST
> device as a LUN to another group for another target.
>
> So create the SCST device, then (example):
> - Map as LUN 0 to security group for SRP target.
> - Map as LUN 0 to security group for iSCSI target.
>
> I tested with an iSCSI target and FC target.

Interesting. I wonder what I did wrong lol. I'll give that a try,
thanks! (and provide feedback as to how vsphere handles it...)

[Dan Swartzendruber]

By golly, you're right. I haven't moved my production storage off the
omniOS zfs pool yet, but I figured your premise would work for comstar
(zfs iscsi/srp SW) too. vsphere reports TWO paths to the storage, one
via srp and one via iSCSI. I can even set the srp to be the preferred
path. Now to see if 'direct san access' in veeam does the right thing.
If this all works, I need to migrate the VMs off this pool and switch to
esos.

[Dan Swartzendruber]

Minor bummer. Veeam apparently chokes on this configuration, since the
datastore shows two paths to the target, one of which it can't see due
to not being connected to the infiniband fabric. Oh well, that was only
a tweak to improve backup speed. The main point: vsphere seems
perfectly happy to have two paths to the same datastore, one on SRP and
one on iSCSI.

[Marc Smith]

Very interesting. Sounds like an HA disk array solution might be possible by putting one disk array local, and one at a remote site.

On Thu, Apr 9, 2015 at 11:31 AM, Dan Swartzendruber <dsw...@druber.com> wrote:

Minor bummer.  Veeam apparently chokes on this configuration, since the datastore shows two paths to the target, one of which it can't see due to not being connected to the infiniband fabric.  Oh well, that was only a tweak to improve backup speed.  The main point: vsphere seems perfectly happy to have two paths to the same datastore, one on SRP and one on iSCSI.

[Dan Swartzendruber]

On 2015-04-09 11:34, 'Marc Smith' via esos-users wrote:
> Very interesting. Sounds like an HA disk array solution might be
> possible by putting one disk array local, and one at a remote site.

Hmmmm...

[Steve Jones]

That's strange..  It would seem that if VEEAM chokes on that because of an unavailable path, that it would also fail if one of any other set of redundant paths are down, in which case, is VEEAM not capable of working if there's any redundancy that's down?  That would be bad..  OR, is there something special about SRP?  I don't know anything about SRP, but I've done the same LUN available over FC and iSCSI before..  My only problem was that I wasn't really able to prioritize the FC over the iSCSI, so I ended up shutting down the iSCSI target to force the servers to use FC..  The target is still there, if I ever need to turn it on quickly.

On Thu, Apr 9, 2015 at 11:31 AM, Dan Swartzendruber <dsw...@druber.com> wrote:

Minor bummer.  Veeam apparently chokes on this configuration, since the datastore shows two paths to the target, one of which it can't see due to not being connected to the infiniband fabric.  Oh well, that was only a tweak to improve backup speed.  The main point: vsphere seems perfectly happy to have two paths to the same datastore, one on SRP and one on iSCSI.

[Dan Swartzendruber]

On 2015-04-09 11:38, Steve Jones wrote:
> That's strange.. It would seem that if VEEAM chokes on that because
> of an unavailable path, that it would also fail if one of any other
> set of redundant paths are down, in which case, is VEEAM not capable
> of working if there's any redundancy that's down? That would be bad..
> OR, is there something special about SRP? I don't know anything
> about SRP, but I've done the same LUN available over FC and iSCSI
> before.. My only problem was that I wasn't really able to prioritize
> the FC over the iSCSI, so I ended up shutting down the iSCSI target to
> force the servers to use FC.. The target is still there, if I ever
> need to turn it on quickly.

Sorry, I was unclear. For direct san access, veeam needs direct access
to the san fabric. The windows proxy then creates an iSCSI connection
to that LUN (readonly) and fetches blocks directly. What I *think* is
happening is veeam asks vsphere for the target information, and spits up
when it sees the srp info - only guessing here, as I don't have access
to their source code. I'm going to look at the veeam logs and see if
that sheds any light...

[Dan Swartzendruber]

you were not able to set the FC target as preferred in vsphere? Why
not?

[Dan Swartzendruber]

I thought I remembered seeing something on this. Short answer: bad idea. Google for drbd dual primary multipath for a post from a drbd developer explaining why...

[Marc Smith]

You need to clarify that statement: "multipath" is a bad idea, in short round-robin is bad. If you control the pathing its fine. We do it successfully and lots of other SCST users do too: http://marcitland.blogspot.com/2013/04/building-using-highly-available-esos.html

On Thu, Apr 9, 2015 at 12:32 PM, Dan Swartzendruber <dsw...@druber.com> wrote:

I thought I remembered seeing something on this.  Short answer: bad idea.  Google for drbd dual primary multipath for a post from a drbd developer explaining why...

--
You received this message because you are subscribed to the Google Groups "esos-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.

[Dan Swartzendruber]

Yeah I read that blog post. Good stuff. Yeah I was referring to basic dual primary on top of drbd. I note you referenced the same post by Lars :)