Tuning iSCSI between Linux and NetAPP

[Frank Bonnet]

Hello

I'm setting up a Samba server that will use iSCSI to access
some shares on a NetAPP filer ( FAS 2050 )

I would like to know if some of you has already build such
configuration and if there are some tricks to optimize it.

The Linux server is a HP Proliant quad CPU and runs
Debian Lenny, it has 16 Gb of RAM.

Thanks a lot.

[benoit plessis]

First i would ask why the hell ?

The netapp filer is a very good CIFS/SMB share server. Using it as an iSCSI target -- which is not is primary
function (netapp filer are more NAS than SAN) -- will only create limitations (unable to resize volume on the fly,
unable to use wafl attributes to store windows security acl, ...) with no visible gain ...

 Also your server seem very overkill to me, i must hope it won't have to be just a samba<=>iscsi interface ...

For iSCSI and netapp in general, first make sure that you have at least
10% of free space inside the volume, and 10% of free space inside the aggregate or else perf could
suffer and more important you won't be able to launch the reallocate process ("defrag").

The following is the recommended netapp/iscsi optimisations, however open-iscsi doesn't support multiple
 connections per session now (iirc), so the best way to have parallel access is to use multipath

iscsi.iswt.max_ios_per_session 64
iscsi.max_connections_per_session 16
iscsi.max_ios_per_session    64

2009/4/14 Frank Bonnet <f.bo...@esiee.fr>

[Frank Bonnet]

benoit plessis wrote:
> First i would ask why the hell ?
>
> The netapp filer is a very good CIFS/SMB share server. Using it as an
> iSCSI target -- which is not is primary
> function (netapp filer are more NAS than SAN) -- will only create
> limitations (unable to resize volume on the fly,
> unable to use wafl attributes to store windows security acl, ...) with
> no visible gain ...

Netapp filers works supports well SMB as long as you have a M$ domain
PDC server or an AD server, I don't

If you use Samba as PDC, SMB won't work natively ( if you have a
solution that works I'll be really happy to get it !! )

>
> Also your server seem very overkill to me, i must hope it won't have to
> be just a samba<=>iscsi interface ...

no it's a complete server, not only disks access, with 800 M$ clients
connected to.

>
> For iSCSI and netapp in general, first make sure that you have at least
> 10% of free space inside the volume, and 10% of free space inside the
> aggregate or else perf could
> suffer and more important you won't be able to launch the reallocate
> process ("defrag").
>
> The following is the recommended netapp/iscsi optimisations, however
> open-iscsi doesn't support multiple
> connections per session now (iirc), so the best way to have parallel
> access is to use multipath
>
> iscsi.iswt.max_ios_per_session 64
> iscsi.max_connections_per_session 16
> iscsi.max_ios_per_session 64
>

thank you for your help

[jnantel]

I use a netapp filer...where are these values set? Host or Array?

iscsi.iswt.max_ios_per_session 64
iscsi.max_connections_per_session 16
iscsi.max_ios_per_session 64

On Apr 14, 6:40 am, benoit plessis <plessis.ben...@gmail.com> wrote:
> First i would ask why the hell ?
>
> The netapp filer is a very good CIFS/SMB share server. Using it as an iSCSI
> target -- which is not is primary
> function (netapp filer are more NAS than SAN) -- will only create
> limitations (unable to resize volume on the fly,
> unable to use wafl attributes to store windows security acl, ...) with no
> visible gain ...
>
>  Also your server seem very overkill to me, i must hope it won't have to be
> just a samba<=>iscsi interface ...
>
> For iSCSI and netapp in general, first make sure that you have at least
> 10% of free space inside the volume, and 10% of free space inside the
> aggregate or else perf could
> suffer and more important you won't be able to launch the reallocate process
> ("defrag").
>
> The following is the recommended netapp/iscsi optimisations, however
> open-iscsi doesn't support multiple
>  connections per session now (iirc), so the best way to have parallel access
> is to use multipath
>
> iscsi.iswt.max_ios_per_session 64
> iscsi.max_connections_per_session 16
> iscsi.max_ios_per_session    64
>

> 2009/4/14 Frank Bonnet <f.bon...@esiee.fr>

[Dmitry Yusupov]

Hi Frank,

May I also suggest an alternative for iSCSI target? We had a very good
experience with open-iscsi and NexentaStor [1]. At the beginning we had
problems with single-threaded traffic latency but the ultimate fix was
on Linux side to ensure that io scheduler is set to "noop":

# echo noop > /sys/block/sd*/queue/scheduler

Once set, we saturated physical networking connection with small block
sizes in single-threaded scenario.

NexentaStor also is very good candidate for CIFS workgroups/AD
environments with the whole SMB stack implemented in the kernel, which
boosts performance over the top. And as far as iSCSI target - I would
recommend to use COMSTAR, which is ZFS integrated in Nexenta [2].

[1] http://www.nexenta.com/products
[2]
http://blogs.nexenta.org/blog/2009/03/03/nexenta-iscsi-with-comstarzfs-integration/

[Bart Van Assche]

On Tue, Apr 14, 2009 at 7:09 PM, Dmitry Yusupov <dmitr...@yahoo.com> wrote:
> NexentaStor also is very good candidate for CIFS workgroups/AD
> environments with the whole SMB stack implemented in the kernel, which
> boosts performance over the top. And as far as iSCSI target - I would
> recommend to use COMSTAR, which is ZFS integrated in Nexenta [2].

Do you have any performance numbers available for the combination of
open-iscsi and the NexentaStor target ?

Bart.

[Dmitry Yusupov]

This highly depends on HW used on both sides, and NexentaStor is HW
agnostic in a sense that it can be installed on any commodity x86
hardware. But out of the 4-core intel we've seen ~ 50k IOPS both
reads/writes.

[Frank Bonnet]

Dmitry Yusupov wrote:
> Hi Frank,

Hi Dimitri

>
> May I also suggest an alternative for iSCSI target? We had a very good
> experience with open-iscsi and NexentaStor [1]. At the beginning we had
> problems with single-threaded traffic latency but the ultimate fix was
> on Linux side to ensure that io scheduler is set to "noop":
>
> # echo noop > /sys/block/sd*/queue/scheduler
>
> Once set, we saturated physical networking connection with small block
> sizes in single-threaded scenario.

OK I'm gonna try this

>
> NexentaStor also is very good candidate for CIFS workgroups/AD
> environments with the whole SMB stack implemented in the kernel, which
> boosts performance over the top. And as far as iSCSI target - I would
> recommend to use COMSTAR, which is ZFS integrated in Nexenta [2].
>
> [1] http://www.nexenta.com/products
> [2]
> http://blogs.nexenta.org/blog/2009/03/03/nexenta-iscsi-with-comstarzfs-integration/
>

It might good products but I still have Netapp filers :-)

[Frank Bonnet]

By default the filer has these iscsi (Ontap 7.3.1) values set

iscsi.enable on
iscsi.isns.rev 22
iscsi.max_connections_per_session use_system_default
iscsi.max_error_recovery_level use_system_default