Help to make iSER working

[Ben RUBSON]

Hello,

I have some difficulties to make iSER (FreeBSD 11 BETA3) working.



### My configuration :

2 servers with Mellanox ConnectX-3 NICs.



### What I did :

I rebuilt kernel adding the following to GENERIC :
options OFED
device mlxen
options COMPAT_LINUXKPI

I built iser :
cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
(I saw later-on that I could have used "device iser" in kernel configuration)

My /etc/ctl.conf :
portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.2.2
offload iser
}
target iqn.2012-06.com.test:target1 {
auth-group no-authentication
portal-group pg0
alias my-iser-disk
lun 0 {
path /dev/da8
}
}



### Error :

As soon as I do the following :
iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
I get this error :
iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument
iscsid[655]: child process 1187 terminated with exit status 1

"sysctl hw.mlxen1.stat.tx_packets" on initiator shows that no packet is sent at all when trying to connect.

Did I miss something ?

Many thanks !

Ben

_______________________________________________
freebs...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
To unsubscribe, send any mail to "freebsd-scsi...@freebsd.org"

[Ben RUBSON]

Hi Max,

Thank you very much for your answer.
Is there any plan to add iSER target to FreeBSD ?

Thank you again,

Ben



> On 10 Aug 2016, at 22:51, Max Gurtovoy <ma...@mellanox.com> wrote:
>
> Hi Ben,
>
> iSER initiator is supported in FreeBSD-11 but not the iser target.
> for the target you should use Linux target such as TGT (user space),
> Lio or SCST.
>
> thanks,
> Max.

[Max Gurtovoy]

Hi Ben,

iSER initiator is supported in FreeBSD-11 but not the iser target.
for the target you should use Linux target such as TGT (user space),
Lio or SCST.

thanks,
Max.


On 8/10/2016 8:13 PM, Ben RUBSON wrote:

[Max Gurtovoy]

Hi Ben,

Our initial goal was to push the initiator driver to FreeBSD and we
accomplished it.
You are welcome to report your benchmarks running iser and compare to
tcp numbers.
Regarding plans for the target side, it's not on my plate for the near
future (but obviously can be later on :)).
I'm not aware of other contributers plans.

Thanks,
Max.

[Ben RUBSON]

> On 10 Aug 2016, at 19:13, Ben RUBSON <ben.r...@gmail.com> wrote:
>
> I built iser :
> cd /usr/src/sys/modules/iser/ ; make ; make install ; kldload iser
> (I saw later-on that I could have used "device iser" in kernel configuration)

I tested the "device iser" in kernel configuration, compilation then fails with an "unknown iser device".
"device iser" can be found in "man iser", but does not seem to be understood by the kernel itself.

[Ben RUBSON]

> On 10 Aug 2016, at 19:13, Ben RUBSON <ben.r...@gmail.com> wrote:
>

> As soon as I do the following :
> iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
> I get this error :
> iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

I also get this error (invalid argument) whatever the IP address I give to the command, as if my initiator did not understand the iSER command "ISCSIDCONNECT".
How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Thank you,

[Ben RUBSON]

I just tested to connect to a Linux iSER enabled target, but I still get this error message at initiator side :

iscsid[1187]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Invalid argument

> as if my initiator did not understand the iSER command "ISCSIDCONNECT".
> How to be sure everything is correctly compiled with ICL_KERNEL_PROXY enabled ?

Ben


> On 10 Aug 2016, at 22:51, Max Gurtovoy <ma...@mellanox.com> wrote:
>

[Max Gurtovoy]

These are the instructions I get from iscsi maintainer that merged our
code to FreeBSD-11:
1. set WITH_OFED=yes in /etc/make.conf
2. "make buildkernel buildworld installkernel installworld"
3. reboot
4. kldload iser

After that, make sure you configured all the interfaces correctly and
that you have ping between the 2 servers.
later you need to check basic RDMA stuff on your servers (Added
Hans/Oded/Meny to help you if needed).

Only if the above are working you can proceed to check iser.

Max.

[Ben RUBSON]

Thank you very much Max for your help.



I managed to compile the kernel with iSER support doing the 3 modifications below.

I added the following to /usr/src/sys/conf/options :
ICL_KERNEL_PROXY opt_global.h

And I added "options ICL_KERNEL_PROXY" to my kernel configuration file.

I then faced this compilation error :
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: error: implicit declaration of function 'icl_conn_handoff_sock'
icl_conn_handoff_sock(cs->cs_conn, so);
/usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: note: did you mean 'icl_soft_handoff_sock'?
/usr/src/sys/dev/iscsi/icl.h:158:7: note: 'icl_soft_handoff_sock' declared here
int icl_soft_handoff_sock(struct icl_conn *ic, struct socket *so);

So I replaced icl_conn_handoff_sock by icl_soft_handoff_sock (sounds like it is the needed correction).

iscsictl -r now does not produce previous "invalid argument" error.

@trasz : perhaps these modifications should be done in FreeBSD 11 source tree ?



Now I'm facing the following issue :

# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
iscsid[8743]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: ERROR: iser_connect_error: conn 0xfffff8096786f000
iscsid[1078]: child process 8743 terminated with exit status 1

If I remove the "-r" option from the iscsictl command, it correctly connects to the target.

On target side, Lio, targetcli :
/> ls iscsi/
o- iscsi ............................................. [1 Target]
o- iqn.2012-06.com.test:target1 ....................... [1 TPG]
o- tpg1 ........................................... [enabled]
o- acls .......................................... [0 ACLs]
o- luns ........................................... [1 LUN]
| o- lun0 ..................... [iblock/block1 (/dev/sdc7)]
o- portals ..................................... [1 Portal]
o- 192.168.2.2:3260 .................. [OK, iser enabled]

I wanted to make some RDMA connection tests with udaddy for example, but I did not found the FreeBSD version.

Any advice ?

Many thanks !

Ben

[Max Gurtovoy]

Hi,

On 8/11/2016 6:33 PM, Ben RUBSON wrote:
> Thank you very much Max for your help.
>
>
>
> I managed to compile the kernel with iSER support doing the 3 modifications below.
>
> I added the following to /usr/src/sys/conf/options :
> ICL_KERNEL_PROXY opt_global.h
>
> And I added "options ICL_KERNEL_PROXY" to my kernel configuration file.
>
> I then faced this compilation error :
> /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: error: implicit declaration of function 'icl_conn_handoff_sock'
> icl_conn_handoff_sock(cs->cs_conn, so);
> /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: note: did you mean 'icl_soft_handoff_sock'?
> /usr/src/sys/dev/iscsi/icl.h:158:7: note: 'icl_soft_handoff_sock' declared here
> int icl_soft_handoff_sock(struct icl_conn *ic, struct socket *so);
>
> So I replaced icl_conn_handoff_sock by icl_soft_handoff_sock (sounds like it is the needed correction).
>
> iscsictl -r now does not produce previous "invalid argument" error.
>
> @trasz : perhaps these modifications should be done in FreeBSD 11 source tree ?
>
>
>
> Now I'm facing the following issue :
>
> # iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
> iscsid[8743]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
> kernel: ERROR: iser_connect_error: conn 0xfffff8096786f000
> iscsid[1078]: child process 8743 terminated with exit status 1

please increase the debug level in both initiator and target and sent it
out.
From first look I think you need to configure the LIO target to work
with demo mode.
Another thing you can try is the user space tgt target (compile with
make ISCSI_RDMA=1) that I work with mostly.

[Ben RUBSON]

> On 11 Aug 2016, at 18:10, Max Gurtovoy <ma...@mellanox.com> wrote:
>
> On 8/11/2016 6:33 PM, Ben RUBSON wrote:
>> Thank you very much Max for your help.
>>
>>
>>
>> I managed to compile the kernel with iSER support doing the 3 modifications below.
>>
>> I added the following to /usr/src/sys/conf/options :
>> ICL_KERNEL_PROXY opt_global.h
>>
>> And I added "options ICL_KERNEL_PROXY" to my kernel configuration file.
>>
>> I then faced this compilation error :
>> /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: error: implicit declaration of function 'icl_conn_handoff_sock'
>> icl_conn_handoff_sock(cs->cs_conn, so);
>> /usr/src/sys/modules/ctl/../../cam/ctl/ctl_frontend_iscsi.c:1358:2: note: did you mean 'icl_soft_handoff_sock'?
>> /usr/src/sys/dev/iscsi/icl.h:158:7: note: 'icl_soft_handoff_sock' declared here
>> int icl_soft_handoff_sock(struct icl_conn *ic, struct socket *so);
>>
>> So I replaced icl_conn_handoff_sock by icl_soft_handoff_sock (sounds like it is the needed correction).
>>
>> iscsictl -r now does not produce previous "invalid argument" error.
>>
>> @trasz : perhaps these modifications should be done in FreeBSD 11 source tree ?

I'm also rebuilding world with the following to see if it can help :
# cat /etc/src.conf
WITH_ICL_KERNEL_PROXY="YES"
WITH_OFED="YES"

>> Now I'm facing the following issue :
>>
>> # iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2
>> iscsid[8743]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
>> kernel: ERROR: iser_connect_error: conn 0xfffff8096786f000
>> iscsid[1078]: child process 8743 terminated with exit status 1
>
> please increase the debug level in both initiator and target and sent it out.

Logs from the initiator :
iscsid[39668]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: INFO: iser_free_ib_conn_res: freeing conn 0xfffff8048d276000 cma_id 0 qp 0
kernel: DEBUG: iser_conn_connect: before cv_wait: 0xfffff8048d276000
kernel: INFO: iser_cma_handler: event 1 status -19 conn 0xfffff8048d276000 id 0xfffff80880606800
kernel: ERROR: iser_connect_error: conn 0xfffff8048d276000
kernel: DEBUG: iser_conn_connect: after cv_wait: 0xfffff8048d276000
iscsid[1078]: child process 39668 terminated with exit status 1

Logs from the target (here I tried TGT) :
No logs, as if TGT did not receive anything.
If I try to connect without the "-r", here is what I get :
tgtd: login_start(502) Target iqn.2012-06.com.test:target1 is RDMA, but conn cid:0 from iqn.1994-09.org.freebsd:srv1 is TCP
So target is correctly iSER configured.
I tried without iSER on both sides, of course it works.

> From first look I think you need to configure the LIO target to work with demo mode.

I double-checked, LIO was in demo mode, I followed this guideline :
https://community.mellanox.com/docs/DOC-1468

One strange thing, on initiator, sounds like no packet goes through the interface when trying to connect :
# sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409

# iscsictl -A -r -t iqn.2012-06.com.test:target1 -p 192.168.2.2

# sysctl hw.mlxen1.stat.rx_packets hw.mlxen1.stat.tx_packets
hw.mlxen1.stat.rx_packets: 386
hw.mlxen1.stat.tx_packets: 409

[Ben RUBSON]

Here are the tests with udaddy from FreeBSD, trying to connect to Linux :

# udaddy -s 192.168.2.2
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: starting client
udaddy: connecting
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
udaddy: event: RDMA_CM_EVENT_ADDR_ERROR, error: -19
test complete
return status -19

Sounds like the issue is with FreeBSD itself.

Ben

[Max Gurtovoy]

Hi,

On 8/11/2016 11:43 PM, Ben RUBSON wrote:
> Here are the tests with udaddy from FreeBSD, trying to connect to Linux :
>
> # udaddy -s 192.168.2.2
> libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
> libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
> udaddy: starting client
> udaddy: connecting
> libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
> udaddy: event: RDMA_CM_EVENT_ADDR_ERROR, error: -19

yes, you got the same event in iser (event 1).
There is a configuration issue with your setup.
Once you overcome this, iSER should work for you.

Meny/Hans,
can you assist here ?

Thanks,
Max.

[Hans Petter Selasky]

Hi,

Error 19 means ENODEV. Possibly something is not supported.

Meny: I wonder if it is better if Ben install our infiniband patches to bring the IB code up to v3.2?

--HPS

[Ben RUBSON]

Hi,

I went with a fresh new install to have a clean status.

What I did :

// FreeBSD 11-RC1 installation
# echo "WITH_OFED=YES" > /etc/make.conf
# cd /usr/src/ ; make buildkernel KERNCONF=GENERIC ; make installkernel KERNCONF=GENERIC
# cd /usr/src/sys/modules/mlx4 ; make ; make install
# cd /usr/src/sys/modules/mlxen ; make ; make install
# cd /usr/src/sys/modules/iser ; make ; make install
# echo "mlx4_load=YES" >> /boot/loader.conf
# echo "mlxen_load=YES" >> /boot/loader.conf
# echo "iser_load=YES" >> /boot/loader.conf
# reboot

Everything went fine, no code modition needed or whatever, perfect.

# iscsictl -A -t iqn.2012-06.com.test:target1 -p 192.168.2.2
On the target, I get :
tgtd[1912]: tgtd: login_start(502) Target iqn.2012-06.com.test:target1 is RDMA, but conn cid:0 from iqn.1994-09.org.freebsd:srv1 is TCP
Perfect, the target is correctly iSER enabled.

# iscsictl -A -t iqn.2012-06.com.test:target1 -p 192.168.2.2 -r
I still get the following error :
iscsid[1062]: 192.168.2.2 (iqn.2012-06.com.test:target1): failed to connect to 192.168.2.2 using ICL kernel proxy: ISCSIDCONNECT: Input/output error
kernel: INFO: iser_free_ib_conn_res: freeing conn 0xfffff80147771000 cma_id 0 qp 0
kernel: DEBUG: iser_conn_connect: before cv_wait: 0xfffff80147771000
kernel: INFO: iser_cma_handler: event 1 status -19 conn 0xfffff80147771000 id 0xfffff8005f0e5400
kernel: ERROR: iser_connect_error: conn 0xfffff80147771000
kernel: DEBUG: iser_conn_connect: after cv_wait: 0xfffff80147771000
iscsid[853]: child process 1062 terminated with exit status 1

Waiting for your instructions, would really be nice to have it working for 11 release.

Many thanks !

Best regards,

Ben

[Ben RUBSON]

> On 13 Aug 2016, at 13:36, Ben RUBSON <ben.r...@gmail.com> wrote:
>
> Hi,
>
> I went with a fresh new install to have a clean status.
>
> What I did :
>
> // FreeBSD 11-RC1 installation
> # echo "WITH_OFED=YES" > /etc/make.conf
> # cd /usr/src/ ; make buildkernel KERNCONF=GENERIC ; make installkernel KERNCONF=GENERIC
> # cd /usr/src/sys/modules/mlx4 ; make ; make install
> # cd /usr/src/sys/modules/mlxen ; make ; make install
> # cd /usr/src/sys/modules/iser ; make ; make install
> # echo "mlx4_load=YES" >> /boot/loader.conf
> # echo "mlxen_load=YES" >> /boot/loader.conf
> # echo "iser_load=YES" >> /boot/loader.conf
> # reboot

This was missing :
cd /usr/src/sys/modules/mlx4ib/ ; make ; make install ; kldload mlx4ib.ko

Now it works perfectly...
I'm really sorry for the useless noise on this list...

I hope this above howto will help others.

Thank you again for your support, and sorry again :-/

Benchmarks will of course follow.

Best regards,

Ben

[Ben RUBSON]

> On 13 Aug 2016, at 15:12, Ben RUBSON <ben.r...@gmail.com> wrote:
>
> Benchmarks will of course follow.

(also posted @freebsd-fs)

I then used Linux TGT as a test target.

### Target : local disk throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes (11 GB) copied, 21.3898 s, 502 MB/s

### Initiator : iscsi disk throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 34.938676 secs (307321843 bytes/sec) - 293 MB/s

### Initiator : iSER disk throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 20.371947 secs (527068838 bytes/sec) - 502 MB/s

In my example, iSER allows to reach the raw disk throughput (on one thread), whereas iSCSI/TCP is a bottleneck.

I also tested with a target RAM device :

### Initiator : iscsi RAM throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 24.194665 secs (443792803 bytes/sec) - 423 MB/s

### Initiator : iSER RAM throughput, one thread :
# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 10.991094 secs (976919873 bytes/sec) - 932 MB/s
(just don't know what is the bottleneck here, of course not the RAM, and network on one thread is around 30GB/s)

> On 11 Aug 2016, at 01:13, Max Gurtovoy <ma...@mellanox.com> wrote:
>
> Regarding plans for the target side, it's not on my plate for the near future (but obviously can be later on :)).
> I'm not aware of other contributers plans.

Can't wait to see this then :)

Many thanks again !

[Max Gurtovoy]

Good to see that you got a nice progress there.
You can try the fio tool and see even better performance.
You can also use Mellanox community site to see how to reach wire spead
and leaverage your numbers. The iser posts are mostly for linux users
but you can find it helpfull for your needs to.

Thanks,
Max.

[Ben RUBSON]

Thank you very much Max for all your advices.

These tests were really interesting as they demonstrated that iSER can easily improve my iSCSI storage infrastructure throughput.

Unfortunately, for production purpose, I need FreeBSD as the target system.
I will then keep an eye on future FreeBSD releases related to iSER.

Thank you again !

Ben

[Max Gurtovoy]

NP.

I never tried it on FreeBSD but you can try to build the user space
target TGT on it.
Maybe you'll need some adaptations but should be easier that adding a
new iser target driver from scratch.