Failure to login iser CentOS 7 3.16.3-1 elrepo

[Moussa Ba]

I have a target running CentOS 7 with LIO.  I am using iser transport via Mellanox Connect-X 3 VPI cards.  I have been having difficulties connecting via an initiator running CentOS 7. My setup is essentially a one target/LUN/PORT. 

 o- iscsi ............................................................................................................ [Targets: 1]

  | o- iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 ....................................................... [TPGs: 1]

  |   o- tpg1 .................................................................................................. [gen-acls, no-auth]

  |     o- acls .......................................................................................................... [ACLs: 1]

  |     | o- iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 .......................................... [Mapped LUNs: 1]

  |     |   o- mapped_lun0 ............................................................. [lun0 block/vg-targetd:test_volume_02 (rw)]

  |     o- luns .......................................................................................................... [LUNs: 1]

  |     | o- lun0 ............................................... [block/vg-targetd:test_volume_02 (/dev/vg-targetd/test_volume_02)]

  |     o- portals .................................................................................................... [Portals: 1]

  |       o- 192.168.111.100:3261 ........................................................................................... [iser]

Target Kernel: 3.16.3-1.el7.elrepo

Initiator Kernel:3.16.3-1.el7.elrepo

iscsiadm: Compiled from git repository.  Latest commit

commit 76a441ba0dc0071a19daeac456aa898889437efd

Author: Hannes Reinecke <.....>

Date:   Fri Sep 12 14:42:01 2014 -0700

    Added new util script to aid in CNA setup

   

Here is my flow:

iscsiadm -m iface -I iface0 --op=new

iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v iser

iscsiadm -m discovery -t st -p 192.168.111.100:3261 -I iface0

Output is:

New interface iface0 added

192.168.111.100:3261,1 iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9

iface0 updated.

iscsiadm: No records found

See below for debug output.

#iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 -I iface0 -p 192.168.111.100:3261 -l - d 8

iscsiadm: Max file limits 1024 4096

iscsiadm: looking for iface conf /etc/iscsi/ifaces/iface0

iscsiadm: updated 'iface.iscsi_ifacename', 'iface0' => 'iface0'

iscsiadm: updated 'iface.transport_name', '' => 'iser'

iscsiadm: updated 'iface.vlan_id', '0' => '0'

iscsiadm: updated 'iface.vlan_priority', '0' => '0'

iscsiadm: updated 'iface.iface_num', '0' => '0'

iscsiadm: updated 'iface.mtu', '0' => '0'

iscsiadm: updated 'iface.port', '0' => '0'

iscsiadm: updated 'iface.tos', '0' => '0'

iscsiadm: updated 'iface.ttl', '0' => '0'

iscsiadm: updated 'iface.tcp_wsf', '0' => '0'

iscsiadm: updated 'iface.tcp_timer_scale', '0' => '0'

iscsiadm: updated 'iface.def_task_mgmt_timeout', '0' => '0'

iscsiadm: updated 'iface.erl', '0' => '0'

iscsiadm: updated 'iface.max_receive_data_len', '0' => '0'

iscsiadm: updated 'iface.first_burst_len', '0' => '0'

iscsiadm: updated 'iface.max_outstanding_r2t', '0' => '0'

iscsiadm: updated 'iface.max_burst_len', '0' => '0'

iscsiadm: exec_node_op: iser:iface0 node [iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9,192.168.111.100,3261] sid 0

iscsiadm: searching iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9

iscsiadm: found 192.168.111.100,3261,1

iscsiadm: iface iter found iface0.

iscsiadm: updated 'node.name', '' => 'iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9'

iscsiadm: updated 'node.tpgt', '-1' => '1'

iscsiadm: updated 'node.startup', 'manual' => 'automatic'

iscsiadm: updated 'node.leading_login', 'No' => 'No'

iscsiadm: updated 'iface.iscsi_ifacename', 'default' => 'iface0'

iscsiadm: updated 'iface.transport_name', 'tcp' => 'tcp'

iscsiadm: updated 'iface.vlan_id', '0' => '0'

iscsiadm: updated 'iface.vlan_priority', '0' => '0'

iscsiadm: updated 'iface.iface_num', '0' => '0'

iscsiadm: updated 'iface.mtu', '0' => '0'

iscsiadm: updated 'iface.port', '0' => '0'

iscsiadm: updated 'iface.tos', '0' => '0'

iscsiadm: updated 'iface.ttl', '0' => '0'

iscsiadm: updated 'iface.tcp_wsf', '0' => '0'

iscsiadm: updated 'iface.tcp_timer_scale', '0' => '0'

iscsiadm: updated 'iface.def_task_mgmt_timeout', '0' => '0'

iscsiadm: updated 'iface.erl', '0' => '0'

iscsiadm: updated 'iface.max_receive_data_len', '0' => '0'

iscsiadm: updated 'iface.first_burst_len', '0' => '0'

iscsiadm: updated 'iface.max_outstanding_r2t', '0' => '0'

iscsiadm: updated 'iface.max_burst_len', '0' => '0'

iscsiadm: updated 'node.discovery_address', '' => '192.168.111.100'

iscsiadm: updated 'node.discovery_port', '0' => '3261'

iscsiadm: updated 'node.discovery_type', 'static' => 'send_targets'

iscsiadm: updated 'node.session.initial_cmdsn', '0' => '0'

iscsiadm: updated 'node.session.initial_login_retry_max', '4' => '8'

iscsiadm: updated 'node.session.xmit_thread_priority', '-20' => '-20'

iscsiadm: updated 'node.session.cmds_max', '128' => '128'

iscsiadm: updated 'node.session.queue_depth', '32' => '32'

iscsiadm: updated 'node.session.nr_sessions', '1' => '1'

iscsiadm: updated 'node.session.auth.authmethod', 'None' => 'None'

iscsiadm: updated 'node.session.timeo.replacement_timeout', '120' => '120'

iscsiadm: updated 'node.session.err_timeo.abort_timeout', '15' => '15'

iscsiadm: updated 'node.session.err_timeo.lu_reset_timeout', '30' => '30'

iscsiadm: updated 'node.session.err_timeo.tgt_reset_timeout', '30' => '30'

iscsiadm: updated 'node.session.err_timeo.host_reset_timeout', '60' => '60'

iscsiadm: updated 'node.session.iscsi.FastAbort', 'Yes' => 'Yes'

iscsiadm: updated 'node.session.iscsi.InitialR2T', 'No' => 'No'

iscsiadm: updated 'node.session.iscsi.ImmediateData', 'Yes' => 'Yes'

iscsiadm: updated 'node.session.iscsi.FirstBurstLength', '262144' => '262144'

iscsiadm: updated 'node.session.iscsi.MaxBurstLength', '16776192' => '16776192'

iscsiadm: updated 'node.session.iscsi.DefaultTime2Retain', '0' => '0'

iscsiadm: updated 'node.session.iscsi.DefaultTime2Wait', '2' => '2'

iscsiadm: updated 'node.session.iscsi.MaxConnections', '1' => '1'

iscsiadm: updated 'node.session.iscsi.MaxOutstandingR2T', '1' => '1'

iscsiadm: updated 'node.session.iscsi.ERL', '0' => '0'

iscsiadm: updated 'node.conn[0].address', '' => '192.168.111.100'

iscsiadm: updated 'node.conn[0].port', '3260' => '3261'

iscsiadm: updated 'node.conn[0].startup', 'manual' => 'manual'

iscsiadm: updated 'node.conn[0].tcp.window_size', '524288' => '524288'

iscsiadm: updated 'node.conn[0].tcp.type_of_service', '0' => '0'

iscsiadm: updated 'node.conn[0].timeo.logout_timeout', '15' => '15'

iscsiadm: updated 'node.conn[0].timeo.login_timeout', '30' => '15'

iscsiadm: updated 'node.conn[0].timeo.auth_timeout', '45' => '45'

iscsiadm: updated 'node.conn[0].timeo.noop_out_interval', '5' => '5'

iscsiadm: updated 'node.conn[0].timeo.noop_out_timeout', '5' => '5'

iscsiadm: updated 'node.conn[0].iscsi.MaxXmitDataSegmentLength', '0' => '0'

iscsiadm: updated 'node.conn[0].iscsi.MaxRecvDataSegmentLength', '262144' => '262144'

iscsiadm: updated 'node.conn[0].iscsi.HeaderDigest', 'None' => 'None'

iscsiadm: updated 'node.conn[0].iscsi.DataDigest', 'None' => 'None'

iscsiadm: updated 'node.conn[0].iscsi.IFMarker', 'No' => 'No'

iscsiadm: updated 'node.conn[0].iscsi.OFMarker', 'No' => 'No'

iscsiadm: match session [iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9,192.168.111.100,3261][iface0 iser,,]:0

iscsiadm: to [iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9,192.168.111.100,3261][iface0 tcp,,]:0

iscsiadm: No records found

[Michael Christie]

The above is kinda hard to read, because I am not sure what output goes with what command.

Before you ran the discovery command did you have some portals discovered? When you ran the discovery command did it fail to find any portals? Is that what the last no records line is from?

Could you tar up /etc/iscsi and send it to me, so I can try running iscsiadm here and debug it.

[Moussa Ba]

See commands below with output in bold. A discovery does generate a portal.  Note however, that the discovery is going over tcp not iser. I follow the discovery with an update to the interface to change the transport_name to iser followed by a login attempt.  It is the login attempt that returns a iscsiadm:No records found.

#iscsiadm -m iface -I iface0 --op=new

New interface iface0 added

#iscsiadm -m discovery -t st -p 192.168.111.100:3261 -I iface0

192.168.111.100:3261,1 iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9

#iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v iser

iface0 updated.

#iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 -I iface0 -p 192.168.111.100:3261 -l

iscsiadm: No records found

[Mike Christie]

Ok. I am looking into it now. As a temp workaround you can do

iscsiadm -m node -l

or

iscsiadm -m node -T yourtarget -l

On 10/13/2014 12:45 PM, Moussa Ba wrote:
> See commands below with output in bold. A discovery does generate a
> portal. Note however, that the discovery is going over tcp not iser. I
> follow the discovery with an update to the interface to change the
> transport_name to iser followed by a login attempt. It is the login
> attempt that returns a iscsiadm:No records found.
>
>
> #iscsiadm -m iface -I iface0 --op=new

> *New interface iface0 added*

> #iscsiadm -m discovery -t st -p 192.168.111.100:3261 -I iface0

> *192.168.111.100:3261,1
> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9*

> #iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v iser

> *iface0 updated.*

> #iscsiadm -m node -T
> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 -I iface0 -p
> 192.168.111.100:3261 -l

> *iscsiadm: No records found*
> *
> *

>
>
> On Saturday, October 11, 2014 4:26:06 PM UTC-7, Mike Christie wrote:
>
>
> On Oct 10, 2014, at 5:06 PM, Moussa Ba <mus...@gmail.com

> > | o- 192.168.111.100:3261 <http://192.168.111.100:3261>

> ...........................................................................................
> [iser]
> >
> >
> > Target Kernel: 3.16.3-1.el7.elrepo
> > Initiator Kernel:3.16.3-1.el7.elrepo
> >
> > iscsiadm: Compiled from git repository. Latest commit
> >
> > commit 76a441ba0dc0071a19daeac456aa898889437efd
> > Author: Hannes Reinecke <.....>
> > Date: Fri Sep 12 14:42:01 2014 -0700
> > Added new util script to aid in CNA setup
> >
> >
> >
> > Here is my flow:
> >
> > iscsiadm -m iface -I iface0 --op=new
> > iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v iser
> > iscsiadm -m discovery -t st -p 192.168.111.100:3261

> <http://192.168.111.100:3261> -I iface0

> >
> > Output is:
> >
> > New interface iface0 added

> > 192.168.111.100:3261 <http://192.168.111.100:3261>,1

> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9
> > iface0 updated.
> > iscsiadm: No records found
> >
>
> The above is kinda hard to read, because I am not sure what output
> goes with what command.
>
> Before you ran the discovery command did you have some portals
> discovered? When you ran the discovery command did it fail to find
> any portals? Is that what the last no records line is from?
>
> Could you tar up /etc/iscsi and send it to me, so I can try running
> iscsiadm here and debug it.
>

> --
> You received this message because you are subscribed to the Google
> Groups "open-iscsi" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to open-iscsi+...@googlegroups.com
> <mailto:open-iscsi+...@googlegroups.com>.
> To post to this group, send email to open-...@googlegroups.com
> <mailto:open-...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/open-iscsi.
> For more options, visit https://groups.google.com/d/optout.

[Mike Christie]

Another workaround, if possible on your setup, is to use port 3260.
Looks like there is a bug in the iscsiadm code when the port is not 3260.

[Moussa Ba]

The problem is that I am using one target/LUN per port as I am sharing PCIe SSDs backed LUNS with high IOPS. Basically, this was the recommendation from Mellanox such that I have one session per port.  In my testing when it was working, I was getting about 800KIOPS for 8 sessions.  I do need to be able to use different ports.

Thank you.

>> <mailto:open-iscsi+unsub...@googlegroups.com>.

[Mike Christie]

Ignore this comment about the port issue. Not what you are hitting.

[Mike Christie]

On 10/13/2014 12:45 PM, Moussa Ba wrote:

> #iscsiadm -m iface -I iface0 --op=new

> *New interface iface0 added*

> #iscsiadm -m discovery -t st -p 192.168.111.100:3261 -I iface0

> *192.168.111.100:3261,1
> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9*

> #iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v iser

> *iface0 updated.*

> #iscsiadm -m node -T
> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 -I iface0 -p
> 192.168.111.100:3261 -l

> *iscsiadm: No records found*
> *

I added a fix for this in git. If you wanted to use the tools that code
with Centos 7 just use the default iser iface.

Just do:

iscsiadm -m discovery -t st -p 192.168.111.100:3261 -I iser
iscsiadm -m node -T your_target -p ip -I iser -l

[Mike Christie]

Ok. Does the other login command work for you or does the use of the
default iser iface work?

> <http://192.168.111.100:3261> -I iface0

> >> *192.168.111.100:3261 <http://192.168.111.100:3261>,1

> >> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9*
> >> #iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v
> iser
> >> *iface0 updated.*
> >> #iscsiadm -m node -T
> >> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 -I
> iface0 -p

> >> 192.168.111.100:3261 <http://192.168.111.100:3261> -l

> <http://192.168.111.100:3261> <http://192.168.111.100:3261>

> >> an email to open-iscsi+...@googlegroups.com <javascript:>
> >> <mailto:open-iscsi+...@googlegroups.com <javascript:>>.

> >> To post to this group, send email to open-...@googlegroups.com

> <javascript:>
> >> <mailto:open-...@googlegroups.com <javascript:>>.

> >> Visit this group at http://groups.google.com/group/open-iscsi

> <http://groups.google.com/group/open-iscsi>.

> >> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

> >
>
> --
> You received this message because you are subscribed to the Google
> Groups "open-iscsi" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to open-iscsi+...@googlegroups.com

> <mailto:open-iscsi+...@googlegroups.com>.

[Mike Christie]

On 10/13/2014 02:44 PM, Mike Christie wrote:
> On 10/13/2014 12:45 PM, Moussa Ba wrote:
>> #iscsiadm -m iface -I iface0 --op=new
>> *New interface iface0 added*
>> #iscsiadm -m discovery -t st -p 192.168.111.100:3261 -I iface0
>> *192.168.111.100:3261,1
>> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9*
>> #iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v iser
>> *iface0 updated.*
>> #iscsiadm -m node -T
>> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 -I iface0 -p
>> 192.168.111.100:3261 -l
>> *iscsiadm: No records found*
>> *
>
> I added a fix for this in git. If you wanted to use the tools that code

Actually, it had a bug. Reverted it.

> with Centos 7 just use the default iser iface.
>
> Just do:
>
> iscsiadm -m discovery -t st -p 192.168.111.100:3261 -I iser
> iscsiadm -m node -T your_target -p ip -I iser -l
>

Also you can just do:

#iscsiadm -m iface -I iface0 --op=new

#iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v iser

Then do discovery and login:

#iscsiadm -m discovery -t st -p 192.168.111.100:3261 -I iface0

[Moussa Ba]

I am confused....Is there new code for me to try?  Trying to do a discovery after updating the interface using iser results in a :

iscsiadm: Connection to discovery portal 192.168.111.100 failed: encountered connection failure

That was the reason why I switched it around where I change the interface to iser after doing a discovery via tcp which works and a login after updating interface to iser.

I am still using git code at commit

commit 76a441ba0dc0071a19daeac456aa898889437efd

Author: Hannes Reinecke <ha...@suse.de>

Date:   Fri Sep 12 14:42:01 2014 -0700

Moussa

[Mike Christie]

On 10/13/2014 03:49 PM, Moussa Ba wrote:
> I am confused....Is there new code for me to try? Trying to do a

No.

> discovery after updating the interface using iser results in a :

> *iscsiadm: Connection to discovery portal 192.168.111.100 failed:
> encountered connection failure*

The way I told you to do it, is normally how it is done. Either that or
you use the iscsi_discovery script which sets the node's transport_name
manually.

Does the discovery error happen with the upstream tools or the one
shipped with Centos 7?

>
> That was the reason why I switched it around where I change the
> interface to iser after doing a discovery via tcp which works and a
> login after updating interface to iser.
>
> I am still using git code at commit
>
> commit 76a441ba0dc0071a19daeac456aa898889437efd
> Author: Hannes Reinecke <ha...@suse.de>
> Date: Fri Sep 12 14:42:01 2014 -0700
>
>
> Moussa
>
> On Monday, October 13, 2014 12:50:24 PM UTC-7, Mike Christie wrote:
>
> On 10/13/2014 02:44 PM, Mike Christie wrote:
> > On 10/13/2014 12:45 PM, Moussa Ba wrote:
> >> #iscsiadm -m iface -I iface0 --op=new
> >> *New interface iface0 added*
> >> #iscsiadm -m discovery -t st -p 192.168.111.100:3261

> <http://192.168.111.100:3261> -I iface0
> >> *192.168.111.100:3261 <http://192.168.111.100:3261>,1

> >> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9*
> >> #iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v
> iser
> >> *iface0 updated.*
> >> #iscsiadm -m node -T
> >> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 -I
> iface0 -p

> >> 192.168.111.100:3261 <http://192.168.111.100:3261> -l

> >> *iscsiadm: No records found*
> >> *
> >
> > I added a fix for this in git. If you wanted to use the tools that
> code
>
> Actually, it had a bug. Reverted it.
>
> > with Centos 7 just use the default iser iface.
> >
> > Just do:
> >
> > iscsiadm -m discovery -t st -p 192.168.111.100:3261

> <http://192.168.111.100:3261> -I iser

> > iscsiadm -m node -T your_target -p ip -I iser -l
> >
>
> Also you can just do:
>
> #iscsiadm -m iface -I iface0 --op=new
> #iscsiadm -m iface -I iface0 -o update -n iface.transport_name -v iser
>
> Then do discovery and login:
>
> #iscsiadm -m discovery -t st -p 192.168.111.100:3261

> <http://192.168.111.100:3261> -I iface0

> #iscsiadm -m node -T
> iqn.2003-01.org.linux-iscsi.localhost.x8664:sn.0fa6846566d9 -I
> iface0 -p

> 192.168.111.100:3261 <http://192.168.111.100:3261> -l

[Mike Christie]

On 10/13/2014 10:04 PM, Mike Christie wrote:
> On 10/13/2014 03:49 PM, Moussa Ba wrote:
>> I am confused....Is there new code for me to try? Trying to do a
>
> No.
>
>> discovery after updating the interface using iser results in a :
>> *iscsiadm: Connection to discovery portal 192.168.111.100 failed:
>> encountered connection failure*
>
> The way I told you to do it, is normally how it is done. Either that or
> you use the iscsi_discovery script which sets the node's transport_name
> manually.
>
> Does the discovery error happen with the upstream tools or the one
> shipped with Centos 7?
>

I am betting you are using the git tools with the Centos 7 kernel? If so
then that will not work. Mellanox added support to do discovery through
the infinniband layer in the upstream git tools and upstream kernel. The
Centos 7 kernel does not support it though.

[Or Gerlitz]

Mike, the RHEL7 kernel has this backported from upstream and I assume it
landed in CentOS too, I see now that the kernel is 3.16.x --> it's there

[Mike Christie]

On 10/13/14, 10:22 PM, Mike Christie wrote:
> On 10/13/2014 10:04 PM, Mike Christie wrote:
>> On 10/13/2014 03:49 PM, Moussa Ba wrote:
>>> I am confused....Is there new code for me to try? Trying to do a
>>
>> No.
>>
>>> discovery after updating the interface using iser results in a :
>>> *iscsiadm: Connection to discovery portal 192.168.111.100 failed:
>>> encountered connection failure*
>>
>> The way I told you to do it, is normally how it is done. Either that or
>> you use the iscsi_discovery script which sets the node's transport_name
>> manually.
>>
>> Does the discovery error happen with the upstream tools or the one
>> shipped with Centos 7?
>>
>
> I am betting you are using the git tools with the Centos 7 kernel? If so

Actually, I will lose that bet :) It looks like we added compat code so
if the kernel does not support the new discovery method then the old
style should be used. Run the discovery command with debugging on

iscsiadm -m discovery -t st -p ip -d 8

and send the output, so we can see why it failed.

[Mike Christie]

Ah yeah, you are right I see the iser kernel changes in RHEL 7, so yeah
it should be in centos.

So we probably need to run the discovery command with debugging -d 8, to
see why it fails.

[Moussa Ba]

[jenkins@l-cheddar1 santa]$ sudo iscsiadm -m discovery -t st -p 192.168.111.100:3261 -I iface0 -d 8

iscsiadm: Max file limits 1024 4096

iscsiadm: updating defaults from '/etc/iscsi/iscsid.conf'

iscsiadm: updated 'discovery.sendtargets.iscsi.MaxRecvDataSegmentLength', '32768' => '32768'

iscsiadm: updated 'node.startup', 'manual' => 'automatic'

iscsiadm: updated 'node.leading_login', 'No' => 'No'

iscsiadm: updated 'node.session.timeo.replacement_timeout', '120' => '120'

iscsiadm: updated 'node.conn[0].timeo.login_timeout', '30' => '15'

iscsiadm: updated 'node.conn[0].timeo.logout_timeout', '15' => '15'

iscsiadm: updated 'node.conn[0].timeo.noop_out_interval', '5' => '5'

iscsiadm: updated 'node.conn[0].timeo.noop_out_timeout', '5' => '5'

iscsiadm: updated 'node.session.err_timeo.abort_timeout', '15' => '15'

iscsiadm: updated 'node.session.err_timeo.lu_reset_timeout', '30' => '30'

iscsiadm: updated 'node.session.err_timeo.tgt_reset_timeout', '30' => '30'

iscsiadm: updated 'node.session.initial_login_retry_max', '4' => '8'

iscsiadm: updated 'node.session.cmds_max', '128' => '128'

iscsiadm: updated 'node.session.queue_depth', '32' => '32'

iscsiadm: updated 'node.session.xmit_thread_priority', '-20' => '-20'

iscsiadm: updated 'node.session.iscsi.InitialR2T', 'No' => 'No'

iscsiadm: updated 'node.session.iscsi.ImmediateData', 'Yes' => 'Yes'

iscsiadm: updated 'node.session.iscsi.FirstBurstLength', '262144' => '262144'

iscsiadm: updated 'node.session.iscsi.MaxBurstLength', '16776192' => '16776192'

iscsiadm: updated 'node.conn[0].iscsi.MaxRecvDataSegmentLength', '262144' => '262144'

iscsiadm: updated 'node.conn[0].iscsi.MaxXmitDataSegmentLength', '0' => '0'

iscsiadm: updated 'node.conn[0].iscsi.HeaderDigest', 'None' => 'None'

iscsiadm: updated 'node.session.nr_sessions', '1' => '1'

iscsiadm: updated 'node.session.iscsi.FastAbort', 'Yes' => 'Yes'

iscsiadm: looking for iface conf /etc/iscsi/ifaces/iface0

iscsiadm: updated 'iface.iscsi_ifacename', 'iface0' => 'iface0'

iscsiadm: updated 'iface.transport_name', '' => 'iser'

iscsiadm: updated 'iface.vlan_id', '0' => '0'

iscsiadm: updated 'iface.vlan_priority', '0' => '0'

iscsiadm: updated 'iface.iface_num', '0' => '0'

iscsiadm: updated 'iface.mtu', '0' => '0'

iscsiadm: updated 'iface.port', '0' => '0'

iscsiadm: updated 'iface.tos', '0' => '0'

iscsiadm: updated 'iface.ttl', '0' => '0'

iscsiadm: updated 'iface.tcp_wsf', '0' => '0'

iscsiadm: updated 'iface.tcp_timer_scale', '0' => '0'

iscsiadm: updated 'iface.def_task_mgmt_timeout', '0' => '0'

iscsiadm: updated 'iface.erl', '0' => '0'

iscsiadm: updated 'iface.max_receive_data_len', '0' => '0'

iscsiadm: updated 'iface.first_burst_len', '0' => '0'

iscsiadm: updated 'iface.max_outstanding_r2t', '0' => '0'

iscsiadm: updated 'iface.max_burst_len', '0' => '0'

iscsiadm: in read_transports

iscsiadm: Adding new transport iser

iscsiadm: Matched transport iser

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'handle'

iscsiadm: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/iser/handle'

iscsiadm: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/iser/handle'

iscsiadm: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/iser/handle' with attribute value '18446744072100208672'

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'caps'

iscsiadm: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/iser/caps'

iscsiadm: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/iser/caps'

iscsiadm: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/iser/caps' with attribute value '0x89'

iscsiadm: Adding new transport tcp

iscsiadm: Matched transport tcp

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'handle'

iscsiadm: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/tcp/handle'

iscsiadm: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/tcp/handle'

iscsiadm: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/tcp/handle' with attribute value '18446744072099696672'

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'caps'

iscsiadm: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/tcp/caps'

iscsiadm: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/tcp/caps'

iscsiadm: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/tcp/caps' with attribute value '0x39'

iscsiadm: Could not match iface[hw=,ip=,net_if=,iscsi_if=iface0] to host.

iscsiadm: Looking for config file /etc/iscsi/send_targets/192.168.111.100,3261

iscsiadm: Looking for config file /etc/iscsi/send_targets/192.168.111.100,3261 config st_config.

iscsiadm: updated 'discovery.startup', 'manual' => 'manual'

iscsiadm: updated 'discovery.type', 'sendtargets' => 'sendtargets'

iscsiadm: updated 'discovery.sendtargets.address', '' => '192.168.111.100'

iscsiadm: updated 'discovery.sendtargets.port', '0' => '3261'

iscsiadm: updated 'discovery.sendtargets.auth.authmethod', 'None' => 'None'

iscsiadm: updated 'discovery.sendtargets.timeo.login_timeout', '15' => '15'

iscsiadm: updated 'discovery.sendtargets.use_discoveryd', 'No' => 'No'

iscsiadm: updated 'discovery.sendtargets.discoveryd_poll_inval', '30' => '30'

iscsiadm: updated 'discovery.sendtargets.reopen_max', '5' => '5'

iscsiadm: updated 'discovery.sendtargets.timeo.auth_timeout', '45' => '45'

iscsiadm: updated 'discovery.sendtargets.timeo.active_timeout', '30' => '30'

iscsiadm: updated 'discovery.sendtargets.iscsi.MaxRecvDataSegmentLength', '32768' => '32768'

iscsiadm: disc rec already exists

iscsiadm: Looking for config file /etc/iscsi/send_targets/192.168.111.100,3261

iscsiadm: starting sendtargets discovery, address 192.168.111.100:3261, 

iscsiadm: in read_transports

iscsiadm: Updating transport iser

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'handle'

iscsiadm: sysfs_attr_get_value: found in cache '/class/iscsi_transport/iser/handle'

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/iser'/'caps'

iscsiadm: sysfs_attr_get_value: found in cache '/class/iscsi_transport/iser/caps'

iscsiadm: Updating transport tcp

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'handle'

iscsiadm: sysfs_attr_get_value: found in cache '/class/iscsi_transport/tcp/handle'

iscsiadm: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'caps'

iscsiadm: sysfs_attr_get_value: found in cache '/class/iscsi_transport/tcp/caps'

iscsiadm: no authentication configured...

iscsiadm: sendtargets discovery to 192.168.111.100:3261 using isid 0x00023d000000

iscsiadm: resolved 192.168.111.100 to 192.168.111.100

iscsiadm: discovery timeouts: login 15, reopen_cnt 6, auth 45.

iscsiadm: in ctldev_open

iscsiadm: created NETLINK_ISCSI socket...

iscsiadm: setting iface iface0, dev , set ip , hw , transport iser.

iscsiadm: iscsi_create_leading_conn discovery ep connect

iscsiadm: in ktransport_ep_connect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: ktransport_ep_connect got handle 1

iscsiadm: in ktransport_ep_poll

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ktransport_ep_disconnect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ctldev_close

iscsiadm: Connection to discovery portal 192.168.111.100 failed: encountered connection failure

iscsiadm: discovery session to 192.168.111.100:3261 sleeping for 1 seconds before next login attempt

iscsiadm: in ctldev_open

iscsiadm: created NETLINK_ISCSI socket...

iscsiadm: setting iface iface0, dev , set ip , hw , transport iser.

iscsiadm: iscsi_create_leading_conn discovery ep connect

iscsiadm: in ktransport_ep_connect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: ktransport_ep_connect got handle 1

iscsiadm: in ktransport_ep_poll

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ktransport_ep_disconnect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ctldev_close

iscsiadm: Connection to discovery portal 192.168.111.100 failed: encountered connection failure

iscsiadm: discovery session to 192.168.111.100:3261 sleeping for 1 seconds before next login attempt

iscsiadm: in ctldev_open

iscsiadm: created NETLINK_ISCSI socket...

iscsiadm: setting iface iface0, dev , set ip , hw , transport iser.

iscsiadm: iscsi_create_leading_conn discovery ep connect

iscsiadm: in ktransport_ep_connect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: ktransport_ep_connect got handle 1

iscsiadm: in ktransport_ep_poll

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ktransport_ep_disconnect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ctldev_close

iscsiadm: Connection to discovery portal 192.168.111.100 failed: encountered connection failure

iscsiadm: discovery session to 192.168.111.100:3261 sleeping for 1 seconds before next login attempt

iscsiadm: in ctldev_open

iscsiadm: created NETLINK_ISCSI socket...

iscsiadm: setting iface iface0, dev , set ip , hw , transport iser.

iscsiadm: iscsi_create_leading_conn discovery ep connect

iscsiadm: in ktransport_ep_connect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: ktransport_ep_connect got handle 1

iscsiadm: in ktransport_ep_poll

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ktransport_ep_disconnect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ctldev_close

iscsiadm: Connection to discovery portal 192.168.111.100 failed: encountered connection failure

iscsiadm: discovery session to 192.168.111.100:3261 sleeping for 1 seconds before next login attempt

iscsiadm: in ctldev_open

iscsiadm: created NETLINK_ISCSI socket...

iscsiadm: setting iface iface0, dev , set ip , hw , transport iser.

iscsiadm: iscsi_create_leading_conn discovery ep connect

iscsiadm: in ktransport_ep_connect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: ktransport_ep_connect got handle 1

iscsiadm: in ktransport_ep_poll

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ktransport_ep_disconnect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ctldev_close

iscsiadm: Connection to discovery portal 192.168.111.100 failed: encountered connection failure

iscsiadm: discovery session to 192.168.111.100:3261 sleeping for 1 seconds before next login attempt

iscsiadm: in ctldev_open

iscsiadm: created NETLINK_ISCSI socket...

iscsiadm: setting iface iface0, dev , set ip , hw , transport iser.

iscsiadm: iscsi_create_leading_conn discovery ep connect

iscsiadm: in ktransport_ep_connect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: ktransport_ep_connect got handle 1

iscsiadm: in ktransport_ep_poll

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ktransport_ep_disconnect

iscsiadm: in __kipc_call

iscsiadm: in kwritev

iscsiadm: in nlpayload_read

iscsiadm: in nlpayload_read

iscsiadm: in ctldev_close

iscsiadm: Connection to discovery portal 192.168.111.100 failed: encountered connection failure

iscsiadm: connection login retries (reopen_max) 5 exceeded

iscsiadm: No portals found

[Mike Christie]

On 10/14/2014 01:50 PM, Moussa Ba wrote:
>
> iscsiadm: iscsi_create_leading_conn discovery ep connect
>
> iscsiadm: in ktransport_ep_connect
> iscsiadm: in __kipc_call
> iscsiadm: in kwritev
> iscsiadm: in nlpayload_read
> iscsiadm: in nlpayload_read
> iscsiadm: ktransport_ep_connect got handle 1
> iscsiadm: in ktransport_ep_poll
> iscsiadm: in __kipc_call
> iscsiadm: in kwritev
> iscsiadm: in nlpayload_read
> iscsiadm: in nlpayload_read
> iscsiadm: in ktransport_ep_disconnect
> iscsiadm: in __kipc_call
> iscsiadm: in kwritev
> iscsiadm: in nlpayload_read
> iscsiadm: in nlpayload_read
> iscsiadm: in ctldev_close
> iscsiadm: Connection to discovery portal 192.168.111.100 failed:
> encountered connection failure
> iscsiadm: discovery session to 192.168.111.100:3261 sleeping for 1
> seconds before next login attempt

Or and Sagi, it looks like we are hitting this in the userspace
discovvery connection code:

do {
rc = t->template->ep_poll(conn, 1);
if (rc < 0) {
rc = ISCSI_ERR_TRANS;
goto disconnect;

So I think iscsi_iser_ep_poll either returned -1 or -ERESTARTSYS. It was
probably not -ERESTARTSYS and the
wait_for_completion_interruptible_timeout call probably timed out. Did
we hit

mutex_lock(&ib_conn->state_mutex);
if (ib_conn->state == ISER_CONN_TERMINATING ||
ib_conn->state == ISER_CONN_DOWN)
rc = -1;
mutex_unlock(&ib_conn->state_mutex);

then? What would be the reason for that? Target not setup to take iser
ib connections?

[Or Gerlitz]

Mike, It's hard for me to tell now... also the team has a holiday, but
it would worth if Moussa would set debug_level=2 for the ib_iser kernel
module and send the resulted dmesg when he runs the discovery attempt.
Ariel/Sagi - note it is 3.16

Or.

[Moussa Ba]

Here is the dmesg from ib_iser debug_level=2 while doing a discovery via iser. 

[512042.466216] iscsi: registered transport (iser)

[512140.195141] iser: iser_connect:connecting to: 192.168.111.100, port 0xbd0c

[512140.195896] iser: iser_cma_handler:event 1 status -19 conn ffff8800d03462c8 id ffff8806acc0ac00

[512140.195901] iser: iscsi_iser_ep_poll:ib conn ffff8800d03462c8 rc = -1

[512140.195922] iser: iscsi_iser_ep_disconnect:ep ffff8800d0346000 ib conn ffff8800d03462c8 state 4

[512140.195923] iser: iser_conn_terminate:Failed to disconnect, conn: 0xffff8800d03462c8 err -22

[512140.195925] iser: iser_free_ib_conn_res:freeing conn ffff8800d03462c8 cma_id ffff8806acc0ac00 qp           (null)

[512141.197712] iser: iser_connect:connecting to: 192.168.111.100, port 0xbd0c

[512141.198377] iser: iser_cma_handler:event 1 status -19 conn ffff8807a271a2c8 id ffff88079fe4e000

[512141.198383] iser: iscsi_iser_ep_poll:ib conn ffff8807a271a2c8 rc = -1

[512141.198403] iser: iscsi_iser_ep_disconnect:ep ffff8807a271a000 ib conn ffff8807a271a2c8 state 4

[512141.198405] iser: iser_conn_terminate:Failed to disconnect, conn: 0xffff8807a271a2c8 err -22

[512141.198408] iser: iser_free_ib_conn_res:freeing conn ffff8807a271a2c8 cma_id ffff88079fe4e000 qp           (null)

[512142.200339] iser: iser_connect:connecting to: 192.168.111.100, port 0xbd0c

[512142.200893] iser: iser_cma_handler:event 1 status -19 conn ffff8807a2cd62c8 id ffff8807eff77000

[512142.200899] iser: iscsi_iser_ep_poll:ib conn ffff8807a2cd62c8 rc = 0

[512143.203350] iser: iscsi_iser_ep_poll:ib conn ffff8807a2cd62c8 rc = -1

[512143.203403] iser: iscsi_iser_ep_disconnect:ep ffff8807a2cd6000 ib conn ffff8807a2cd62c8 state 4

[512143.203410] iser: iser_conn_terminate:Failed to disconnect, conn: 0xffff8807a2cd62c8 err -22

[512143.203416] iser: iser_free_ib_conn_res:freeing conn ffff8807a2cd62c8 cma_id ffff8807eff77000 qp           (null)

[512144.205638] iser: iser_connect:connecting to: 192.168.111.100, port 0xbd0c

[512144.205781] iser: iscsi_iser_ep_poll:ib conn ffff8807a271a2c8 rc = 0

[512144.205790] iser: iser_cma_handler:event 1 status -19 conn ffff8807a271a2c8 id ffff88079fe49000

[Or Gerlitz]

On 10/16/2014 08:44 PM, Moussa Ba wrote:
> Here is the dmesg from ib_iser debug_level=2 while doing a discovery
> via iser.
>
> [512042.466216] iscsi: registered transport (iser)
> [512140.195141] iser: iser_connect:connecting to: 192.168.111.100,
> port 0xbd0c
> [512140.195896] iser: iser_cma_handler:event 1 status -19 conn
> ffff8800d03462c8 id ffff8806acc0ac00
> [512140.195901] iser: iscsi_iser_ep_poll:ib conn ffff8800d03462c8 rc = -1

So there's some address translation error here, can you ping that
address? through which net-device the route to this IP address goes? is
it the mlx4_en netdevice? can you send

$ route -n
$ ip a s

[Moussa Ba]

I can ping the address, it seems to be going through the correct device.

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

0.0.0.0         10.102.40.1     0.0.0.0         UG    1024   0        0 enp2s0f1

10.102.40.0     0.0.0.0         255.255.248.0   U     0      0        0 enp2s0f1

192.168.111.0   0.0.0.0         255.255.255.0   U     0      0        0 enp1s0

192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

[jenkins@l-cheddar1 fio]$ sudo ip a s

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: enp2s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000

    link/ether a0:36:9f:35:a5:00 brd ff:ff:ff:ff:ff:ff

3: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether a0:36:9f:35:a5:02 brd ff:ff:ff:ff:ff:ff

    inet 10.102.43.86/21 brd 10.102.47.255 scope global enp2s0f1

       valid_lft forever preferred_lft forever

    inet6 fe80::a236:9fff:fe35:a502/64 scope link 

       valid_lft forever preferred_lft forever

4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 

    link/ether fe:1d:0a:e1:9d:bb brd ff:ff:ff:ff:ff:ff

    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

       valid_lft forever preferred_lft forever

5: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:25:90:07:f1:5d brd ff:ff:ff:ff:ff:ff

    inet 192.168.111.101/24 brd 192.168.111.255 scope global enp1s0

       valid_lft forever preferred_lft forever

    inet6 fe80::225:90ff:fe07:f15d/64 scope link 

       valid_lft forever preferred_lft forever

[Or Gerlitz]

does  /sys/class/net/enp1s0 point to the ConnectX PCI device?

does the mlx4_ib driver loaded?

Can you please do the following

$ yum install libmlx4
$ yum install libibverbs
$ yum install libibverbs-utils
$ yum install rdma

start the rdma service

$ modprobe -v ib_uverbs

$ ibv_devinfo -v

[Moussa Ba]

FYI the system is a supermicro based system with integrated ConnextX3 card running 2.30....firmware info is below (obtained when installing mlnx-en-2.3.1.0.0 from Mellanox website).

Device #1:

----------

  Device Type:      ConnectX3

  Part Number:      AOC-CIBF-M1

  Description:      ConnectX-3 single-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6

  PSID:             SM_1101000001000

  PCI Device Name:  0000:01:00.0

  Port1 GUID:       002590ffff07f15d

  Port2 MAC:        00259007f15e

  Versions:         Current        Available

     FW             2.30.3000      N/A

#ibv_devinfo -v

No IB devices found

lspci -vvv | grep -i Mellanox -A 40 

01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

Subsystem: Mellanox Technologies Device 0015

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 16

Region 0: Memory at f7200000 (64-bit, non-prefetchable) [size=1M]

Region 2: Memory at f5000000 (64-bit, prefetchable) [size=8M]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-

Vector table: BAR=0 offset=0007c000

PBA: BAR=0 offset=0007d000

Capabilities: [60] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited

ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-

DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-

LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited

ClockPM- Surprise- LLActRep- BwNot-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

Compliance De-emphasis: -6dB

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)

ARICap: MFVC- ACS-, Next Function: 0

ARICtl: MFVC- ACS-, Function Group: 0

Capabilities: [148 v1] Device Serial Number 00-25-90-ff-ff-07-f1-5c

Capabilities: [108 v1] Single Root I/O Virtualization (SR-IOV)

IOVCap: Migration-, Interrupt Message Number: 000

IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-

IOVSta: Migration-

Initial VFs: 63, Total VFs: 63, Number of VFs: 0, Function Dependency Link: 00

[root@l-cheddar1 fio]# ls /sys/class/net/enp1s0/device/ -l

total 0

.

.

-r--r--r--. 1 root root    4096 Oct 16 22:05 dma_mask_bits

lrwxrwxrwx. 1 root root       0 Oct 10 23:58 driver -> ../../../../bus/pci/drivers/mlx4_core

-rw-r--r--. 1 root root    4096 Oct 16 22:05 driver_override

-rw-r--r--. 1 root root    4096 Oct 16 22:05 enabled

lrwxrwxrwx. 1 root root       0 Oct 16 22:05 firmware_node -> ../../../LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:00/device:01

-r-../../../../bus/pci

-r--r--r--. 1 root root    4096 Oct 16 22:05 subsystem_device

-r--r--r--. 1 root root    4096 Oct 16 22:05 subsystem_vendor

-rw-r--r--. 1 root root    4096 Oct 10 23:36 uevent

-r--r--r--. 1 root root    4096 Oct 10 23:37 vendor

[root@l-cheddar1 fio]# 

[Or Gerlitz]

On 10/16/2014 09:14 PM, Moussa Ba wrote:
> FYI the system is a supermicro based system with integrated ConnextX3
> card running 2.30....firmware info is below (obtained when installing
> mlnx-en-2.3.1.0.0 from Mellanox website).

oops, you can't mix two driver (mlx4_core and mlx4_en) from an overlay
install of mlnx-en install with the rest of the rdma stack originated
from the inbox kernel. This will not work, AFAIK. Please remove that
package or @ least the rpm which installs these two kernel drivers (if
you need to locate it do rpm -qf on the module files and them rpm -e)

>
>
> Device #1:
> ----------
>
> Device Type: ConnectX3
> Part Number: AOC-CIBF-M1
> Description: ConnectX-3 single-port QSFP; FDR IB (56Gb/s) and
> 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
> PSID: SM_1101000001000
> PCI Device Name: 0000:01:00.0
> Port1 GUID: 002590ffff07f15d
> Port2 MAC: 00259007f15e
> Versions: Current Available
> FW 2.30.3000 N/A
>
>
>
> #ibv_devinfo -v
> No IB devices found
>

no IB device --> no RDMA

[Moussa Ba]

On Thursday, October 16, 2014 4:01:49 PM UTC-7, Or Gerlitz wrote:

On 10/16/2014 09:14 PM, Moussa Ba wrote:
> FYI the system is a supermicro based system with integrated ConnextX3
> card running 2.30....firmware info is below (obtained when installing
> mlnx-en-2.3.1.0.0 from Mellanox website).

oops, you can't mix two driver (mlx4_core and mlx4_en) from an overlay
install of mlnx-en install with the rest of the rdma stack originated
from the inbox kernel. This will not work, AFAIK. Please remove that
package or @ least the rpm which installs these two kernel drivers (if
you need to locate it do rpm -qf on the module files and them rpm -e)

The network interface only showed up after installing mlnx-en-2.3.1.0.0. What is the proper way of enabling iser/ROCE with the inbox kernel?

 

>
>
> Device #1:
> ----------
>
>   Device Type:      ConnectX3
>   Part Number:      AOC-CIBF-M1
>   Description:      ConnectX-3 single-port QSFP; FDR IB (56Gb/s) and
> 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
>   PSID:             SM_1101000001000
>   PCI Device Name:  0000:01:00.0
>   Port1 GUID:       002590ffff07f15d
>   Port2 MAC:        00259007f15e
>   Versions:         Current        Available
>      FW             2.30.3000      N/A
>
>
>
> #ibv_devinfo -v
> No IB devices found
>
no IB device --> no RDMA

Well as you can see the device does show up under lspci. Am i missing a module? 

Moussa 

[Or Gerlitz]

On 10/17/2014 02:05 AM, Moussa Ba wrote:

On Thursday, October 16, 2014 4:01:49 PM UTC-7, Or Gerlitz wrote:

On 10/16/2014 09:14 PM, Moussa Ba wrote:
> FYI the system is a supermicro based system with integrated ConnextX3
> card running 2.30....firmware info is below (obtained when installing
> mlnx-en-2.3.1.0.0 from Mellanox website).

oops, you can't mix two driver (mlx4_core and mlx4_en) from an overlay
install of mlnx-en install with the rest of the rdma stack originated
from the inbox kernel. This will not work, AFAIK. Please remove that
package or @ least the rpm which installs these two kernel drivers (if
you need to locate it do rpm -qf on the module files and them rpm -e)

The network interface only showed up after installing mlnx-en-2.3.1.0.0.

so that's a pre-condition/issue we need to solve

remove the mlx4_core and mlx4_en from ofed, reload the inbox mlx4_core with debug_level=1 and see if there are errors, you need to get into a situation where the inbox EN driver is up and running

What is the proper way of enabling iser/ROCE with the inbox kernel?

 

it should just work once the EN (mlx4_en) driver is OK and you load the IB driver (mlx4_ib)

>
>
> Device #1:
> ----------
>
>   Device Type:      ConnectX3
>   Part Number:      AOC-CIBF-M1
>   Description:      ConnectX-3 single-port QSFP; FDR IB (56Gb/s) and
> 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
>   PSID:             SM_1101000001000
>   PCI Device Name:  0000:01:00.0
>   Port1 GUID:       002590ffff07f15d
>   Port2 MAC:        00259007f15e
>   Versions:         Current        Available
>      FW             2.30.3000      N/A
>
>
>
> #ibv_devinfo -v
> No IB devices found
>
no IB device --> no RDMA

Well as you can see the device does show up under lspci. Am i missing a module?

oh I wanted to say no kernel software IB device --> no RDMA services

Moussa 

[Moussa Ba]

The problem was finally identified as being incompatibility issues between ofed drivers and inbox kernel drivers. Solution was to remove all ofed drivers and enable kernel modules only. All is good. Thank you for your support all.

Moussa

[Sagi Grimberg]

The problem was finally identified as being incompatibility issues between ofed drivers and inbox kernel drivers. Solution was to remove all ofed drivers and enable kernel modules only. All is good. Thank you for your support all.

Hey Moussa,

Sorry for the lack of response, as Or said its a holiday in Israel and I was very busy with some urgent stuff this week... I'm glad you got things working (joining the thanks to Mike and Or for assisting!).

Are you going to give a try to the RoCE MSIX sharing patch I sent a while ago? This patch won't make it upstream obviously, but some form of it will, so I would like to know it resolves the issues you were seeing with LIO performance scaling.

 Cheers,

Sagi.