trying to run open-iscsi in an lxc container on suse

[pmullaney]

Hi all,

Running into problems running under lxc. I am running iscsid in the
container and when it
attempts to connect to the netlink iscsi socket it is getting a
connection refused(111) and
the following in the log:

sendmsg: bug? ctrl_fd 5

which causes iscsid to exit.

Any thoughts on what could be wrong?

Thanks.

[Mike Christie]

The iscsi netlink code never returns 111/ECONNREFUSED.

It looks like the netlink code will return this when the netlink socket
is not setup.

What kernel are you using? What iscsi modules? Did they come with your
kernel?

[Patrick Mullaney]

I am using sles11sp1. My kernel is 2.6.32.49-0.3-default.

I tried the packages that came with sles11sp1 as a start and ran into the issue. I am currently using
my own build of open-iscsi-2.0-872 and I have tried building from head of the git repo,
both with the same result.

I have been able to get this to work under the same kernel and versions on the host machine as
opposed to in the lxc container. I suspect that the af_netlink code is refusing the connection because
the connection is initiated in the container(perhaps its a capabilities issue but that is just a guess at this point).

[Mike Christie]

On 12/28/2011 04:10 PM, Patrick Mullaney wrote:
> I am using sles11sp1. My kernel is 2.6.32.49-0.3-default.
>
> I tried the packages that came with sles11sp1 as a start and ran into the
> issue. I am currently using
> my own build of open-iscsi-2.0-872 and I have tried building from head of
> the git repo,
> both with the same result.
>
> I have been able to get this to work under the same kernel and versions on
> the host machine as
> opposed to in the lxc container. I suspect that the af_netlink code is
> refusing the connection because
> the connection is initiated in the container(perhaps its a capabilities
> issue but that is just a guess at this point).

Yeah, sounds like a reasonable guess. I am not exactly sure what is in
the SLES kernel. Could you run a patch with some debugging, so we can
check where exactly it is failing in the netlink code.

What version of lxc are you using? Does it come with SLES or did you
download it from sourceforge?

[Patrick Mullaney]

I ran the stock sles lxc(0.6.5), tried the latest on sourceforge and the most recent stable - all
had the same result. I was just about to try to put some debug code in the netlink
code. So I'd be interested in any patch you would be nice enough to send - thanks!

[Mike Christie]

On 12/29/2011 08:08 AM, Patrick Mullaney wrote:
> I ran the stock sles lxc(0.6.5), tried the latest on sourceforge and the
> most recent stable - all
> had the same result. I was just about to try to put some debug code in the
> netlink
> code. So I'd be interested in any patch you would be nice enough to send -
> thanks!
>

Could you send me the commands you are running?

[Patrick Mullaney]

Sure - here is a run with debug level 7 on from within a container:

tenant199-vm3:/open-iscsi-2.0-

872 # /sbin/iscsid -f -d 7 -c /etc/iscsi/iscsid.conf &
[1] 1508
tenant199-vm3:/open-iscsi-2.0-872 # iscsid: sysfs_init: sysfs_path='/sys'

iscsid: sysfs_attr_get_value: open '/module/scsi_transport_iscsi'/'version'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/module/scsi_transport_iscsi/version'

iscsid: sysfs_attr_get_value: add to cache '/sys/module/scsi_transport_iscsi/version'

iscsid: sysfs_attr_get_value: cache '/sys/module/scsi_transport_iscsi/version' with attribute value '2.0-871'

iscsid: transport class version 2.0-871. iscsid version 2.0-872
iscsid: InitiatorName=iqn.1996-04.de.suse:01:c722a7ebfb5a
iscsid: InitiatorName=iqn.1996-04.de.suse:01:c722a7ebfb5a
iscsid: InitiatorAlias=tenant199-vm3
iscsid: Max file limits 1024 8192

iscsid: found 192.168.58.40,3260

iscsid: Looking for config file /etc/iscsi/send_targets/192.168.58.40,3260

iscsid: Looking for config file /etc/iscsi/send_targets/192.168.58.40,3260 config st_config.
iscsid: st_start 192.168.58.40:3260 0
tenant199-vm3:/open-iscsi-2.0-872 # iscsiadm -m node -L all -d 7iscsiadm: Max file limits 1024 8192

iscsiadm: searching iqn.1992-04.com.emc:500009730005fda4

iscsiadm: found 192.168.58.40,3260,1

iscsiadm: iface iter found default.
Logging in to [iface: default, target: iqn.1992-04.com.emc:500009730005fda4, portal: 192.168.58.40,3260]
iscsid: poll result 1
iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'handle'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/tcp/handle'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/tcp/handle'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/tcp/handle' with attribute value '18446744072104513120'

iscsid: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'caps'

iscsid: sysfs_attr_get_value: new uncached attribute '/sys/class/iscsi_transport/tcp/caps'

iscsid: sysfs_attr_get_value: add to cache '/sys/class/iscsi_transport/tcp/caps'

iscsid: sysfs_attr_get_value: cache '/sys/class/iscsi_transport/tcp/caps' with attribute value '0x39'

iscsid: Matched transport tcp

iscsid: Allocted session 0x6d4f80
iscsid: no authentication configured...
iscsid: resolved 192.168.58.40 to 192.168.58.40
iscsid: setting iface default, dev , set ip , hw , transport tcp.

iscsid: set TCP recv window size to 524288, actually got 262142
iscsid: set TCP send window size to 524288, actually got 262142
iscsid: connecting to 192.168.58.40:3260
iscsid: Setting login timer 0x6da448 timeout 15
iscsid: connected local port 36508 to 192.168.58.40:3260
iscsid: sendmsg:
iscsid: sendmsg: bug? ctrl_fd 4 errno 111
iscsiadm: got read error (0/0), daemon died?
iscsiadm: Could not login to [iface: default, target: iqn.1992-04.com.emc:500009730005fda4, portal: 192.168.58.40,3260].
iscsiadm: initiator reported error (18 - could not communicate to iscsid)
[1]+  Exit 255                /sbin/iscsid -f -d 7 -c /etc/iscsi/iscsid.conf
tenant199-vm3:/open-iscsi-2.0-872 #

[Mike Christie]

On 12/28/2011 02:36 AM, Mike Christie wrote:
> On 12/22/2011 04:44 PM, pmullaney wrote:
>> Hi all,
>>
>> Running into problems running under lxc. I am running iscsid in the
>> container and when it
>> attempts to connect to the netlink iscsi socket it is getting a
>> connection refused(111) and
>> the following in the log:
>>
>> sendmsg: bug? ctrl_fd 5
>>
>> which causes iscsid to exit.
>>
>> Any thoughts on what could be wrong?
>>
>
> The iscsi netlink code never returns 111/ECONNREFUSED.
>
> It looks like the netlink code will return this when the netlink socket
> is not setup.
>

Ok, I think we need to modify the iscsi netlink socket to support
multiple namespaces.

Have not been able to get this working in fedora 16 exactly, so this is
just a guess based on looking at the code and git commits.

[Mike Christie]

Here is a patch that should implement the needed functionality on the
netlink side. It is not optimal though, but should give us an idea if we
are on the right track.

[Patrick Mullaney]

I tried this patch and iscsid ends up in a tightloop in the kernel. I attempted some
basic debugging and it definitely gets beyond my previous issue but then locks.

I also attempted to change the patch around such that the netlink socket that
gets create per net namespace is kept with the namespace and accessed off
that structure. I wanted to avoid the locking. While this works, it appears to
crash elsewhere sometime after nlmsg_multicast(no stacktrace gets generated
unfortunately).

I had some general questions after doing this. Is the traversal of and send on the per net
namespace sockets needed? I'm new to the code, but it seems like these might need
to be isolated from one another?

[Mike Christie]

On 01/11/2012 01:14 PM, Patrick Mullaney wrote:
> I tried this patch and iscsid ends up in a tightloop in the kernel. I
> attempted some
> basic debugging and it definitely gets beyond my previous issue but then
> locks.
>
> I also attempted to change the patch around such that the netlink socket
> that
> gets create per net namespace is kept with the namespace and accessed off
> that structure. I wanted to avoid the locking. While this works, it appears
> to
> crash elsewhere sometime after nlmsg_multicast(no stacktrace gets generated
> unfortunately).
>
> I had some general questions after doing this. Is the traversal of and send
> on the per net
> namespace sockets needed? I'm new to the code, but it seems like these
> might need
> to be isolated from one another?
>

I actually thought that we only wanted to send to specific sockets, but
I was looking at some other patches that added namespace support and
they did the loop so that is why I ended up adding it. I need to do some
more investigation.

[Mike Christie]

Oh yeah, I thought sending to all the sockets would be ok because iscsid
would end up figuring things out. That is why in the other mail I was
saying it would not be efficient but hopefully would get us working.

Do you know lxc though do we need the iscsi sysfs code to be per namespace?

[Patrick Mullaney]

I didn't know that a iscsi sysfs per namespace would be required but it doesn't
surprise me now that you mention it. Right now, I have access to all iscsi
sysfs data from any lxc container - would this suffice to just get the iscsi
initiator operational in a container?

[jha...@redmind.ca]

I am reviving this rather old thread (and breaking all manner of etiquette, I'm sure) to ask if anyone knows the status of getting the iSCSI Netlink code namespace aware?  I'm happy to try "experimental" patches, aelnd help QA things as having iSCSI support from inside an Linux Container (LXC) would be more than ideal for me.

FWIW, there's an open bug against the 'kernel' and 'lxc' packages on Ubuntu's launchpad (https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1226855).

I'm currently running Ubuntu 14.04.1, open-iscsi 2.0.873-3ubuntu9, kernel 3.13.0-39-generic, and LXC 1.0.6-0ubuntu0.1.

Any and all hints, tips or suggestions welcome.

./JRH