iscsistart takeover vs iface names

[Ferenc Wagner]

Hi,

When booting Linux from iSCSI, the initial connections are set up like

modprobe iscsi_tcp
modprobe crc32c
modprobe iscsi_ibft
iscsistart -N
iscsistart -b

that is, all the parameters are set in the offload card via the adapter
ROM utility and the initramfs scripts read them from the iBFT.

My questions:

1. Can the iBFT contain two target specifications (one per NIC) only? I
can set up more in the adapter ROM utility.

2. The iscsistart utility starts up sessions with iface_name like
<transport>.<MAC> (be2iscsi.aa:bb:cc:dd:ee:ff in my case), but iscsid
uses iface_names like <transport>.<MAC>.ipv4.1. This leads to 6
sessions instead of 4, because iscsid does not take over the two
created by iscsistart, since the interface names are different. It's
possible to define ifaces with the simple names and override the
interface names used by iscsid, but that's a hassle and the qualified
interface names appear nevertheless. Is there an elegant way to get
both utilities use the same names? Looks like ac38eee2 introduced
this discrepancy.
--
Thanks,
Feri.

[Mike Christie]

On 10/6/15, 5:58 AM, Ferenc Wagner wrote:
> Hi,
>
> When booting Linux from iSCSI, the initial connections are set up like
>
> modprobe iscsi_tcp
> modprobe crc32c
> modprobe iscsi_ibft
> iscsistart -N
> iscsistart -b
>
> that is, all the parameters are set in the offload card via the adapter
> ROM utility and the initramfs scripts read them from the iBFT.
>

Did you implement this solution yourself or what distro are you using?

> My questions:
>
> 1. Can the iBFT contain two target specifications (one per NIC) only? I
> can set up more in the adapter ROM utility.

If there are more than one targets exported in sysfs then the iscsi
tools will try to log into them all. Some iBFT/FW implementations only
export the specific target they used though. For example, you can define
2 targets, but one is only used by the iBFT initiator for failover, and
the backup one is sometimes not exported.

>
> 2. The iscsistart utility starts up sessions with iface_name like
> <transport>.<MAC> (be2iscsi.aa:bb:cc:dd:ee:ff in my case), but iscsid

Are you using iscsi_tcp and ibft or be2iscsi or both? iscsi_tcp is for
software iscsi and you would normally boot from that driver with iBFT.
Your example above has be2iscsi which is the emulex/advogato offload
driver and that does not use ibft.

> uses iface_names like <transport>.<MAC>.ipv4.1. This leads to 6
> sessions instead of 4, because iscsid does not take over the two
> created by iscsistart, since the interface names are different. It's
> possible to define ifaces with the simple names and override the
> interface names used by iscsid, but that's a hassle and the qualified
> interface names appear nevertheless. Is there an elegant way to get
> both utilities use the same names? Looks like ac38eee2 introduced
> this discrepancy.

Yeah, sorry that is a bug in the code. Thanks for debugging.

Do you have records setup for the boot targets (did you run the iscsiadm
discovery command to the boot targets and so in /etc/iscsi/targets you
have info for them)? If you run

iscsiadm -m node -P 1

from the booted buggy system what do you see?

When the system is booted in the buggy state could you also run:

iscsiadm -m session -P 1

and send the output?

[Mike Christie]

On 10/6/15, 4:44 PM, Mike Christie wrote:
> Do you have records setup for the boot targets (did you run the iscsiadm
> discovery command to the boot targets and so in /etc/iscsi/targets you
> have info for them)? If you run
>
> iscsiadm -m node -P 1
>

Also if you are not using SUSE and if when you run the command above you
see this:

Target: your target
Portal: your_ip:port
<transport>.<MAC>.ipv4.1

Could you delete it manually

iscsiadm -m session -T your_target -p your_ip -I
<transport>.<MAC>.ipv4.1 --logout
iscsiadm -m node -T your_target -p your_ip -I <transport>.<MAC>.ipv4.1


and reboot and see if you still get the extra sessions? This is just to
confirm where the extra sessions are coming from. To fix this properly I
will need to modify the iscsistart default iface name creation, and you
will have make sure your solution loads be2iscsi before running
iscsistart if that is the driver you are using.

[Ferenc Wagner]

Mike Christie <mich...@cs.wisc.edu> writes:

> On 10/6/15, 5:58 AM, Ferenc Wagner wrote:
>
>> When booting Linux from iSCSI, the initial connections are set up like
>>
>> modprobe iscsi_tcp
>> modprobe crc32c
>> modprobe iscsi_ibft
>> iscsistart -N
>> iscsistart -b
>>
>> that is, all the parameters are set in the offload card via the adapter
>> ROM utility and the initramfs scripts read them from the iBFT.
>
> Did you implement this solution yourself or what distro are you using?

I'm using Debian jessie, and the above is from an initramfs hook script
I implemented myself.

>> My questions:
>>
>> 1. Can the iBFT contain two target specifications (one per NIC) only? I
>> can set up more in the adapter ROM utility.
>
> If there are more than one targets exported in sysfs then the iscsi
> tools will try to log into them all. Some iBFT/FW implementations only
> export the specific target they used though. For example, you can
> define 2 targets, but one is only used by the iBFT initiator for
> failover, and the backup one is sometimes not exported.

It's actually a dual-port card. I set up two targets for each port, but
in sysfs I can only see one for each. I don't know whether the adapter
ROM logged in to both or not, though that wouldn't make much sense, as I
doubt it did multipathing.

>> 2. The iscsistart utility starts up sessions with iface_name like
>> <transport>.<MAC> (be2iscsi.aa:bb:cc:dd:ee:ff in my case), but iscsid
>
> Are you using iscsi_tcp and ibft or be2iscsi or both? iscsi_tcp is for
> software iscsi and you would normally boot from that driver with
> iBFT. Your example above has be2iscsi which is the emulex/advogato
> offload driver and that does not use ibft.

I don't use iscsi_tcp, that's left in inadvertently, sorry for creating
confusion. I use a dual port Emulex offload card. I thought I needed
iscsi_ibft to get iscsistart read out the target parameters from sysfs,
but if it's not needed, then I'll remove that as well. Thanks for
clearing this up.

>> uses iface_names like <transport>.<MAC>.ipv4.1. This leads to 6
>> sessions instead of 4, because iscsid does not take over the two
>> created by iscsistart, since the interface names are different. It's
>> possible to define ifaces with the simple names and override the
>> interface names used by iscsid, but that's a hassle and the qualified
>> interface names appear nevertheless. Is there an elegant way to get
>> both utilities use the same names? Looks like ac38eee2 introduced
>> this discrepancy.
>
> Yeah, sorry that is a bug in the code. Thanks for debugging.

Well then, if it's possible to fix with a straightforward patch, which
can be backported to 2.0.873 (3b4b4500, more precisely), it could be
possible to include the fix in a stable update. What do you think?

> Do you have records setup for the boot targets (did you run the
> iscsiadm discovery command to the boot targets and so in
> /etc/iscsi/targets you have info for them)?

I did not run discovery, I added the nodes by hand after boot.

> If you run
> iscsiadm -m node -P 1
> from the booted buggy system what do you see?

I've already defined interfaces with the iscsistart names, so it's

Target: iqn.2000-02.hu.niif:dx90.pc
Portal: 10.0.3.1:3260,-1
Iface Name: be2iscsi.90:1b:0e:23:cc:51
Portal: 10.0.3.2:3260,-1
Iface Name: be2iscsi.90:1b:0e:23:cc:51
Portal: 10.0.4.1:3260,-1
Iface Name: be2iscsi.90:1b:0e:23:cc:55
Portal: 10.0.4.2:3260,-1
Iface Name: be2iscsi.90:1b:0e:23:cc:55

I'm installing a new system with similar hardware, and will send you
output from all (even buggy) stages. This is what I have in the
/etc/iscsi/ifaces directory:

be2iscsi.90:1b:0e:23:cc:51 be2iscsi.90:1b:0e:23:cc:55
be2iscsi.90:1b:0e:23:cc:51.ipv4.0 be2iscsi.90:1b:0e:23:cc:55.ipv4.0
be2iscsi.90:1b:0e:23:cc:51.ipv6.0 be2iscsi.90:1b:0e:23:cc:55.ipv6.0

I added the short ones by hand, and the long ones are always recreated
if I delete them.

> When the system is booted in the buggy state could you also run:
> iscsiadm -m session -P 1
> and send the output?

I will. But I remember pretty well: SID 1 used Iface Name:
be2iscsi.90:1b:0e:23:cc:51, SID 2 be2iscsi.90:1b:0e:23:cc:55. Without
explicit iface configuration, iscsiadm only allowed the long iface names
above, and then created SID 3-6 with them when I logged in to two
portals from each. Multipathd then created two failover groups with 3-3
paths in each.

> Also if you are not using SUSE and if when you run the command above
> you see this:
>
> Target: your target
> Portal: your_ip:port
> <transport>.<MAC>.ipv4.1
>
> Could you delete it manually
>
> iscsiadm -m session -T your_target -p your_ip -I <transport>.<MAC>.ipv4.1 --logout
> iscsiadm -m node -T your_target -p your_ip -I <transport>.<MAC>.ipv4.1
>
> and reboot and see if you still get the extra sessions?

In a pristine system (iscsistart only, no targets configured by
iscsiadm) I only get two sessions (with short interfaces names). With
an older open-iscsi version I could then add the four needed node, and
after login iscsid took over the two from iscsistart and also started
the two other. Now, that iscsid uses long interface names, it creates
four new sessions and leaves the two from iscsistart unmanaged. Than
is, unless I define interfaces with the old (short) names and use them
when adding the nodes with iscsiadm.

> This is just to confirm where the extra sessions are coming from. To
> fix this properly I will need to modify the iscsistart default iface
> name creation, and you will have make sure your solution loads
> be2iscsi before running iscsistart if that is the driver you are
> using.

Sounds sweet, thanks! The be2iscsi module seems to be loaded
automatically early during the boot (initramfs phase), probably by udev.
But I'll load it explicitly if needed.
--
Regards,
Feri.

[Mike Christie]

On 10/7/15, 1:37 AM, Ferenc Wagner wrote:
> In a pristine system (iscsistart only, no targets configured by
> iscsiadm) I only get two sessions (with short interfaces names). With
> an older open-iscsi version I could then add the four needed node, and
> after login iscsid took over the two from iscsistart and also started
> the two other. Now, that iscsid uses long interface names, it creates
> four new sessions and leaves the two from iscsistart unmanaged. Than

How do you know they are unmanaged by iscsid? Is the iscsid session
state in the iscsiadm -m session -P 1 info showing:

Internal iscsid Session State: Unknown

[Ferenc Wagner]

Maybe I was wrong on this point. After an iscsistart -b in the
initramfs, but with no ifaces or nodes configured later:

# iscsiadm -m session -P1
Target: iqn.2000-02.hu.niif:dx90.pc (non-flash)
Current Portal: 10.0.3.2:3260,3
Persistent Portal: 10.0.3.2:3260,3
**********
Interface:
**********
Iface Name: be2iscsi.90:1b:0e:23:ce:a9
Iface Transport: be2iscsi
Iface Initiatorname: iqn.2000-02.hu.niif:vhbl04.pc
Iface IPaddress: <empty>
Iface HWaddress: 90:1b:0e:23:ce:a9
Iface Netdev: <empty>
SID: 1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
Current Portal: 10.0.4.1:3260,2
Persistent Portal: 10.0.4.1:3260,2
**********
Interface:
**********
Iface Name: be2iscsi.90:1b:0e:23:ce:ad
Iface Transport: be2iscsi
Iface Initiatorname: iqn.2000-02.hu.niif:vhbl04.pc
Iface IPaddress: <empty>
Iface HWaddress: 90:1b:0e:23:ce:ad
Iface Netdev: <empty>
SID: 2
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE

on the two sessions created by iscsistart (and there are no other
sessions yet). But I can't use the above Iface Name values:

# iscsiadm -m node -T iqn.2000-02.hu.niif:dx90.pc -I be2iscsi.90:1b:0e:23:ce:a9 -p 10.0.3.1 -o new
iscsiadm: Could not read iface info for be2iscsi.90:1b:0e:23:ce:a9.

Otherwise, the above command creates the default suffixed ifaces:

# ls /etc/iscsi/ifaces/
be2iscsi.90:1b:0e:23:ce:a9.ipv4.0 be2iscsi.90:1b:0e:23:ce:ad.ipv4.0
be2iscsi.90:1b:0e:23:ce:a9.ipv6.0 be2iscsi.90:1b:0e:23:ce:ad.ipv6.0
--
Regards,
Feri.

[Mike Christie]

On 10/08/2015 04:10 PM, Ferenc Wagner wrote:
> Mike Christie <mich...@cs.wisc.edu> writes:
>
>> > On 10/7/15, 1:37 AM, Ferenc Wagner wrote:
>> >
>>> >> In a pristine system (iscsistart only, no targets configured by
>>> >> iscsiadm) I only get two sessions (with short interfaces names). With
>>> >> an older open-iscsi version I could then add the four needed node, and
>>> >> after login iscsid took over the two from iscsistart and also started
>>> >> the two other. Now, that iscsid uses long interface names, it creates
>>> >> four new sessions and leaves the two from iscsistart unmanaged. Than
>> >
>> > How do you know they are unmanaged by iscsid? Is the iscsid session
>> > state in the iscsiadm -m session -P 1 info showing:
>> >
>> > Internal iscsid Session State: Unknown
> Maybe I was wrong on this point. After an iscsistart -b in the
> initramfs, but with no ifaces or nodes configured later:

No problem. I think I fully understand the problem. I am working on a
patch. I messed up the firmware on my card, so I cannot access the boot
setup any more here, but am looking for one in the red hat lab.

[The Lee-Man]

Mike: I know you've been busy. Any progress on this? Can I help, as I have the same issue?

On Thursday, October 8, 2015 at 2:48:30 PM UTC-7, Mike Christie wrote:

On 10/08/2015 04:10 PM, Ferenc Wagner wrote:

[Mike Christie]

I think we can do something like the attached compile tested only patch
which has the boot code use the same iface name format as the normal setup.

My concern was compat support. The patch will use whatever the kernel
supports, and so it should just work for boot if you are using
iscsistart or "iscsiadm -m fw -l", but I am not sure if during a upgrade
things will work with mismatch of old and new kernel and tools and old
stored /etc/ifaces and how distros were using the tools.

I got busy with day time work stuff, and I killed my card (updated the
fw and now it only supports networking instead of iscsi and I cannot
covert back) so I was not able to go over all the combos. If you can
test it out it and think of the SUSE upgrade scenarios you guys support
it would help.

On 05/27/2016 01:13 PM, The Lee-Man wrote:
> Mike: I know you've been busy. Any progress on this? Can I help, as I
> have the same issue?
>
> On Thursday, October 8, 2015 at 2:48:30 PM UTC-7, Mike Christie wrote:
>
> On 10/08/2015 04:10 PM, Ferenc Wagner wrote:

> > Mike Christie <m*@cs.wisc.edu <mailto:mich...@cs.wisc.edu>> writes:
> >
> >> > On 10/7/15, 1:37 AM, Ferenc Wagner wrote:
> >> >
> >>> >> In a pristine system (iscsistart only, no targets configured by
> >>> >> iscsiadm) I only get two sessions (with short interfaces
> names). With
> >>> >> an older open-iscsi version I could then add the four needed
> node, and
> >>> >> after login iscsid took over the two from iscsistart and also
> started
> >>> >> the two other. Now, that iscsid uses long interface names,
> it creates
> >>> >> four new sessions and leaves the two from iscsistart
> unmanaged. Than
> >> >
> >> > How do you know they are unmanaged by iscsid? Is the iscsid
> session
> >> > state in the iscsiadm -m session -P 1 info showing:
> >> >
> >> > Internal iscsid Session State: Unknown
> > Maybe I was wrong on this point. After an iscsistart -b in the
> > initramfs, but with no ifaces or nodes configured later:
>
> No problem. I think I fully understand the problem. I am working on a
> patch. I messed up the firmware on my card, so I cannot access the boot
> setup any more here, but am looking for one in the red hat lab.
>

> --
> You received this message because you are subscribed to the Google
> Groups "open-iscsi" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to open-iscsi+...@googlegroups.com
> <mailto:open-iscsi+...@googlegroups.com>.
> To post to this group, send email to open-...@googlegroups.com
> <mailto:open-...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/open-iscsi.
> For more options, visit https://groups.google.com/d/optout.

[The Lee-Man]

Hi Mike:

I tested your patch, and it fixes the issue I was seeing with Emulex cards
using the be2iscsi driver.

I thought about the upgrade implications while testing on this system,
and I discussed my results with Hannes. It looks like there should
not be an upgrade issue. Instead, a system being upgraded with
be2iscsi would end up with three interface files for each interface:
the original interface file, named:

* be2iscsi.MAC_ADDR          -- the original interface file
* be2iscsi.MAC_ADDR.ipv4.0 -- a new interface file for IPv4
* be2iscsi.MAC_ADDR.ipv6.0 -- a new interface file for IPv6

So I would say go ahead and submit this patch. :)

On Friday, May 27, 2016 at 2:46:31 PM UTC-7, Mike Christie wrote:
I think we can do something like the attached compile tested only patch
which has the boot code use the same iface name format as the normal setup.

My concern was compat support. The patch will use whatever the kernel
supports, and so it should just work for boot if you are using
iscsistart or "iscsiadm -m fw -l", but I am not sure if during a upgrade
things will work with mismatch of old and new kernel and tools and old
stored /etc/ifaces and how distros were using the tools.

I got busy with day time work stuff, and I killed my card (updated the
fw and now it only supports networking instead of iscsi and I cannot
covert back) so I was not able to go over all the combos. If you can
test it out it and think of the SUSE upgrade scenarios you guys support
it would help.


On 05/27/2016 01:13 PM, The Lee-Man wrote:
> Mike: I know you've been busy. Any progress on this? Can I help, as I
> have the same issue?
>
> On Thursday, October 8, 2015 at 2:48:30 PM UTC-7, Mike Christie wrote:
>
> On 10/08/2015 04:10 PM, Ferenc Wagner wrote:

Hi Mike:

I tested your patch, and it fixes the issue I was seeing with Emulex cards
using the be2iscsi driver.

I thought about the upgrade implications while testing on this system,
and I discussed my results with Hannes. It looks like there should
not be an upgrade issue. Instead, a system being upgraded with
be2iscsi would end up with three interface files for each interface:
the original interface file, named:

  * be2iscsi.MAC_ADDR            -- the original interface file
  * be2iscsi.MAC_ADDR.ipv4.0   -- a new interface file for IPv4
  * be2iscsi.MAC_ADDR.ipv6.0  -- a new interface file for IPv6

On Friday, May 27, 2016 at 2:46:31 PM UTC-7, Mike Christie wrote:

I think we can do something like the attached compile tested only patch
which has the boot code use the same iface name format as the normal setup.

My concern was compat support. The patch will use whatever the kernel
supports, and so it should just work for boot if you are using
iscsistart or "iscsiadm -m fw -l", but I am not sure if during a upgrade
things will work with mismatch of old and new kernel and tools and old
stored /etc/ifaces and how distros were using the tools.

I got busy with day time work stuff, and I killed my card (updated the
fw and now it only supports networking instead of iscsi and I cannot
covert back) so I was not able to go over all the combos. If you can
test it out it and think of the SUSE upgrade scenarios you guys support
it would help.


On 05/27/2016 01:13 PM, The Lee-Man wrote:
> Mike: I know you've been busy. Any progress on this? Can I help, as I
> have the same issue?
>
> On Thursday, October 8, 2015 at 2:48:30 PM UTC-7, Mike Christie wrote:
>
>     On 10/08/2015 04:10 PM, Ferenc Wagner wrote:

> an email to open-iscsi+unsubscribe@googlegroups.com
> <mailto:open-iscsi+unsub...@googlegroups.com>.

> To post to this group, send email to open-...@googlegroups.com

> <mailto:open-iscsi@googlegroups.com>.

[Christian Seiler]

Hi,

On 06/01/2016 11:35 PM, Mike Christie wrote:
> Ccing Christian and dropping list to make sure the mail does not get
> lost in his open-iscsi filters.

It's actually the other way around: I get emails from the list just
fine, but in the end I resorted to creating a gmail account to be able
to post to the list, because emails from my real email address were
just rejected by the list for no apparent reason. (cc'ing the list
again.)

> How does debian implement iscsi boot?
>
> Does it:
> 1. Run iscsistart for the root sessions/targets in the initramfs and
> then start iscsid during the normal startup procedure and run iscsiadm
> at that time for other non boot/root related sessions?

Yes. You can find the current code used by Debian (and by extension
Ubuntu) in the initramfs here:
http://sources.debian.net/src/open-iscsi/2.0.873%2Bgit0.3b4b4500-15/debian/extra/initramfs.local-top/

Basically, there are 3 ways you can boot from iSCSI at the moment in
Debian (via the officially supported hook to initramfs-tools):

1. You create a special file /etc/iscsi/iscsi.initramfs (source by the
shell in the initramfs) with a couple of settings (ISCSI_TARGET_IP=...,
ISCSI_USERNAME=..., etc.). That file is copied into the initramfs
image, which then calls iscsistart. If you install Debian on iSCSI via
the official installer, it will create this file with the corresponding
settings.

2. You create the file /etc/iscsi/iscsi.initramfs with the setting
ISCSI_AUTO=true, then the initramfs will use iscsistart -f to auto-
configure the iSCSI sessions. (And also iscsistart -N to configure the
the network interfaces, if required.)

3. This has been added very recently: root=iscsi:... (RFC 4173 syntax)
is now also supported, there you don't need to create any special file,
just have the package installed. In that case, also iscsistart will be
used.

In no case is the iSCSI database or iscsid used in the initramfs at
the moment. (That means that you can use the iscsi.initramfs file to
configure a session separate from the iscsi database btw.)

(Note that dracut is available as an alternative in Debian, and it
brings its own iSCSI support. I haven't gotten around to testing that
so far, though; but I'm pretty sure it won't work, because dracut does
start iscsid in the initramfs since fall of last year IIRC, but due to
Debian policies we install the iSCSI database directly in /etc/iscsi
instead of /var/lib/..., whereas IIRC the dracut code tries to copy the
database from the official upstream location. Making sure dracut also
works in Debian with iSCSI is on my TODO list for Debian 9.)

The system startup code (once the root file system is mounted) is
independent of the initramfs and boils down to basically the following
two steps:

1. start iscsid
2. iscisadm -m node --loginall=automatic

If a session is configured both in the database as well as the
iscsi.initramfs file, iscsiadm will display a warning and return exit
code 15 (already logged in), but we use SuccessExitStatus in our
systemd service file, so it's not treated as an error.

(The shutdown logic is more involved, because we try to detect which
sessions are used for the root and /usr filesystems, if any, and only
shut down all the _other_ sessions. But we also allow the admins to
override that and not shutdown any session at all, in case they have a
storage configuration that we don't properly detect.)

> 2. Run iscsiadm and iscsid in the initramfs, stop iscsid, and then
> restart it and maybe run iscsiadm again for non root/boot related
> sessions/targets during the normal startup procedures?

We have a low-priority TODO item to think about maybe (perhaps
optionally) supporting using iscsid in the initramfs, but haven't
gotten around to that so far.

Hope that answers your question.

That all said, we're not married to the internals of the current
approach, so if there were to be good reasons to do something
differently, so for example if you decided to drop iscsistart upstream
and say the only supported way is to have iscsid running in the
initramfs, we'd be open to that, as long as we can support all the
potential previous use cases with it.

Regards,
Christian