[PATCH] export session & conn handles
Hannes Reinecke
now that session and connection handles are 64bit safe we could as well
export them via sysfs.
With this information we can attempt to startup iscsid in a defined
state, ie either tear down any existing connection or do a session resync.
The latter is quite desperately needed for root on iSCSI ...
Oh, and there's a compilation error, spinlock_irq etc should be called
with the correct parameters.
Please apply.
Cheers,
Hannes
--
Dr. Hannes Reinecke ha...@suse.de
SuSE Linux Products GmbH S390 & zSeries
Maxfeldstraße 5 +49 911 74053 688
90409 Nürnberg http://www.suse.de
Mike Christie
> Hi all,
>
> now that session and connection handles are 64bit safe we could as well
> export them via sysfs.
> With this information we can attempt to startup iscsid in a defined
> state, ie either tear down any existing connection or do a session resync.
> The latter is quite desperately needed for root on iSCSI ...
Are you working on boot BTW. I have started this...
>
> Oh, and there's a compilation error, spinlock_irq etc should be called
> with the correct parameters.
>
What the heck? I should not be allowed to merge patches in the middle of
the night :)
> Please apply.
I applied the locking fixes.
> .param = ISCSI_PARAM_OFMARKER_EN,
> .value = &zero,/* FIXME: session->ofmarker_en */
> .conn_only = 0,
> + }, {
> + .param = ISCSI_PARAM_SESSION_HANDLE,
> + .value = &zero,
> + .conn_only = 0,
> + }, {
> + .param = ISCSI_PARAM_CONN_HANDLE,
> + .value = &zero,
> + .conn_only = 0,
> }
>
For the other part of the patch, what is the proper handling of
userspace calling into the kernel to set these values? iscsi_tcp ignores
it and always returns zero so we luck out, but do we even want to call
set_param for those values here or ever?
Mike Christie
>
> Hannes Reinecke wrote:
>
>> Hi all,
>>
>> now that session and connection handles are 64bit safe we could as well
>> export them via sysfs.
>> With this information we can attempt to startup iscsid in a defined
>> state, ie either tear down any existing connection or do a session
>> resync.
>> The latter is quite desperately needed for root on iSCSI ...
>
>
> Are you working on boot BTW. I have started this...
>
Oh yeah, I hit the send button too soon. I guess some of the problems or
open-isues we have had are:
1. DB is too heavy wieght for just the boot part.
2. The daemon is consider by some to be overkill.
3. How to tell the program run in the initramfs what target to boot from.
4. I am lazy and do not want to rewrite iscsid and iscsiadm for a
special boot program and try to port fixes bacj and forth.
I started to do try to just run this from the initramfs:
load modules and start network stuff
# iscsid
# in a script loop, through some discovery addresses defined in a text
file like
/etc/sysconfig/iscsi
# iscsiadm -discover target but do not write to DB
# iscsiadm -login
# kill iscsid (add iscsiadm -shutdown_daemon option)
# mount real root ....
Then when we are in the initd scripts, we can restart iscsid. When
mkinitramfs is run /etc/sysconfig/iscsi and /etc/iscsid.conf can be
brought into the initramfs.
But it appears running iscsid and iscsiadm are still too much of hassle
becuase they require the DB code even though we are not going to use it
and running iscsid as a daemon is overkill. So I have been trying to
reorg the userspace code in a way that I could write a small program
that could reuse the existing code. In the initramfs I could do:
# in a script loop through some targets definitions in a text file like
/etc/sysconfig/iscsi
# iscsiboot -setup target
# mount real root ....
In this case /etc/sysconfig/iscsi would contain the actual target info
instead of the disocery info so we could slim down what we need even
more. Is it better to do discovery here though? I guess some people have
problems with this too. Is this overkill when we can just use iscsid and
iscsiadm already.
Is there any preferences or what have you been trying?
Doug Maxey
Excellent suggestion!
Not having DB code required in the initramfs will certainly cause less
indigestion. A script or even a small binary would be win.
I would suspect most administrators would already have to enable a
target to talk to the initiator. Adding the target info at install
time would be a plus. patmans has done some work in that area.
One other point, the method to get there from POWER systems will be
essentially different that what is used for IAxx. Need to convince OFW
where to find the boot disk. Same information, different source...
Dan Aloni
>
> # in a script loop through some targets definitions in a text file like
> /etc/sysconfig/iscsi
> # iscsiboot -setup target
> # mount real root ....
>
> In this case /etc/sysconfig/iscsi would contain the actual target info
> instead of the disocery info so we could slim down what we need even
> more. Is it better to do discovery here though? I guess some people have
> problems with this too. Is this overkill when we can just use iscsid and
> iscsiadm already.
That would be cool, if you'd be able to operate open-iscsi in a
daemon-less mode. I am willing to contribute the changes to make
that work.
Something like:
# iscsi-up [target info] [connection parameters]
Block until time-out or the full feature phase
# iscsi-down [session id]
I guess that should work in the case where the daemon is only
required for authentication during connection establishment.
--
Dan Aloni
da...@monatomic.org, da...@colinux.org, da...@gmx.net, d...@xiv.co.il
Mike Christie
Yeah, I think the major nasty problem with running daemonless is what to
do when the kernel wants to fire off a connection error. So for boot
between initramfs time and "/etc/initd/iscsi start" we are in a pickle,
if iscsi is your root disk and a command times out, the scsi-eh fires,
and we end up having to go to the iscsi_tcp host reset function can
either fail or hang :)
Patrick Mansfield
> I would suspect most administrators would already have to enable a
> target to talk to the initiator. Adding the target info at install
> time would be a plus. patmans has done some work in that area.
I don't store anything special with the Anconda patches I posted, it just
starts iscsid, runs send targets, and does a login. But it would not be
hard to modify it to also store the target IP address (and any other
relevant settings) into /etc/sysconfig/iscsi for use by mkinitrd (or iscsi
startup scripts).
> One other point, the method to get there from POWER systems will be
> essentially different that what is used for IAxx. Need to convince OFW
> where to find the boot disk. Same information, different source...
You mean parse in initrd dynamically determine the iSCSI root device based
on firmware settings?
IMHO ... we shouldn't do that, especially base on boot (not root) data.
-- Patrick Mansfield
Patrick Mansfield
> Oh yeah, I hit the send button too soon. I guess some of the problems or
> open-isues we have had are:
>
> 1. DB is too heavy wieght for just the boot part.
Can DB usage in iscsid be replaced with normal files (or such)? Then this
initrd/initramfs work (at least long term) is easier, and maybe we can
have an iscsi object library and/or have iscsid options for use in initrd.
> I started to do try to just run this from the initramfs:
>
> load modules and start network stuff
> # iscsid
Did that run? Did you include db4 shared libs in initrams? I could not
get a static link with db4 to work (with fc devel on ppc).
> # in a script loop through some targets definitions in a text file like
> /etc/sysconfig/iscsi
> # iscsiboot -setup target
> # mount real root ....
Yes, sounds good.
-- Patrick Mansfield
Mike Christie
> On Tue, Jan 24, 2006 at 04:07:35PM -0600, Mike Christie wrote:
>
>
>>Oh yeah, I hit the send button too soon. I guess some of the problems or
>>open-isues we have had are:
>>
>>1. DB is too heavy wieght for just the boot part.
>
>
> Can DB usage in iscsid be replaced with normal files (or such)? Then this
> initrd/initramfs work (at least long term) is easier, and maybe we can
> have an iscsi object library and/or have iscsid options for use in initrd.
>
>
>>I started to do try to just run this from the initramfs:
>>
>>load modules and start network stuff
>># iscsid
>
>
> Did that run? Did you include db4 shared libs in initrams? I could not
> get a static link with db4 to work (with fc devel on ppc).
>
I actually cheated and just ripped it out and simulated (by hardcoding)
the stuff I needed. I was just trying to figure out what needed to be
done to get an idea of how long the work might take.
>
>># in a script loop through some targets definitions in a text file like
>>/etc/sysconfig/iscsi
>># iscsiboot -setup target
>># mount real root ....
>
>
> Yes, sounds good.
>
Bah! The harder option :)
Mike Christie
If we are going to just export these values from sysfs and have the
transport class set them like in your patch, we should probably make
them private attrs like what the FC class uses.
Dan Aloni
Perhaps it would be possible in the scsi_transport_iscsi level to detect
whether the daemon is connected and if it is not connected replace it
with some in-kernel stub that would allow safe sort of execution until
iscsid comes back to life. I wonder if netlink even provides the ability
to know if the daemon is there.
Mike Christie
We should get a error value like -ECONNREFUSED, when we call
netlink_unicast().
Doug Maxey
On Tue, 24 Jan 2006 14:39:25 PST, Patrick Mansfield wrote:
>
>On Tue, Jan 24, 2006 at 04:18:44PM -0600, Doug Maxey wrote:
>
>> I would suspect most administrators would already have to enable a
>> target to talk to the initiator. Adding the target info at install
>> time would be a plus. patmans has done some work in that area.
>
>I don't store anything special with the Anconda patches I posted, it just
>starts iscsid, runs send targets, and does a login. But it would not be
>hard to modify it to also store the target IP address (and any other
>relevant settings) into /etc/sysconfig/iscsi for use by mkinitrd (or iscsi
>startup scripts).
On power, the current scheme is to treat all iscsi boots as SWI. The
parameters used would be available in the device-tree, both from
bootdevice, and another, to be named property. Could be extracted from
sysfs provided by the driver for the HBA boot case, but something would
need to assist obtaining what used to be available in the parameter
block (on IA32).
The 4.x version used to look for the "...PXE" string in the first 1k of
kernel memory. Have not looked at the latest, was hoping that the data
fairy would come along and fix it. Finding the data in kernel memory
will not exist on power in any event. The _only_ place the data could
be found was in the device tree.
>
>> One other point, the method to get there from POWER systems will be
>> essentially different that what is used for IAxx. Need to convince OFW
>> where to find the boot disk. Same information, different source...
>
>You mean parse in initrd dynamically determine the iSCSI root device based
>on firmware settings?
>
>IMHO ... we shouldn't do that, especially base on boot (not root) data.
I don't disagree. :) However, to get there, the device-tree has to come
into play for power. It could be hardcoded, but FW has to know, and
FW can echo it back to us. Via the device-tree.
++doug
Hannes Reinecke
Well, actually I've been quite lazy and just poked these values into the
existing framework.
Upon revisiting I found that there is even an easier way of doing
things, without having to touch the iSCSI params list.
New patch attached.
Hannes Reinecke
>
> Oh yeah, I hit the send button too soon. I guess some of the problems or
> open-isues we have had are:
>
> 1. DB is too heavy wieght for just the boot part.
Once you have glibc in initramfs nothing shocks you anymore ...
> 2. The daemon is consider by some to be overkill.
Partially true. The restarting bit is somewhat awkward.
> 3. How to tell the program run in the initramfs what target to boot from.
Boot parameter? Otherwise defaulting to the values used at the time if
initramfs creation.
> 4. I am lazy and do not want to rewrite iscsid and iscsiadm for a
> special boot program and try to port fixes bacj and forth.
>
That is more important. When doing so iscsiadm / iscsid would have to be
split up into a common code part and a daemon specific part, so that
only the 'interface' is different.
> I started to do try to just run this from the initramfs:
>
> load modules and start network stuff
> # iscsid
> # in a script loop, through some discovery addresses defined in a text
> file like
> /etc/sysconfig/iscsi
> # iscsiadm -discover target but do not write to DB
> # iscsiadm -login
> # kill iscsid (add iscsiadm -shutdown_daemon option)
> # mount real root ....
>
> Then when we are in the initd scripts, we can restart iscsid. When
> mkinitramfs is run /etc/sysconfig/iscsi and /etc/iscsid.conf can be
> brought into the initramfs.
>
Well, you should restart initrd. Currently you can't as iscsid will
establish a new session instead of resyncing with the existing ones.
>
> But it appears running iscsid and iscsiadm are still too much of hassle
> becuase they require the DB code even though we are not going to use it
> and running iscsid as a daemon is overkill. So I have been trying to
> reorg the userspace code in a way that I could write a small program
> that could reuse the existing code. In the initramfs I could do:
>
> # in a script loop through some targets definitions in a text file like
> /etc/sysconfig/iscsi
> # iscsiboot -setup target
> # mount real root ....
>
> In this case /etc/sysconfig/iscsi would contain the actual target info
> instead of the disocery info so we could slim down what we need even
> more. Is it better to do discovery here though? I guess some people have
> problems with this too. Is this overkill when we can just use iscsid and
> iscsiadm already.
>
I don't think discovery is required in initramfs as this introduces too
much leeway in detecting the root disk.
IMHO the user / administrator should really know from which target he is
trying to boot; if not the whole exercise is quite moot.
Discovery would only be required if the iSCSI target name is known, but
not the current IP address; ie when using portal groups or somesuch.
Of which I'm not sure whether open-iscsi supports this at the moment, so
I think we can disregard this for now.
For booting actually I don't mind having to use iscsid & iscsiadm,
although with them there are some problems:
- Restarting iscsid: iscsid has to be restarted when switching over from
initramfs to the real rootfs; not implemented yet.
- temporary netlink event handling: when iscsid is being restarted there
is a window during which no iscsid runs, ie no-one picks up the iscsi
netlink events. The kernel driver needs some sort of stub to handle
this situation gracefully.
AFAICS the latter could may reduced to in-kernel NOP-IN and ASYNC event
handling. All other command sequences are purely initiator driven, ie
when no iscsid is running no commands will be sent :-).
And we're not doing ASYNC event handling, so that's easy, too.
Unfortunately we can't defer any NOP-in handling until the iscsid is
ready again, so we'll probably have to do that in the kernel.
And this leads to the question whether we shouldn't move to NOP handling
into the kernel entirely; this way we maybe can avoid having such a
'stub' handling. Plus connections would stay alive even without a
daemon; this would increase reliability a lot.
For having a separate iscsiboot program; actually, I don't mind having
one, but so far I don't see that as a mandatory feature.
More important is to have iscsid pickup existing sessions ...
So a question to the powers that be:
Can we move NOP-in handling into the kernel?
If not, how do we handle situations where there is _no_ daemon but
existing sessions which _have_ to be kept alive (eg because it's my root
fs).
Arguments like 'iscsid will never crash' don't count.
Unless someone tells me how to move the namespace of iscsid from
initramfs context onto rootfs context without doing a execve().
Mike Christie
> So a question to the powers that be:
> Can we move NOP-in handling into the kernel?
If we put in nop-in handling becuase we want to handle the case where
targets send pings to make sure the target is there, then for targets
like Equallogic we could need to handle async events in the kernel too
since that target will send msgs to indicate it is going to drop the
connection. If we are drawing the line that we are only putting in the
very basics for what needs to be done so the target does not initiate
the session/connection closure on us (like target sending in nop-in to
sync sequence numbers and may panic and drop session if we do not
reply), then I can see I can nop-in handling going in. And, EQL sending
async events for load balancing gets the "no commands will be sent"
response and we just happen to get to support the target using nop-in as
pings to verify we are alive as a side benefit becuase we need it for
the sequence number case. I am just trying to say we need to draw some
line in the sand as to what goes in the kernel and what is going to stay
out.
I think the target using nops as pings can usually be turned off from
the target. And I think the target sending nops just to sync the
sequence numbers or the a target using redirect to load balance or
divert traffic is going to be a rare case during boot, a lot more rare
then a command timing out and us kicking off iscsi_conn_failure in
.iscsi_eh_host_reset. Well, probably all these happening is going to be
pretty rare during the short time that iscsid will be unavailable during
boot so I am not sure how big a deal this is.
It is just an opnion, and not set in stone in my mind at this point
though. I am open to patches when we can state just what we are putting
in the kernel and what is staying out, but of course they do have to go
upstream so you have to be able to pass the christoph police :)
Maybe Dimitry and Alex have a better idea....
> If not, how do we handle situations where there is _no_ daemon but
> existing sessions which _have_ to be kept alive (eg because it's my root
> fs).
> Arguments like 'iscsid will never crash' don't count.
> Unless someone tells me how to move the namespace of iscsid from
> initramfs context onto rootfs context without doing a execve().
>
What does dm-multipath's daemon do today? Does it restart during the
boot process?
Hannes Reinecke
>
> Hannes Reinecke wrote:
> > So a question to the powers that be:
> > Can we move NOP-in handling into the kernel?
>
> If we put in nop-in handling becuase we want to handle the case where
> targets send pings to make sure the target is there, then for targets
> like Equallogic we could need to handle async events in the kernel too
> since that target will send msgs to indicate it is going to drop the
> connection. If we are drawing the line that we are only putting in the
> very basics for what needs to be done so the target does not initiate
> the session/connection closure on us (like target sending in nop-in to
> sync sequence numbers and may panic and drop session if we do not
> reply), then I can see I can nop-in handling going in. And, EQL
> sending
> async events for load balancing gets the "no commands will be sent"
> response and we just happen to get to support the target using nop-in
> as
> pings to verify we are alive as a side benefit becuase we need it for
> the sequence number case. I am just trying to say we need to draw some
> line in the sand as to what goes in the kernel and what is going to
> stay
> out.
>
Exactly. That's why I was asking.
I would vote for NOP-in in the kernel (to ensure normal operation)
and everything else in userland, including async event handling.
We're not doing much of async handling nowadays anyway; and since
no-one complained so far I don't think it'll be a problem.
And anyway, ASYNC event could be about anything (think of LUN change).
So there is a fair chance we would need a fully functional userspace
anyway to deal with those events properly.
Hence if an ASYNC event arrives during booting we'll probably won't
be able to handle it and we're not losing much by not handling it,
even if this would mean I'll have to reboot.
> I think the target using nops as pings can usually be turned off from
> the target. And I think the target sending nops just to sync the
> sequence numbers or the a target using redirect to load balance or
> divert traffic is going to be a rare case during boot, a lot more rare
> then a command timing out and us kicking off iscsi_conn_failure in
> .iscsi_eh_host_reset. Well, probably all these happening is going to
> be
> pretty rare during the short time that iscsid will be unavailable
> during
> boot so I am not sure how big a deal this is.
>
D'accord. That's why I think we're mostly done when we're doing NOP-in
handling in the kernel.
Regarding failure case: Maybe we can work around this problem by
delaying
the error recovery if no daemon is available.
AFAIK there is no upper limit how long an error recovery should take ...
I know, baaad design. But better than losing connection and not be able
to
boot ...
> It is just an opnion, and not set in stone in my mind at this point
> though. I am open to patches when we can state just what we are
> putting
> in the kernel and what is staying out, but of course they do have to
> go
> upstream so you have to be able to pass the christoph police :)
>
Maybe you should send the patches :-)
> > If not, how do we handle situations where there is _no_ daemon but
> > existing sessions which _have_ to be kept alive (eg because it's my
> > root
> > fs).
> > Arguments like 'iscsid will never crash' don't count.
> > Unless someone tells me how to move the namespace of iscsid from
> > initramfs context onto rootfs context without doing a execve().
> >
>
> What does dm-multipath's daemon do today? Does it restart during the
> boot process?
No need. For multipathing you'll just have to set up the correct tables
to ensure basic functionality. The daemon is started once the rootfs is
available via normal init script.
Again, we'll be losing event handling during initrd but this will just
be an intermittent failure. I can live with rebooting in those cases.
What would put me off if we had a consistent failure case, eg a target
sending NOP-ins at certain intervals and a system is failing to boot
because booting takes longer than those intervals.
IMHO it's a quite sad excuse to say 'sorry, you have to reconfigure your
target because we too lazy to deal with it properly'.
Cheers,
Hannes
--
No signature today as working offline.
Mike Christie
>
> D'accord. That's why I think we're mostly done when we're doing NOP-in
> handling in the kernel.
> Regarding failure case: Maybe we can work around this problem by
> delaying
> the error recovery if no daemon is available.
> AFAIK there is no upper limit how long an error recovery should take ...
> I know, baaad design. But better than losing connection and not be able
> to
> boot ...
Fun. I guess we can hook into scsi times out and return DID_IMM_RETRY
and let the commands keep looping around until they finally succeed, or
in the scsi eh return SUCCESS and let the TUR that gets sent figure it
out or send us back in the eh. Those are pretty ugly.
Maybe something with the transport blocked code would work, but there
would need to be a way to detect the connection is back, or I guess we
could take a different approach than the rport to what happens when the
block times out. If we threw the ping code from userspace in the kernel
then that would work too, but we already saw what happened with
dm-multipath's test IO daemon.
>
>
>>It is just an opnion, and not set in stone in my mind at this point
>>though. I am open to patches when we can state just what we are
>>putting
>>in the kernel and what is staying out, but of course they do have to
>>go
>>upstream so you have to be able to pass the christoph police :)
>>
>
> Maybe you should send the patches :-)
>
>
>>>If not, how do we handle situations where there is _no_ daemon but
>>>existing sessions which _have_ to be kept alive (eg because it's my
>>>root
>>>fs).
>>>Arguments like 'iscsid will never crash' don't count.
>>>Unless someone tells me how to move the namespace of iscsid from
>>>initramfs context onto rootfs context without doing a execve().
>>>
>>
>>What does dm-multipath's daemon do today? Does it restart during the
>>boot process?
>
>
> No need. For multipathing you'll just have to set up the correct tables
Ah ok, I thought it could run from its own ram fs or something when
doing root boot. Is it possible to just run iscsid from its own ramfs
and not need to be shutdown and restarted during boot?
Mike Christie
>
> Ah ok, I thought it could run from its own ram fs or something when
> doing root boot. Is it possible to just run iscsid from its own ramfs
> and not need to be shutdown and restarted during boot?
Scratch that. I guess it is possible, but not nice and people do not
like to do it.
Alvin Starr
Well. I have a feeling that the ramfs will have to be mounted. and
mounted to / or the initrd while booting.
So now you will have a ramfs that you can't unmount from the initrd and
the initrd.
You could make the iscsid a kernel process then it would not have to be
restarted. But that caries a whole bunch of baggage with it.
--
Alvin Starr || voice: (416)585-9971
Interlink Connectivity || fax: (416)585-9974
al...@iplink.net ||