Open-iSCSI multiple sessions per target.
Mike Vallaly
We are looking to implement multiple sessions per portal target
utilizing open-iscsi, but realized its a bigger project than we first
thought. This is in reguards to a post on the open-iscsi mailing list
on Feb 26 entitled "Open-iSCSI and I/O Multipathing with Equallogic
Arrays".
Features we would like to add to open-iscsi:
1. Enable iscsid to create multiple sessions to a target (one per
local mac address)
2. Enable "iscsiadm -m session -i" to accurately display multi-
session targets (hopefully by interface/MAC address)
3. Enable an iscsiadm command line option to force iscsid to re-
read the target configurations, and add/remove sessions bound to new/
removed network iface MAC addresses (i.e.: "iscsiadm -m node -T
iqn.target.lun -p portal.ip.address -reload")
4. Enable an iscsiadm command line option to add/remove sessions
for new/removed network iface/MAC addresses. (i.e.: "iscsiadm -m -T
iqn.target.lun -p portal.ip.address -I 00:0E:0C:C2:C3:BB --login")
Technical Issues:
We have applied the multipath patch changes which SO_BINDTODEVICE
on the network iface by specifying the MAC addr (configured by
"node.conn[0].device")
However, there is no sysfs file that shows which iface was used for
each connection (i.e.: "/sys/class/iscsi_connection/connection%d:%d/
iface").
The "iface" sysfs file would be used to within
iscsi_sysfs.c:get_session_by_sysfs() to determine which sessions and
connections may be bound to particular iface, in turn, would be used
by initiator.c:session_is_running() to validate that a session is
running for a particular iface.
Within the code, the configuration record (node_rec_t) are directly
associated with a single target and a single session (session_rec_t
for configuration and iscsi_session_t for active sessions). So this
could be changed to associate multiple sessions with a target.
The following example config file illustrates our proposed changes:
node.name =
iqn.2001-05.com.equallogic:6-8a0900-fdf687001-3e3000035b045ecb-test
node.transport_name = tcp
node.tpgt = 1
node.active_conn = 1
node.startup = manual
# Multi-session defaults or single session parameters:
# See node_rec_t.session_default
node.session.initial_cmdsn = 0
node.session.auth.authmethod = None
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 10
node.session.err_timeo.reset_timeout = 30
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 0
node.session.iscsi.MaxConnections = 0
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
# Multi-session node overrides from defaults above,
# implies multiple sessions for the target:
# To keep backwards compatability with existing node configurations
files,
# the absence of node.session[x] would imply a single session.
# See node_rec_t.session[0].iface
node.session[0].iface = 00:0E:0C:C2:C1:63
node.session[1].iface = 00:0E:0C:C2:C3:BB
# Overriding a specific session connection port:
# See node_rec_t.session[].conn[]
node.session[0].conn[0].port = 3261
# Connection defaults for each multi-session connection:
# See node_rec_t.conn_default
node.conn.address = 10.33.0.10
node.conn.port = 3260
node.conn.startup = manual
node.conn.tcp.window_size = 524288
node.conn.tcp.type_of_service = 0
node.conn.timeo.logout_timeout = 15
node.conn.timeo.login_timeout = 15
node.conn.timeo.auth_timeout = 45
node.conn.timeo.active_timeout = 5
node.conn.timeo.idle_timeout = 60
node.conn.timeo.ping_timeout = 5
node.conn.timeo.noop_out_interval = 0
node.conn.timeo.noop_out_timeout = 0
node.conn.iscsi.MaxRecvDataSegmentLength = 65536
node.conn.iscsi.HeaderDigest = None
node.conn.iscsi.DataDigest = None
node.conn.iscsi.IFMarker = No
node.conn.iscsi.OFMarker = No
The following structs changes support this:
typedef struct node_rec {
int dbversion;
char name[TARGET_NAME_MAXLEN];
char transport_name[ISCSI_TRANSPORT_NAME_MAXLEN];
int tpgt;
int active_conn;
iscsi_startup_e startup;
session_rec_t session_default; /* <=== */
session_rec_t session[ISCI_NODE_SESSIONS_MAX]; /*
<=== */
conn_rec_t conn_default; /* <=== */
} node_rec_t;
typedef struct session_rec {
int initial_cmdsn;
int reopen_max;
struct iscsi_auth_config auth;
struct iscsi_session_timeout_config timeo;
struct iscsi_error_timeout_config err_timeo;
struct iscsi_session_operational_config iscsi;
char iface[MAX_DEV_ID_SIZE + 1]; /* <===
*/
conn_rec_t conn_default; /* <=== */
conn_rec_t conn[ISCI_CONN_MAX]; /* <=== */
} session_rec_t;
typedef struct conn_rec {
iscsi_startup_e startup;
char address[NI_MAXHOST];
int port;
char iface[MAX_DEV_ID_SIZE + 1]; /* <=== */
struct iscsi_tcp_config tcp;
struct iscsi_connection_timeout_config timeo;
struct iscsi_conn_operational_config iscsi;
} conn_rec_t;
The use of the name "device" to specify the network iface is a bit
confusing, we think it may be less confusing if it was called "iface".
mgmt_icp.h:struct iscsiadm_req could have an additional iface field
for the iscsiadm -I option.
/* IPC Request */
typedef struct iscsiadm_req {
iscsiadm_cmd_e command;
union {
/* messages */
struct msg_session {
int sid;
node_rec_t rec;
char iface[MAX_DEV_ID_SIZE + 1];
} session;
struct msg_conn {
int sid;
int cid;
char iface[MAX_DEV_ID_SIZE + 1];
} conn;
} u;
} iscsiadm_req_t;
We have implemented some of these changes, however now it's clear
that this is a bigger project than we first realized, and it is not
directly in our area of expertise. Is there anything we can do to
help the open-iscsi developers?
Thanks for your time.
- Kurt & Mike
--
Kurt Stephens
Michael Vallaly
Mike Christie
want to take a look at.
Mike Vallaly wrote:
> Greetings,
Hi,
Thanks for posting this and for working on it.
>
> We are looking to implement multiple sessions per portal target
> utilizing open-iscsi, but realized its a bigger project than we first
> thought. This is in reguards to a post on the open-iscsi mailing list
> on Feb 26 entitled "Open-iSCSI and I/O Multipathing with Equallogic
> Arrays".
>
> Features we would like to add to open-iscsi:
>
> 1. Enable iscsid to create multiple sessions to a target (one per
> local mac address)
>
> 2. Enable "iscsiadm -m session -i" to accurately display multi-
> session targets (hopefully by interface/MAC address)
>
> 3. Enable an iscsiadm command line option to force iscsid to re-
> read the target configurations, and add/remove sessions bound to new/
> removed network iface MAC addresses (i.e.: "iscsiadm -m node -T
> iqn.target.lun -p portal.ip.address -reload")
>
> 4. Enable an iscsiadm command line option to add/remove sessions
> for new/removed network iface/MAC addresses. (i.e.: "iscsiadm -m -T
> iqn.target.lun -p portal.ip.address -I 00:0E:0C:C2:C3:BB --login")
>
I agree these items will be useful.
>
> Technical Issues:
>
> We have applied the multipath patch changes which SO_BINDTODEVICE
> on the network iface by specifying the MAC addr (configured by
> "node.conn[0].device")
>
> However, there is no sysfs file that shows which iface was used for
> each connection (i.e.: "/sys/class/iscsi_connection/connection%d:%d/
> iface").
>
> The "iface" sysfs file would be used to within
> iscsi_sysfs.c:get_session_by_sysfs() to determine which sessions and
> connections may be bound to particular iface, in turn, would be used
> by initiator.c:session_is_running() to validate that a session is
> running for a particular iface.
>
My only issue with this is that hardware iscsi needs to export the mac
address and I sent a patch to export that on the iscsi_host. For
qla4xxx, we have to put it on the host because we may need to address
cards by the mac. For example to run
iscsiadm -m discovery -t st -p 192.168.0.19
and make it go through a specific qlogic card, or even to a specific
cards ip address we need a way to specify a specific card.
It looks like we cannot use a the card's serial number because if a card
has two ports, it has two pci resources, and two macs and this shows up
as two scsi_hosts/pci_devs (is this correct for all cards Dave?).
For software iscsi, since we are doing single connection sessions and a
session per host, we could put it there too and then we would have a
common place to look.
Of course we could also put it on the session or connection too. For
software iscsi, it might be better to put on the connection since if we
ever did MC/s we would have to put it there.
Any thoughts?
I think something like this would be fine. If we export the mac on the
connection in sysfs we probably want to set it there too.
My only concern was that we may want different settings for different
cards. In that case we could just name a file with the
{mac,target,portal} tuple. For software iscsi most likely we are going
to have the same settings, but if qla4xxx or other cards store their
data in userspace in our db, then we may want to allow software iscsi
and qlogic to connect to the same portal so we can do multi-initiated
multipath or failover like we do today. For this I think we may want
different settings. Well, maybe it could? Dave? People on the list?
What is up with the two session and session array?
That is fine.
> mgmt_icp.h:struct iscsiadm_req could have an additional iface field
> for the iscsiadm -I option.
>
> /* IPC Request */
> typedef struct iscsiadm_req {
> iscsiadm_cmd_e command;
>
> union {
> /* messages */
> struct msg_session {
> int sid;
> node_rec_t rec;
> char iface[MAX_DEV_ID_SIZE + 1];
If we do the array of ifaces in the node record file, then I think we
actually want the iface array in the node_rec_t.
Kurt Stephens
Maybe we should enumerate connections using an open-ended record
format like: "<target>,<portal-IP>,<port>,<device-type>,<device-
specific-info>..."
For sw iscsi this would be "<target>,<IP>,<port>,sw,eth,
00:11:22:33:44:55"; for hw iscsi it could be
"<target>,<IP>,<port>,qla4xxx,pci/0000:00:00.0,11:22:33:44:55:66" to
keep it unique.
If we make the connection identifiers true records we can search
them in a unified manner.
node_rec_t.session_default holds the defaults for all sessions,
node_rec_t.session[0] hold specifics for each session.
The node.session[0].conn[0].port = 3261" conf syntax assumes that
there still may be multiple connections per each session.
My next questions about the code is the "session id", appears in the
code often as function parameter "int sid". Is this sid global across
all targets or per target?
If we unify the connection enumeration this could be simpler:
/* IPC Request */
typedef struct iscsiadm_req {
iscsiadm_cmd_e command;
union {
/* messages */
struct msg_session {
> > node_rec_t rec;
> > int sid;
> > char iface[MAX_DEV_ID_SIZE + 1];
>
> If we do the array of ifaces in the node record file, then I think we
> actually want the iface array in the node_rec_t.
>
> > } session;
> > struct msg_conn {
> > int sid;
> > int cid;
> > char iface[MAX_DEV_ID_SIZE + 1];
> > } conn;
> > We have implemented some of these changes, however now it's clear
Kurt Stephens
Sorry accidentally sent reply too early, last comment should have
been:
If we unify the connection enumeration this could be simpler:
/* IPC Request */
typedef struct iscsiadm_req {
iscsiadm_cmd_e command;
union {
/* messages */
struct msg_session {
node_rec_t rec;
int sid;
char conn_id[???]; /* "<target>,<ip>,<port>,<dev-
type>,<dev-detail>..." */
} session;
...
} ...
}
Kurt Stephens
luosn
ALL functions you said we implemented But I can not say
until I leave this company.
So sad.
Ming Zhang
GPL license?
Ming
>
> >
David Somayajulu
> Sent: Monday, March 12, 2007 9:01 AM
port of the adapter has an iSCSI device associated with it. The HBA is a
multifunction PCI device, so effectively each iSCSI device on the board
appears as a separate PCI device. Each of the PCI devices of course has
its own MAC address. If you just want port level fail-over, binding the
IP address to a MAC address as suggested above should be fine. However
HBA level failover is more complex.
I was actually thinking on the lines of adding a iscsi host# parameter
to iscsiadm to choose the appropriate qla4xxx device. We can add the HBA
serial no. as a sysfs parameter for every qla4xxx device. This can be
used to provide HBA Level failover.
iscsiadm -m discovery -t st -p 192.168.0.19 -h <iscsi_host#>
David S.:I need to think thru this a little bit. For now we would like
to keep the above info. in flash for persistent targets. The current
qla4xxx open-iscsi does support multiple-sessions per target.
luosn
Yes, but others think not like this. But I can give you some hints.
When begin discovery: add this, you also must add some head file.
memset(&daemon_config, 0, sizeof(daemon_config));
daemon_config.pid_file = pid_file;
daemon_config.config_file = config_file;
daemon_config.initiator_name_file = initiatorname_file;
daemon_config.initiator_name =
get_iscsi_initiatorname(daemon_config.initiator_name_file);
if (daemon_config.initiator_name == NULL) {
log_warning("exiting due to configuration error");
exit(1);
}
//choice
daemon_config.initiator_alias =
get_iscsi_initiatoralias(daemon_config.initiator_name_file);
if (!daemon_config.initiator_alias) {
memset(&host_info, 0, sizeof(host_info));
if (uname(&host_info) >= 0) {
daemon_config.initiator_alias = strdup(host_info.nodename);
}
}
//choice
When "login" redo it
Mutilsession takes some trouble, if connect be cutted. Reconnect waste
much CPU.
Minish Maxconnection: 30 -> ?
The else easily.
Mike Christie
> David S.: Mike is correct (as always). On a qla4xxx based adapter, each
> port of the adapter has an iSCSI device associated with it. The HBA is a
> multifunction PCI device, so effectively each iSCSI device on the board
> appears as a separate PCI device. Each of the PCI devices of course has
> its own MAC address. If you just want port level fail-over, binding the
> IP address to a MAC address as suggested above should be fine. However
> HBA level failover is more complex.
>
> I was actually thinking on the lines of adding a iscsi host# parameter
> to iscsiadm to choose the appropriate qla4xxx device. We can add the HBA
> serial no. as a sysfs parameter for every qla4xxx device. This can be
> used to provide HBA Level failover.
>
What is HBA level failover? Would the kernel only see one scsi_host
instead of the scsi_host per PCI device it sees today? And then the HBA
would run across both ports on the card itself without the OS getting
involved?
Or would we still see two scsi_hosts and the driver would do failover
across the ports like with the failover code in some of the distro drivers?