[RFC] : network configuration for iscsi hba

[Anil Veerabhadrappa]

Hi,

We'd like to propose a general scheme for configuring network
parameters (IP address/mask/DHCP, etc) for iSCSI NICs that use a private
IP address. The current scheme of using sysfs is not very good because
bootproto, IP address, netmask, VLAN ID, etc all need to be set
together. Having separate sysfs entries and updating them separately and
independently will not work. Putting everything in a netlink message
seems to be a better solution. After weighing different solutions, we
feel expanding existing netlink family, NETLINK_ISCSI between iscsid and
scsi_transport_iscsi is a flexible solution which allows information
flow to be initiated from either side. Also this solution is flexible
and elegantly handles network devices with multiple IP addresses.

The objective of this proposal is to make this interface common for
all iscsi offload solutions supported by open-iscsi. We would like to
hear comments and suggestions from other iscsi offload vendors in
defining this interface.

Regards,
Anil Veerabhadrappa

Proposal to add netlink message type:
-------------------------------------
3 new netlink message types are required to support network config
message exchange between user and kernel components,
1. ISCSI_UEVENT_SET_NET_CONFIG - push iface info to driver to configure
an iscsi network interface
2. ISCSI_UEVENT_GET_NET_CONFIG - get MAC address list, etc'
3. ISCSI_KEVENT_NET_CONFIG - propagate attribute changes from
adapter to iscsid (e.g. advertise newly obtained dhcp address)

iscsid will use this netlink messages to pass network configuration
between user mode application and the driver. Once this message is
received by scsi_transport_iscsi module it will call driver's newly
added callback handlers in the iscsi_transport structure(net_config &
get_net_config) and also broadcast netlink message back to any hardware
vendor's user level daemons

ISCSI_XEVENT_NET_CONFIG message payload format:
-----------------------------------------------
Payload consists of one or more config parameters defined in TLV
(Type - Length - Value) format.

struct net_cfg_tlv {
uint32_t type;
uint32_t length;
uint8_t value[0];
};

Message types:
--------------
This interface is envisioned to support standard network parameters
such as netdev name, MAC address, IPv4/IPv6 address, VLAN, boot protocol
(static or dhcp), etc'. Please refer to 'net_cfg_e' in proposed header
file changes below.

Advantages:
-----------
This approach is much cleaner and delinks network configuration
parameters currently bound to host attributes. Old scheme actually
breaks the layering scheme as seen below,

SCSI (net config parametes currently resides here)
|
iSCSI
|
TCP/IP

This new approach is a deviation from current host attributes
approach and places emphasis in iface components of iscsid and the LLD.

Changes to iscsi transport headers:
-----------------------------------

diff --git a/include/iscsi_if.h b/include/iscsi_if.h
index afa86e8..76eea8c 100644
--- a/include/iscsi_if.h
+++ b/include/iscsi_if.h
@@ -51,6 +51,8 @@ enum iscsi_uevent_e {
ISCSI_UEVENT_SET_HOST_PARAM = UEVENT_BASE + 16,
ISCSI_UEVENT_UNBIND_SESSION = UEVENT_BASE + 17,
ISCSI_UEVENT_CREATE_BOUND_SESSION = UEVENT_BASE + 18,
+ ISCSI_UEVENT_SET_NET_CONFIG = UEVENT_BASE + 19,
+ ISCSI_UEVENT_GET_NET_CONFIG = UEVENT_BASE + 20,

/* up events */
ISCSI_KEVENT_RECV_PDU = KEVENT_BASE + 1,
@@ -59,6 +61,7 @@ enum iscsi_uevent_e {
ISCSI_KEVENT_DESTROY_SESSION = KEVENT_BASE + 4,
ISCSI_KEVENT_UNBIND_SESSION = KEVENT_BASE + 5,
ISCSI_KEVENT_CREATE_SESSION = KEVENT_BASE + 6,
+ ISCSI_KEVENT_NET_CONFIG = KEVENT_BASE + 7,
};

enum iscsi_tgt_dscvr {
@@ -317,6 +320,42 @@ enum iscsi_host_param {
#define ISCSI_HOST_NETDEV_NAME (1ULL << ISCSI_HOST_PARAM_NETDEV_NAME)
#define ISCSI_HOST_IPADDRESS (1ULL << ISCSI_HOST_PARAM_IPADDRESS)

+/* iscsi network stack parameters */
+enum iscsi_net_cfg_e {
+ ISCSI_NET_CFG_UNKNOWN 0x00,
+
+ ISCSI_NET_CFG_NETDEV_NAME ISCSI_NET_CFG_UNKNOWN + 1,
+ ISCSI_NET_CFG_MAC_ADDR ISCSI_NET_CFG_UNKNOWN + 2,
+ ISCSI_NET_CFG_IPV4_ADDR ISCSI_NET_CFG_UNKNOWN + 3,
+ ISCSI_NET_CFG_IPV6_ADDR ISCSI_NET_CFG_UNKNOWN + 4,
+ ISCSI_NET_CFG_IPV4_NETMASK ISCSI_NET_CFG_UNKNOWN + 5,
+ ISCSI_NET_CFG_IPV6_NETMASK ISCSI_NET_CFG_UNKNOWN + 6,
+ ISCSI_NET_CFG_IPV4_GATEWAY ISCSI_NET_CFG_UNKNOWN + 7,
+ ISCSI_NET_CFG_IPV6_GATEWAY ISCSI_NET_CFG_UNKNOWN + 8,
+ ISCSI_NET_CFG_BOOTPROTO ISCSI_NET_CFG_UNKNOWN + 9,
+ ISCSI_NET_CFG_IPV6_AUTO_CFG ISCSI_NET_CFG_UNKNOWN + 10,
+ ISCSI_NET_CFG_MTU ISCSI_NET_CFG_UNKNOWN + 11,
+ ISCSI_NET_CFG_VLAN ISCSI_NET_CFG_UNKNOWN + 12,
+};
+
+enum net_cfg_param {
+ /* MAC address list for all devices managed by the transports,
+ * bnx2i/cxgb3, qla4xxx */
+ ISCSI_NET_MAC_ADDR_LIST,
+ /* Given a MAC address driver will return device attributes such
+ * as scsi_host no, netdev name, etc' */
+ ISCSI_NET_DEV_INFO,
+ /* give a MAC address, return IP address info for network interfaces
+ * configured */
+ ISCSI_NET_MACS_IP_INFO,
+ /* return IP address info of all network interfaces configured for
+ * the given netdev name */
+ ISCSI_NET_NETDEV_IP_INFO,
+ /* return IP address info of all network interfaces configured for
+ * the given scsi host number */
+ ISCSI_NET_HOSTNO_IP_INFO,
+};
+
#define iscsi_ptr(_handle) ((void*)(unsigned long)_handle)
#define iscsi_handle(_ptr) ((uint64_t)(unsigned long)_ptr)

diff --git a/kernel/scsi_transport_iscsi.h
b/kernel/scsi_transport_iscsi.h
index b65c96a..37ca2ac 100644
--- a/kernel/scsi_transport_iscsi.h
+++ b/kernel/scsi_transport_iscsi.h
@@ -124,6 +124,8 @@ struct iscsi_transport {
void (*ep_disconnect) (struct iscsi_endpoint *ep);
int (*tgt_dscvr) (struct Scsi_Host *shost, enum iscsi_tgt_dscvr type,
uint32_t enable, struct sockaddr *dst_addr);
+ int (*set_net_config)(void *buf, int buflen);
+ int (*get_net_config)(enum net_if_param param, void *buf);
};

/*

[Karen Xie]

Hi,

Regards,
Anil Veerabhadrappa

[Karen] What does ISCSI_NET_CFG_IPV6_AUTO_CFG mean?

+ ISCSI_NET_CFG_MTU ISCSI_NET_CFG_UNKNOWN +
11,
+ ISCSI_NET_CFG_VLAN ISCSI_NET_CFG_UNKNOWN +
12,
+};
+

[Karen] how about adding DHCP server settings too?

+enum net_cfg_param {
+ /* MAC address list for all devices managed by the transports,
+ * bnx2i/cxgb3, qla4xxx */
+ ISCSI_NET_MAC_ADDR_LIST,
+ /* Given a MAC address driver will return device attributes such
+ * as scsi_host no, netdev name, etc' */
+ ISCSI_NET_DEV_INFO,
+ /* give a MAC address, return IP address info for network
interfaces
+ * configured */
+ ISCSI_NET_MACS_IP_INFO,
+ /* return IP address info of all network interfaces configured
for
+ * the given netdev name */
+ ISCSI_NET_NETDEV_IP_INFO,
+ /* return IP address info of all network interfaces configured
for
+ * the given scsi host number */
+ ISCSI_NET_HOSTNO_IP_INFO,

[Karen] I assume ISCSI_NET_XXX_IP_INFOs are for the iSCSI IP addresses
only?

+};
+

#define iscsi_ptr(_handle) ((void*)(unsigned long)_handle)
#define iscsi_handle(_ptr) ((uint64_t)(unsigned long)_ptr)

diff --git a/kernel/scsi_transport_iscsi.h
b/kernel/scsi_transport_iscsi.h
index b65c96a..37ca2ac 100644
--- a/kernel/scsi_transport_iscsi.h
+++ b/kernel/scsi_transport_iscsi.h
@@ -124,6 +124,8 @@ struct iscsi_transport {
void (*ep_disconnect) (struct iscsi_endpoint *ep);
int (*tgt_dscvr) (struct Scsi_Host *shost, enum iscsi_tgt_dscvr
type,
uint32_t enable, struct sockaddr *dst_addr);
+ int (*set_net_config)(void *buf, int buflen);
+ int (*get_net_config)(enum net_if_param param, void *buf);
};

/*

[Karen]So how the user going to manage all these
information/configuration, I assume via open-iscsi's ifaces/ files?

[Anil Veerabhadrappa]

configure IPv6 stateless and stateful address autoconfiguration

> + ISCSI_NET_CFG_MTU ISCSI_NET_CFG_UNKNOWN +
> 11,
> + ISCSI_NET_CFG_VLAN ISCSI_NET_CFG_UNKNOWN +
> 12,
> +};
> +
>
> [Karen] how about adding DHCP server settings too?
>

You are welcome to add necessary parameters as required by any solution.
Can you provide more details on parameters you have in mind?

> +enum net_cfg_param {
> + /* MAC address list for all devices managed by the transports,
> + * bnx2i/cxgb3, qla4xxx */
> + ISCSI_NET_MAC_ADDR_LIST,
> + /* Given a MAC address driver will return device attributes such
> + * as scsi_host no, netdev name, etc' */
> + ISCSI_NET_DEV_INFO,
> + /* give a MAC address, return IP address info for network
> interfaces
> + * configured */
> + ISCSI_NET_MACS_IP_INFO,
> + /* return IP address info of all network interfaces configured
> for
> + * the given netdev name */
> + ISCSI_NET_NETDEV_IP_INFO,
> + /* return IP address info of all network interfaces configured
> for
> + * the given scsi host number */
> + ISCSI_NET_HOSTNO_IP_INFO,
>
> [Karen] I assume ISCSI_NET_XXX_IP_INFOs are for the iSCSI IP addresses
> only?
>

correct, it deals with iscsi interface only and has nothing to do with
host interface.

> +};
> +
>
> #define iscsi_ptr(_handle) ((void*)(unsigned long)_handle)
> #define iscsi_handle(_ptr) ((uint64_t)(unsigned long)_ptr)
>
> diff --git a/kernel/scsi_transport_iscsi.h
> b/kernel/scsi_transport_iscsi.h
> index b65c96a..37ca2ac 100644
> --- a/kernel/scsi_transport_iscsi.h
> +++ b/kernel/scsi_transport_iscsi.h
> @@ -124,6 +124,8 @@ struct iscsi_transport {
> void (*ep_disconnect) (struct iscsi_endpoint *ep);
> int (*tgt_dscvr) (struct Scsi_Host *shost, enum iscsi_tgt_dscvr
> type,
> uint32_t enable, struct sockaddr *dst_addr);
> + int (*set_net_config)(void *buf, int buflen);
> + int (*get_net_config)(enum net_if_param param, void *buf);
> };
>
> /*
>
> [Karen]So how the user going to manage all these
> information/configuration, I assume via open-iscsi's ifaces/ files?
>

That's correct.

> >
>

[Karen Xie]

Thanks,
What are the possible values of "ISCSI_NET_CFG_BOOTPROTO"?

Karen

[Anil Veerabhadrappa]

On Thu, 2009-02-05 at 16:00 -0800, Karen Xie wrote:
> Thanks,
> What are the possible values of "ISCSI_NET_CFG_BOOTPROTO"?
>

static and dhcp

[Karen Xie]

If it is dhcp, does the dhcp server settings need to be provided? Or it
is going to be handled outside of iscsi?

Also a question about the set_net_config, suppose a list of parameters
are send down, if one or more values are not supported, does the
operation marked as fail and the user has re-do the iface file (to edit
out the unsupported values)?

Thanks,

[Anil Veerabhadrappa]

On Thu, 2009-02-05 at 16:36 -0800, Karen Xie wrote:
> If it is dhcp, does the dhcp server settings need to be provided? Or it
> is going to be handled outside of iscsi?
>

dhcp setting should be zero-conf right? It works based on initial
broadcast request, DHCPDISCOVER and subsequent unicast exchange between
dhcp client and the server. Are you referring to choosing the preferred
dhcp server if more than one is present in the network?

> Also a question about the set_net_config, suppose a list of parameters
> are send down, if one or more values are not supported, does the
> operation marked as fail and the user has re-do the iface file (to edit
> out the unsupported values)?
>

Please provide specific examples for discussion. Any device specific
limitation will be handled by the corresponding driver, our objective
here is to define a generic interface that works for
cxgb3i/qla4xxx/bnx2i/...

[Karen Xie]

For dhcp, I am just thinking of the cases where one physical NIC have
multiple IP addresses, maybe one is for iscsi only, one for RDMA, or
another for admin/control traffic. They can operate in different subnet
and managed by different DHCP servers. Maybe this is not typical setup?

For the set_config, say a NIC does not support IPv6 but received a
request to set it. So how should the driver treat any setting it does
not support? Ignore or return an error?

[Anil Veerabhadrappa]

On Thu, 2009-02-05 at 17:34 -0800, Karen Xie wrote:
> For dhcp, I am just thinking of the cases where one physical NIC have
> multiple IP addresses, maybe one is for iscsi only, one for RDMA, or
> another for admin/control traffic. They can operate in different subnet

RDMA shares ip/mac address with host stack, right? Even in the example
provided above the device should support 3 at least separate MAC
addresses, so by specifying multiple iface definitions user should be
able to get this configuration working.

> and managed by different DHCP servers. Maybe this is not typical setup?
>
> For the set_config, say a NIC does not support IPv6 but received a
> request to set it. So how should the driver treat any setting it does
> not support? Ignore or return an error?
>

This should be considered as user error. Users are expected to create
iface definitions based on guidelines (supported parameter list)
outlined in driver's README file.

[Karen Xie]

Okay, I think cxgb3i driver can work with the proposed mechanism.

[Mike Christie]

Karen Xie wrote:
> Okay, I think cxgb3i driver can work with the proposed mechanism.
>

Do you have a plan to support DHCP (the non boot case where you need a
software client)? Do you need details on how to hook into that part, or
have you started working on it with broadcom already?

[Karen Xie]

Hi, Mike,

I don't plan to support DHCP for cxgb3i. We do have a dependency on the
interface to be configured (i.e., cxgb3 driver brought up and IP address
set) before it can be used for iSCSI, though the IP address configured
on the interface does not needed to be used for iscsi.

Thanks,
Karen

-----Original Message-----
From: open-...@googlegroups.com [mailto:open-...@googlegroups.com]

[Mike Christie]

Karen Xie wrote:
> Hi, Mike,
>
> I don't plan to support DHCP for cxgb3i.

Ah you wuss :) The other day, you know when I asked if you needed a
software client or if you could do it in firmware I was asking for iscsi
right :)

[Karen Xie]

Hi, Mike,

Do you have any details on how to hook up with the DHCP software client?
I can look into it and see how much effort it is. We were not planning
it but it could be supported if needed.

Thanks,
Karen


-----Original Message-----
From: open-...@googlegroups.com [mailto:open-...@googlegroups.com]
On Behalf Of Mike Christie
Sent: Monday, February 09, 2009 8:06 PM
To: open-...@googlegroups.com
Cc: mchr...@redhat.com; Michael Chan; Benjamin Li
Subject: Re: [RFC] : network configuration for iscsi hba

[Mike Christie]

Karen Xie wrote:
> Hi, Mike,
>

> Do you have any details on how to hook up with the DHCP software client?

The broadcom guys do.

> I can look into it and see how much effort it is. We were not planning
> it but it could be supported if needed.
>

You do not have to support it. I would think a user might request it. If
you guys are going to say we just do not support it that is fine with me.

[Michael Chan]

On Tue, 2009-02-10 at 09:48 -0800, Mike Christie wrote:
> Karen Xie wrote:
> > Hi, Mike,
> >
> > Do you have any details on how to hook up with the DHCP software
> client?
>
> The broadcom guys do.

We'll have a user mode driver to receive and transmit DHCP/ARP/ICMP etc
packets for the iSCSI HBA using the UIO interface. For DHCP, this user
mode driver will listen to NETLINK_ISCSI for the ISCSI_KEVENT_NET_CONFIG
message that indicates DHCP setting. Once we obtain the leased IP
address, we'll send it to the kernel using ISCSI_UEVENT_NET_CONFIG.

[Mike Christie]

On 02/05/2009 01:23 PM, Anil Veerabhadrappa wrote:
> +enum net_cfg_param {
> + /* MAC address list for all devices managed by the transports,
> + * bnx2i/cxgb3, qla4xxx */
> + ISCSI_NET_MAC_ADDR_LIST,
> + /* Given a MAC address driver will return device attributes such
> + * as scsi_host no, netdev name, etc' */
> + ISCSI_NET_DEV_INFO,
> + /* give a MAC address, return IP address info for network interfaces
> + * configured */
> + ISCSI_NET_MACS_IP_INFO,
> + /* return IP address info of all network interfaces configured for
> + * the given netdev name */
> + ISCSI_NET_NETDEV_IP_INFO,
> + /* return IP address info of all network interfaces configured for
> + * the given scsi host number */
> + ISCSI_NET_HOSTNO_IP_INFO,
> +};

Hey Anil,

I am working on this so we can set net params for offload cards like
qla4xxx and be2iscsi, and I would like to also make sure we can use it
to set vlan stuff or any other net params for drivers like bnx2i.

I was wondering though why above some commands are more net based (pass
in a MAC addr or netdev and get params), but in the patches you ended up
sending the set_net_config netlink event used the host_no.

If I use the set_net_config like how you defined it using the host_no
instead of some net value like the MAC, do you see any problems? Or did
you hit problems using values like the MAC, so is that why you ended up
using the host no?

Thanks

[Yao Wei]

Dear sir:

I have tested iscsi protocols on two machines using open-iscsi and
iscsi-target. On IBM X3550 machine which has two NICs (SUSE Linux
Enterprise
10.1) I simulate iscsi initiator , and on Dell Optiplex 755 machine
(Fedora 8 ) for iscsi target . The network environment is 1GbE.

Unfortunately, when I discovery the target, and used hdparm to test
the read speed, I found the speed could only obtain 10MB/s.

Later, I let the IBM X3550 to be the target, and let Dell 755 to be
initiator, the speed could obtain above 80MB/s.

I can not know the reason. Looking forword to your replies.

--