[ANNOUNCE 0/2] Serial Attached SCSI (SAS) support for the Linux kernel

Luben Tuikov

unread,

Sep 9, 2005, 3:30:48 PM9/9/05

to Linux Kernel Mailing List, SCSI Mailing List

Hi,

The following announcements and patches introduce
Serial Attached SCSI (SAS) support for the Linux kernel.
Everything is supported.

The infrastructure is broken into
* SAS LLDD,
* SAS Layer.

The SAS LLDD does phy/OOB management, and generates SAS events
to the SAS Layer. Those events are *the only way* a SAS LLDD
communicates with the SAS Layer. If you can generate 2 types
of event, then you can use this infrastructure. The first two
are, loosely, "link was severed", "bytes were dmaed". The third
kind is "received a primitive", used for domain revalidation.

A SAS LLDD should implement the Execute Command SCSI RPC and
at least one SCSI TMF (Task Management Function), in order for
the SAS Layer to communicate with the SAS LLDD.

The SAS Layer is concerned with
* SAS Phy/Port/HA event management (LLDD generates,
SAS Layer processes),
* SAS Port management (creation/destruction),
* SAS Domain discovery and revalidation,
* SAS Domain device management,
* SCSI Host registration/deregistration,
* Device registration with SCSI Core (SAS) or libata
(SATA/PI), and
* Expander management and exporting expander control
to user space.

The SAS Layer uses the Execute Command SCSI RPC, and the TMFs
implemented by the SAS LLDD in order to manage the domain and
the domain devices.

For details please see drivers/scsi/sas-class/README.

The SAS Layer represents the SAS domain in sysfs. For each
object represented, its parent is the physical entity it attaches
to in the physical world. So in effect, kobject_get, gets
the whole chain up on which that object depends on.

In effect, the sysfs representation of the SAS domain(s)
is what you'd see in the physical world.

Hot plugging and hot unplugging of devices, domains and subdomains
is supported. Repeated hot plugging and hot unplugging is
also supported, naturally.

SAS introduces a new physical entity, an expander.
Expanders are _not_ SAS devices, and thus are _not_ SCSI devices.
Expanders are part of the Service Delivery Subsystem, in this case
SAS.

Expanders are controlled using the Serial Management Protocol (SMP).
Complete control is given to user space of all expanders found
in the domain, using an "smp_portal". More of this in the second
and third email in this series.

A user space program, "expander_conf.c" is also presented to show
how one controls expanders in the domain. It is located here:
drivers/scsi/sas-class/expanders_conf.c

The second email in this series shows an example of SAS domains
and their representation in sysfs.

The third email in this series shows an example of using the
"expander_conf.c" program to query all expanders in the domain,
showing their attributes, their phys, and their routing tables.

If you have the hardware, please give it a try. If you have
expander(s) it would be even more interesting.

Patches of the SAS Layer and of the AIC94XX SAS LLDD follow.

Luben
P.S. You can also download the patches from
http://www.geocities.com/ltuikov/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Luben Tuikov

unread,

Sep 9, 2005, 3:31:29 PM9/9/05

to Linux Kernel Mailing List, SCSI Mailing List

This email shows an example of two domains.
The first domain has only two devices: the SAS initiator
and a SAS end device (a disk).
The second domain has expanders, SAS disks, a SATA II disk
and a SATAPI disk.

A succinct and full listings of the sysfs tree are shown
using the tree(1) program.

Sample configuration
--------------------

Directly attached devices, cascaded expanders and
wide ports

1. SAS disk on HA phy 0, forming a narrow port, port 0.

2. Expander on HA phys 4, 5, 6, and 7, forming
a wide port, port 1.

3. Port 0, narrow, of phy 0, is formed between
the host adapter and a SAS end device with SAS port identifier
(aua/aka, SAS address) 500000e000031c12. This device
has a single LU, with LUN 0, which is a disk device.

4. Port 1, wide, of phys 4, 5, 6 and 7, is formed between
the Host adapter's phys 4-7 and the expander with SAS
address 50001c1716010600, Expander A.

5. Expander A, 50001c1716010600, has 4 devices attached to it:
- An expander with SAS address 50001c1071609c00,
expander B (using a 2 link wide port, see below).
- A SATA II disk, 50001c1716010603, using the expanders'
STP/SATA bridge.
- An SES device, LUN 0 of 50001c171601060d.
- And a SAS disk, LUN 0 of 5000c50000513329.

6. Expander B, 50001c1071609c00, has 3 devices attached to it:
- A SATAPI device, 50001c1071609c02, using the expander's
STP/SATA bridge.
- A SAS disk, LUN 0 of 5000c50000102a65.
- And an expander, 5005076a000001e0, expander C,
(using a 2 link wide port, see below).

7. Expander C, 5005076a000001e0, has 3 devices attached to it:
- A SAS disk, LUN 0 of 5000c50000513385.
- An SES device, LUN 0 of 5005076a000001ed.
- A SAS disk, LUN 0 of 5c50000000409c11.

Succinct listing of the tree
----------------------------

tree -d /sys/devices/pci0000\:01/0000\:01\:04.0/host11/sas/ha

/sys/devices/pci0000:01/0000:01:04.0/host11/sas/ha
|-- phys
| |-- 0
| | `-- port -> ../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/ports/0
| |-- 1
| |-- 2
| |-- 3
| |-- 4
| | `-- port -> ../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/ports/1
| |-- 5
| | `-- port -> ../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/ports/1
| |-- 6
| | `-- port -> ../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/ports/1
| `-- 7
| `-- port -> ../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/ports/1
`-- ports
|-- 0
| |-- domain
| | `-- 500000e000031c12
| | `-- LUNS
| | `-- 0000000000000000
| `-- phys
| `-- 0 -> ../../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/phys/0
`-- 1
|-- domain
| `-- 50001c1716010600
| |-- 50001c1071609c00
| | |-- 50001c1071609c02
| | |-- 5000c50000102a65
| | | `-- LUNS
| | | `-- 0000000000000000
| | `-- 5005076a000001e0
| | |-- 5000c50000513385
| | | `-- LUNS
| | | `-- 0000000000000000
| | |-- 5005076a000001ed
| | | `-- LUNS
| | | `-- 0000000000000000
| | `-- 5c50000000409c11
| | `-- LUNS
| | `-- 0000000000000000
| |-- 50001c1716010603
| |-- 50001c171601060d
| | `-- LUNS
| | `-- 0000000000000000
| `-- 5000c50000513329
| `-- LUNS
| `-- 0000000000000000
`-- phys
|-- 4 -> ../../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/phys/4
|-- 5 -> ../../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/phys/5
|-- 6 -> ../../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/phys/6
`-- 7 -> ../../../../../../../../../devices/pci0000:01/0000:01:04.0/host11/sas/ha/phys/7

52 directories

Full listing of the tree
------------------------

52 directories, 308 files

Luben Tuikov

unread,

Sep 9, 2005, 3:32:03 PM9/9/05

to Linux Kernel Mailing List, SCSI Mailing List

Using the User Space Expander configuration utility
===================================================

The file "expander_conf.c" is a user space program
which uses the "smp_portal" SMP portal to communicate
with expanders in the domain.

It gives you complete control of the expander you're
talking to, so please be careful if you're changing
route tables, phy settings, etc.

The "expander_conf.c" file is located here:
drivers/scsi/sas-class/expander_conf.c

How it works can be found in
drivers/scsi/sas-class/README, and of course in the
source code.

Here we show complete information about the three
expanders we saw in the previous email.

Dumping configuration info for Expander A
-----------------------------------------

A) As you can see phys 11, 10, 9 and 8 of this expander
are connected to phys 4, 5, 7, and 6 of the Host Adapter,
thus forming a wide port.

This is also the subtractive boundary of this
edge expander device set and of this expander.

B) As you can see phys 7 and 6 of this expander are
connected to phys 8 and 9 of expander B, thus forming
a wide port.

They use table routing attributes.

C) The routing table is also printed for phys 7 and 6.
It is of course identical.

D) End devices attached are clearly identifiable.

#expander_conf /sys/devices/pci0000\:01/0000\:01\:04.0/host11/sas/ha/ports/1/domain/50001c1716010600/smp_portal
Vendor: VITESSE
Product: VSC7160 Eval Brd
Revision: 4
Component: VITESSE
Component ID: 29024
Component revision: 4
Expander Change Count: 12
Expander Route Indexes: 160
Number of phys: 13
Configuring: No
Configurable route table: Yes
Enclosure Logical Identifier: 00000000000000000000

Phy00:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3221225471
RD error count: 4294958971
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy01:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3706272315
RD error count: 2079374150
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy02:D attached: 50001c1716010603:00 chg count:05
Attached device: SATA device Link rate: G1 (1,5 Gb/s)
Tproto: STP Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 4216795329
RD error count: 4277909111
DW sync loss count: 0
Reset problem count: 0

Phy03:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 828522753
RD error count: 27724688
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy04:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy05:D attached: 5000c50000513329:01 chg count:01
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: SSP Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 4292849663
RD error count: 4261371839
DW sync loss count: 0
Reset problem count: 0

Phy06:T attached: 50001c1071609c00:09 chg count:01
Attached device: edge expander Link rate: G2 (3 GB/s)
Tproto: SMP Iproto: SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3585802240
RD error count: 3870195680
DW sync loss count: 0
Reset problem count: 0
Routing Table
00 enabled 5000c50000102a65
01 enabled 50001c1071609c02
02 enabled 5005076a000001e0
03 enabled 5000c50000513385
04 enabled 5c50000000409c11
05 enabled 5005076a000001ed

Phy07:T attached: 50001c1071609c00:08 chg count:01
Attached device: edge expander Link rate: G2 (3 GB/s)
Tproto: SMP Iproto: SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3229920258
RD error count: 1111358524
DW sync loss count: 0
Reset problem count: 0
Routing Table
00 enabled 5000c50000102a65
01 enabled 50001c1071609c02
02 enabled 5005076a000001e0
03 enabled 5000c50000513385
04 enabled 5c50000000409c11
05 enabled 5005076a000001ed

Phy08:S attached: 50000d1000281260:06 chg count:01
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: Iproto: SSP|STP|SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 4154908636
RD error count: 2808853200
DW sync loss count: 0
Reset problem count: 0

Phy09:S attached: 50000d1000281260:07 chg count:01
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: Iproto: SSP|STP|SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 371055385
RD error count: 3645431138
DW sync loss count: 0
Reset problem count: 0

Phy10:S attached: 50000d1000281260:05 chg count:01
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: Iproto: SSP|STP|SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3790805948
RD error count: 2017124313
DW sync loss count: 0
Reset problem count: 0

Phy11:S attached: 50000d1000281260:04 chg count:01
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: Iproto: SSP|STP|SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0

Phy12:D attached: 50001c171601060d:00 chg count:00
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: SSP Iproto: SSP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Virtual phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0

Dumping configuration info for Expander B
-----------------------------------------

A) As you can see phys 9 and 8 of this expander are
connected to phys 6 and 7 of Expander A as we already
noted above. This forms the subtractive boundary of this
expander.

B) As you can see phys 3 and 2 of this expander are
connected to phys 7 and 3 of Expander C, thus forming
a wide port.

They use the table routing attribute.

C) The routing table of phys 3 and 2 is also
printed. It is of course identical.

D) End devices attached are clearly identifiable.

expander_conf /sys/devices/pci0000\:01/0000\:01\:04.0/host11/sas/ha/ports/1/domain/50001c1716010600/50001c1071609c00/smp_portal

Vendor: VITESSE
Product: VSC7160 Eval Brd
Revision: 4
Component: VITESSE
Component ID: 29024
Component revision: 4
Expander Change Count: 6
Expander Route Indexes: 160
Number of phys: 12
Configuring: No
Configurable route table: Yes
Enclosure Logical Identifier: 00000000000000000000

Phy00:D attached: 5000c50000102a65:00 chg count:01
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: SSP Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 2351827995
RD error count: 3908042718
DW sync loss count: 0
Reset problem count: 0

Phy01:D attached: 50001c1071609c02:00 chg count:01
Attached device: SATA device Link rate: G1 (1,5 Gb/s)
Tproto: STP Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0

Phy02:T attached: 5005076a000001e0:03 chg count:01
Attached device: edge expander Link rate: G2 (3 GB/s)
Tproto: SMP Iproto: SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 1939984985
RD error count: 2207511641
DW sync loss count: 0
Reset problem count: 0
Routing Table
00 enabled 5000c50000513385
01 enabled 5c50000000409c11
02 enabled 5005076a000001ed

Phy03:T attached: 5005076a000001e0:07 chg count:01
Attached device: edge expander Link rate: G2 (3 GB/s)
Tproto: SMP Iproto: SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 2343129041
RD error count: 3405770672
DW sync loss count: 0
Reset problem count: 0
Routing Table
00 enabled 5000c50000513385
01 enabled 5c50000000409c11
02 enabled 5005076a000001ed

Phy04:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy05:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy06:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3216754055
RD error count: 4159684279
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy07:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3687816959
RD error count: 4261377627
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy08:S attached: 50001c1716010600:07 chg count:01
Attached device: edge expander Link rate: G2 (3 GB/s)
Tproto: SMP Iproto: SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 790459193
RD error count: 3202994802
DW sync loss count: 0
Reset problem count: 0

Phy09:S attached: 50001c1716010600:06 chg count:01
Attached device: edge expander Link rate: G2 (3 GB/s)
Tproto: SMP Iproto: SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0

Phy10:S attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3888053628
RD error count: 3011959286
DW sync loss count: 0
Reset problem count: 0

Phy11:S attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 222822400
RD error count: 246415626
DW sync loss count: 0
Reset problem count: 0

Dumping configuration info for Expander C
-----------------------------------------

A) As you can see phys 7 and 3 of this expander are
connected to phys 3 and 2 of Expander B, thus forming
a wide port. This forms the subtractive boundary
of this expander.

B) End devices attached are clearly identifiable.

expander_conf /sys/devices/pci0000\:01/0000\:01\:04.0/host11/sas/ha/ports/1/domain/50001c1716010600/50001c1071609c00/5005076a000001e0/smp_portal

Vendor: VITESSE
Product: VSC7160 Eval Brd
Revision: 4
Component: VITESSE
Component ID: 29024
Component revision: 3
Expander Change Count: 4
Expander Route Indexes: 160
Number of phys: 13
Configuring: No
Configurable route table: Yes
Enclosure Logical Identifier: 00000000000000000000

Phy00:D attached: 5000c50000513385:01 chg count:01
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: SSP Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0

Phy01:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy02:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 2147352567
RD error count: 4294964735
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy03:S attached: 50001c1071609c00:02 chg count:01
Attached device: edge expander Link rate: G2 (3 GB/s)
Tproto: SMP Iproto: SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0

Phy04:S attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 1558851767
RD error count: 2009595383
DW sync loss count: 0
Reset problem count: 0

Phy05:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 4277141467
RD error count: 4223134719
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy06:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 2147483647
RD error count: 4026499071
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy07:S attached: 50001c1071609c00:03 chg count:01
Attached device: edge expander Link rate: G2 (3 GB/s)
Tproto: SMP Iproto: SMP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3243694232
RD error count: 749555105
DW sync loss count: 0
Reset problem count: 0

Phy08:S attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0

Phy09:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 4091248127
RD error count: 2936004581
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy10:T attached: 0000000000000000:00 chg count:00
Attached device: none Link rate: unknown
Tproto: Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 3758030847
RD error count: 4294963183
DW sync loss count: 0
Reset problem count: 0
Routing Table
Empty (all zero)

Phy11:D attached: 5c50000000409c11:00 chg count:01
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: SSP Iproto:
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Physical phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0

Phy12:D attached: 5005076a000001ed:00 chg count:00
Attached device: end device Link rate: G2 (3 GB/s)
Tproto: SSP Iproto: SSP
Programmed MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Hardware MIN-MAX linkrate: G1 (1,5 Gb/s) - G2 (3 GB/s)
Virtual phy
Partial pathway timeout value: 7 microseconds
Connector type: No information
Connector element index: 0
Connector physical link: 0
Invalid DW count: 0
RD error count: 0
DW sync loss count: 0
Reset problem count: 0

Christoph Hellwig

unread,

Sep 11, 2005, 5:21:04 AM9/11/05

to Luben Tuikov, Linux Kernel Mailing List, SCSI Mailing List

Thanks for finally posting your code.

At the core it's some really nice code dealing with host-based SAS
implementations. What's not nice is that it's not intgerating with the
SAS transport class I posted, it's duplicating things like LUN disocvery
from the SCSI core code, and adding it's own sysfs representation that's
very different from the way the SCSI core and transport classes do it.

Are you willing to work with us to intgerate it with the infrastructure
we have?

Luben Tuikov

unread,

Sep 12, 2005, 5:35:41 PM9/12/05

to Christoph Hellwig, Linux Kernel Mailing List, SCSI Mailing List

On 09/11/05 05:20, Christoph Hellwig wrote:
> Thanks for finally posting your code.
>
> At the core it's some really nice code dealing with host-based SAS
> implementations.

Thank you Christoph. Much appreciated.

> What's not nice is that it's not intgerating with the
> SAS transport class I posted,

I wish there was something I could do. HP and LSI
were aware of my efforts since the beginning of the year.

As well, you had a copy of my code July 14 this year,
long before starting your work on your SAS class for LSI and
HP (so its acceptance is guaranteed), after OLS.

We did meet at OLS and we did have the SAS BOF. I'm not sure
why you didn't want to work together?

> it's duplicating things like LUN disocvery

This is a much more involved subject than meets the eye.

> from the SCSI core code, and adding it's own sysfs representation that's
> very different from the way the SCSI core and transport classes do it.

Yes, it is time to evolve.

I've pointed out many times the shortcomings of expanding the
JB's "transport _attribute_ class" into a "transport layer" in
recent threads.

I'm sorry but over everything else, we need a common base,
(what you call "techno-gibberish") in order to see eye to eye.

> Are you willing to work with us to intgerate it with the infrastructure
> we have?

I'm sure you've already taken a closer look at the SAS code
I posted. Study it, read the spec, read the code again.

Let me know if I can help with anything.

Overall, MPT is very different in design than a disclosed
transport. Talk to HP, LSI and Dell and see what they think.

Luben

Andrew Patterson

unread,

Sep 12, 2005, 7:19:54 PM9/12/05

to Luben Tuikov, Christoph Hellwig, Linux Kernel Mailing List, SCSI Mailing List

On Mon, 2005-09-12 at 17:35 -0400, Luben Tuikov wrote:
> On 09/11/05 05:20, Christoph Hellwig wrote:
> > Thanks for finally posting your code.
> >
> > At the core it's some really nice code dealing with host-based SAS
> > implementations.
>
> Thank you Christoph. Much appreciated.
>
> > What's not nice is that it's not intgerating with the
> > SAS transport class I posted,
>
> I wish there was something I could do. HP and LSI
> were aware of my efforts since the beginning of the year.

I know I am going to regret getting pulled into this ;-(.

This effort started on April. Eric Moore, Mike Miller and I started
work on a SAS transport class and then later pulled Luben it at the
suggestion of Douglas Gilbert (if I remember correctly). We later
mutually agreed that Luben would take over the transport class work as
he seemed to have much more experience with this sort of thing. The
original idea was to implement a SAS transport class that would allow
the LSI and Adaptec driver to get into kernel.org (or others at the
time) and to find a way to get SDI/CSMI API's into the kernel without
the use of IOCTL's. Luben then went off on his own and came up with his
effectively Adaptec only solution.

>
> As well, you had a copy of my code July 14 this year,
> long before starting your work on your SAS class for LSI and
> HP (so its acceptance is guaranteed), after OLS.

HP never suggested that Christoph do a SAS transport layer. We were
happy to provide some equipment when we found out that he was working on
it. Please don't speak for HP. I am sure that LSI would prefer you
don't speak for them either.

>
> We did meet at OLS and we did have the SAS BOF. I'm not sure
> why you didn't want to work together?

If my memory serves correctly, there were 10-12 people at that BOF,
representing the SCSI kernel maintainers and all of the vendors
currently providing SAS hardware. Virtually everyone disagreed with
your implementation (which you indeed emailed shortly before the
conference) that would only work with one vendor's card. The suggestion
was made that you convert your code to various library layers so that it
would work with all vendors. A suggestion which it seems that you
continue to reject.

>
> > it's duplicating things like LUN disocvery
>
> This is a much more involved subject than meets the eye.
>
> > from the SCSI core code, and adding it's own sysfs representation that's
> > very different from the way the SCSI core and transport classes do it.
>
> Yes, it is time to evolve.
>
> I've pointed out many times the shortcomings of expanding the
> JB's "transport _attribute_ class" into a "transport layer" in
> recent threads.
>
> I'm sorry but over everything else, we need a common base,
> (what you call "techno-gibberish") in order to see eye to eye.
>
> > Are you willing to work with us to intgerate it with the infrastructure
> > we have?
>
> I'm sure you've already taken a closer look at the SAS code
> I posted. Study it, read the spec, read the code again.
>
> Let me know if I can help with anything.
>
> Overall, MPT is very different in design than a disclosed
> transport. Talk to HP, LSI and Dell and see what they think.

From HP's point of view (or mine at least). We would prefer something
that works with every vendors card. Not Adaptec's (or is it Luben's)
vision of the perfect card.

Andrew
--
Andrew Patterson
Hewlett-Packard

signature.asc

Christoph Hellwig

unread,

Sep 13, 2005, 6:14:34 AM9/13/05

to Luben Tuikov, Christoph Hellwig, Linux Kernel Mailing List, SCSI Mailing List

On Mon, Sep 12, 2005 at 05:35:04PM -0400, Luben Tuikov wrote:
> > What's not nice is that it's not intgerating with the
> > SAS transport class I posted,
>
> I wish there was something I could do. HP and LSI
> were aware of my efforts since the beginning of the year.

As was I. And the reason I wrote this upper layer is that you
clearly stated multiple times (at the SAS BOF and in mail) that
you're not interested in this upper layer.

> As well, you had a copy of my code July 14 this year,

That code didn't have anything that overlaps with the code I wrote.

> long before starting your work on your SAS class for LSI and
> HP (so its acceptance is guaranteed), after OLS.

Just in case it was clear: I'm paid for this transport class by Dell.
I don't have any contractural relationship with LSI or HP, although these
companies (like most sucessfull hardware vendors) know that giving hardware
to linux people active in the area they care about helps to get thos people
actually fixing things about instead of just bitching around..

> We did meet at OLS and we did have the SAS BOF. I'm not sure
> why you didn't want to work together?

I abosultely want to. To quote from my first minimal transport class
announcement mail:

"I hope this will integrate nicely with the top-down work Luben has done
once he finally releases it publically, but for now I think we should have
something so SAS drivers can go in the tree."

> > from the SCSI core code, and adding it's own sysfs representation that's
> > very different from the way the SCSI core and transport classes do it.
>
> Yes, it is time to evolve.
>
> I've pointed out many times the shortcomings of expanding the
> JB's "transport _attribute_ class" into a "transport layer" in
> recent threads.

We need both a transport class in the original sense aswell as a library
for host-based SAS HBAs, and they need to play together nicely - whatever
term you give to them.

> Overall, MPT is very different in design than a disclosed
> transport.

I know. And we still want to cover it with a common base for what we
can have common.

Luben Tuikov

unread,

Sep 13, 2005, 10:24:45 AM9/13/05

to Christoph Hellwig, Linux Kernel Mailing List, SCSI Mailing List

On 09/13/05 06:14, Christoph Hellwig wrote:
> On Mon, Sep 12, 2005 at 05:35:04PM -0400, Luben Tuikov wrote:
>
>>>What's not nice is that it's not intgerating with the
>>>SAS transport class I posted,
>>
>>I wish there was something I could do. HP and LSI
>>were aware of my efforts since the beginning of the year.
>
>
> As was I. And the reason I wrote this upper layer is that you
> clearly stated multiple times (at the SAS BOF and in mail) that
> you're not interested in this upper layer.

Well, I never really said "Not interested". There was just
nothing I could do. By that time I know from our
original LSI/HP/SAS list that those folks have moved away
from what I was doing, plus I had my own work to do.

As to your work, or integration or whatever you decide
to do, I'm willing to help in anyway I can.

>>As well, you had a copy of my code July 14 this year,
>
> That code didn't have anything that overlaps with the code I wrote.

Well, both source code are public, people can read the code themselves
and make that decision for themselves. There is no point us in discussing
this.

> Just in case it was clear: I'm paid for this transport class by Dell.
> I don't have any contractural relationship with LSI or HP, although these
> companies (like most sucessfull hardware vendors) know that giving hardware
> to linux people active in the area they care about helps to get thos people
> actually fixing things about instead of just bitching around..

Christoph, there is smart engineers in companies too, not only in the Linux
community.

>>We did meet at OLS and we did have the SAS BOF. I'm not sure
>>why you didn't want to work together?
>
>
> I abosultely want to. To quote from my first minimal transport class
> announcement mail:
>
> "I hope this will integrate nicely with the top-down work Luben has done
> once he finally releases it publically, but for now I think we should have
> something so SAS drivers can go in the tree."

I see.

> We need both a transport class in the original sense aswell as a library
> for host-based SAS HBAs, and they need to play together nicely - whatever
> term you give to them.

I don't mind using "transport attribute class" for JB's class,
and "SAS Transport Layer" for the recent code submission.

That is, by design, JB tried to unify _attributes_ across _all_
transports. Not an easy feat by any means, but we shall leave
that at that.

While, by design (and SAM), the SAS Transport Layer, literally
sits between the interconnect and SCSI Core (SAM to be).

This separation is clear and distinct.

>>Overall, MPT is very different in design than a disclosed
>>transport.
>
> I know. And we still want to cover it with a common base for what we
> can have common.

Yes, this is very noble to have and do.

Figuring out _what_ we can have common, in such different and
distinct engineering architectures, Open Transport vs. MPT,
is not an easy feat by any means.

I don't have an indepth knowlege in MPT as you do, clearly since
I don't have specs, etc, but if you have questions or concerns
please don't hesitate to email, I'd gladly help with anything I can.

Luben