PCIe multipliers, how do they work?

[Dieter BSD]

Looking into ways to get additional expansion slots, since board
designers cannot count past 7 and often not even that high. :-(

There are PCIe riser cards that split a wide slot into 2 or more
narrower slots, for example 1 x8 slot becomes 2 x4 slots. These
would be very useful, except it appears that they require
"bifurcation" support in the mainboard's firmware. Which most
boards do not provide. And most boards are not supported by
FLOSS firmware, so adding bifurcation support would be rather
difficult.

There are also PCIe cards which provide multiple slots, typically
connected with a usb cable. These tend to convert 1 PCIe_x1 slot
into multiple PCIe_x1 slots. I get the impression that these do
not require bifurcation support. They seem to be aimed at "miners"
for attaching multiple gpu cards. I'm not interested in mining or
in attaching multiple gpu cards. I'm interesting in adding additional
sata cards, Ethernet cards, and such. Unlike the bifurcation type
riser splitter cards, which seem to conserve PCIe lanes, these
are more like a sata port multiplier, with the same type of bandwidth
limitation.

I'm wondering how these things work. The wikipedia PCIe page [1] says:
"PCI Express switches can create multiple endpoints out of one endpoint
to allow sharing one endpoint with multiple devices." So maybe they
use a PCIe switch? Poking around wikipedia and google has thus far
uncovered very little info about PCIe switches. Wikipedia is less
helpful than usual, and they keep making google less and less useful
for no apparent reason. I don't see any other obvious keywords
to google for.

It isn't obvious how slot id/address is handled. How do commands and
data get routed to/from the correct card?

Is any firmware or OS support required?

Is there some other solution that I haven't stumbled across?
I'd really like to split an x8 slot into 4 x2 slots, which
doesn't seem to be an off-the-shelf option either way.

[1] en.wikipedia.org/wiki/PCI_Express
_______________________________________________
freebsd-...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hardware
To unsubscribe, send any mail to "freebsd-hardwa...@freebsd.org"

[Wojciech Puchar]

> There are also PCIe cards which provide multiple slots, typically
> connected with a usb cable. These tend to convert 1 PCIe_x1 slot
> into multiple PCIe_x1 slots. I get the impression that these do
> not require bifurcation support. They seem to be aimed at "miners"

these cards consist of PCIe switch which is supported out of the box.
former ones require BIOS to reconfigure CPUs PCI lanes so instead of eg
one 16x lane there will be 4 4x lanes.

[Warner Losh]

On Fri, May 11, 2018 at 1:03 PM, Wojciech Puchar <woj...@puchar.net> wrote:

> There are also PCIe cards which provide multiple slots, typically
>> connected with a usb cable. These tend to convert 1 PCIe_x1 slot
>> into multiple PCIe_x1 slots. I get the impression that these do
>> not require bifurcation support. They seem to be aimed at "miners"
>>
>
> these cards consist of PCIe switch which is supported out of the box.
> former ones require BIOS to reconfigure CPUs PCI lanes so instead of eg
> one 16x lane there will be 4 4x lanes.
>

Usually they require a driver for management functions, like if you want to
turn off one of the ports (because you know there's a bad card in it, for
example). But for normal probe / attach they are usually drop in. Some
mobos can require some BIOS adjustment, though to properly setup lane
bifurcation and such... The ones that are true switches, and not just
signal retimers, usually don't: they take the full 8 or 16 lanes and
multiplex it amongst the 16-32 downstream lanes they typically provide.
We're looking at one that does x4 lanes and expands to 4 cards at x4 lanes.
The devices on the other side fill just over a lane each, but we'd run out
of lanes if we did x2 fan-out. This the cards we're looking at, we can get
x4 combined rates form the 4 cards that on their own are kinda crappy.

Warner

[Kurt Lidl]

> I'm wondering how these things work. The wikipedia PCIe page [1] says:
> "PCI Express switches can create multiple endpoints out of one endpoint
> to allow sharing one endpoint with multiple devices." So maybe they
> use a PCIe switch? Poking around wikipedia and google has thus far
> uncovered very little info about PCIe switches. Wikipedia is less
> helpful than usual, and they keep making google less and less useful
> for no apparent reason. I don't see any other obvious keywords
> to google for.
>
> It isn't obvious how slot id/address is handled. How do commands and
> data get routed to/from the correct card?
>
> Is any firmware or OS support required?

It's worth reading the following two pages closely, as they talk
about PCI Express endpoints, and how PCI-Express switches have to
work to enable multiple cards.

http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-1
http://xillybus.com/tutorials/pci-express-tlp-pcie-primer-tutorial-guide-2

One such family of PCI-express switch parts is from PLX Technologies,
now owned by Broadcom. One of typical functions of such as switch
is to provide "non transparent bridging" - basically mapping the
address space from one machine's PCI address into the address space
of another PCI system. This can be used a communication path between
devices, for example, using a block and slot type software architecture.

There is some support for NTB in FreeBSD's master branch, under
sys/dev/ntb - with drivers for the PLX/Avago/Broadcom parts, as well
as a driver for the Intel parts found on certain Xeon and Atom CPUs.

-Kurt