Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
using ConnectX card as Ethernet (mlxen)
414 views
Skip to first unread message
John Nielsen
Sep 24, 2012, 12:40:41 PM9/24/12
to freebsd...@freebsd.org
I have a machine running "FreeBSD 10.0-CURRENT #0 r240887" amd64 with two ConnectX (InfiniBand) cards. Relevant bits of dmesg and pciconf -lv below. The cards are connected directly to a 10GB Ethernet switch so I need to run them in "eth" mode rather than "ib". Unfortunately they come up in "ib" mode and I don't know how to change it.
The same hardware works fine under CentOS 6.3, though I need to manually set the cards to 'eth' there as well (which I do using a 'connectx_port_config script from Mellanox that twiddles the mlx4_port1 entries under /sys (sysfs). Under FreeBSD I see these sysctls but I can't set them to 'eth' either via /boot/loader.conf or by sysctl after boot, with or without mlxen and/or mlx4ib loaded:
sys.device.mlx4_core0.mlx4_port1: ib
sys.device.mlx4_core1.mlx4_port1: ib
Assuming mlxen is actually supported, how do I configure the card so it will attach?
mlx4_core0: <mlx4_core> mem 0xdfa00000-0xdfafffff,0xdd800000-0xddffffff irq 32 at device 0.0 on pci4
mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.2 (August 4, 2010)
mlx4_core: Initializing mlx4_core
mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.2 (July 2010)
mlx4_en mlx4_core0: UDP RSS is not supported on this device.
mlx4_core1: <mlx4_core> mem 0xdf900000-0xdf9fffff,0xdd000000-0xdd7fffff irq 42 at device 0.0 on pci7
mlx4_core: Initializing mlx4_core
mlx4_core0@pci0:4:0:0: class=0x0c0600 card=0x002215b3 chip=0x673c15b3 rev=0xb0 hdr=0x00
vendor = 'Mellanox Technologies'
device = 'MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]'
class = serial bus
mlx4_core1@pci0:7:0:0: class=0x028000 card=0x001715b3 chip=0x100315b3 rev=0x00 hdr=0x00
vendor = 'Mellanox Technologies'
device = 'MT27500 Family [ConnectX-3]'
class = network
Thanks,
JN
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
The same hardware works fine under CentOS 6.3, though I need to manually set the cards to 'eth' there as well (which I do using a 'connectx_port_config script from Mellanox that twiddles the mlx4_port1 entries under /sys (sysfs). Under FreeBSD I see these sysctls but I can't set them to 'eth' either via /boot/loader.conf or by sysctl after boot, with or without mlxen and/or mlx4ib loaded:
sys.device.mlx4_core0.mlx4_port1: ib
sys.device.mlx4_core1.mlx4_port1: ib
Assuming mlxen is actually supported, how do I configure the card so it will attach?
mlx4_core0: <mlx4_core> mem 0xdfa00000-0xdfafffff,0xdd800000-0xddffffff irq 32 at device 0.0 on pci4
mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.2 (August 4, 2010)
mlx4_core: Initializing mlx4_core
mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.2 (July 2010)
mlx4_en mlx4_core0: UDP RSS is not supported on this device.
mlx4_core1: <mlx4_core> mem 0xdf900000-0xdf9fffff,0xdd000000-0xdd7fffff irq 42 at device 0.0 on pci7
mlx4_core: Initializing mlx4_core
mlx4_core0@pci0:4:0:0: class=0x0c0600 card=0x002215b3 chip=0x673c15b3 rev=0xb0 hdr=0x00
vendor = 'Mellanox Technologies'
device = 'MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]'
class = serial bus
mlx4_core1@pci0:7:0:0: class=0x028000 card=0x001715b3 chip=0x100315b3 rev=0x00 hdr=0x00
vendor = 'Mellanox Technologies'
device = 'MT27500 Family [ConnectX-3]'
class = network
Thanks,
JN
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
John Baldwin
Jul 9, 2013, 12:13:00 PM7/9/13
to freebsd...@freebsd.org, John Nielsen
On Monday, September 24, 2012 12:37:30 pm John Nielsen wrote:
> I have a machine running "FreeBSD 10.0-CURRENT #0 r240887" amd64 with two
ConnectX (InfiniBand) cards. Relevant bits of dmesg and pciconf -lv below. The
cards are connected directly to a 10GB Ethernet switch so I need to run them
in "eth" mode rather than "ib". Unfortunately they come up in "ib" mode and I
don't know how to change it.
>
> The same hardware works fine under CentOS 6.3, though I need to manually set
the cards to 'eth' there as well (which I do using a 'connectx_port_config
script from Mellanox that twiddles the mlx4_port1 entries under /sys (sysfs).
Under FreeBSD I see these sysctls but I can't set them to 'eth' either via
/boot/loader.conf or by sysctl after boot, with or without mlxen and/or mlx4ib
loaded:
> sys.device.mlx4_core0.mlx4_port1: ib
> sys.device.mlx4_core1.mlx4_port1: ib
So this was just fixed (finally) in HEAD in r253048. You can how use the
> I have a machine running "FreeBSD 10.0-CURRENT #0 r240887" amd64 with two
ConnectX (InfiniBand) cards. Relevant bits of dmesg and pciconf -lv below. The
cards are connected directly to a 10GB Ethernet switch so I need to run them
in "eth" mode rather than "ib". Unfortunately they come up in "ib" mode and I
don't know how to change it.
>
> The same hardware works fine under CentOS 6.3, though I need to manually set
the cards to 'eth' there as well (which I do using a 'connectx_port_config
script from Mellanox that twiddles the mlx4_port1 entries under /sys (sysfs).
Under FreeBSD I see these sysctls but I can't set them to 'eth' either via
/boot/loader.conf or by sysctl after boot, with or without mlxen and/or mlx4ib
loaded:
> sys.device.mlx4_core0.mlx4_port1: ib
> sys.device.mlx4_core1.mlx4_port1: ib
sysctls to change this.
--
John Baldwin
John Nielsen
Jul 9, 2013, 4:10:47 PM7/9/13
to John Baldwin, freebsd...@freebsd.org
On Jul 9, 2013, at 9:58 AM, John Baldwin <j...@freebsd.org> wrote:
> On Monday, September 24, 2012 12:37:30 pm John Nielsen wrote:
>> I have a machine running "FreeBSD 10.0-CURRENT #0 r240887" amd64 with two
> ConnectX (InfiniBand) cards. Relevant bits of dmesg and pciconf -lv below. The
> cards are connected directly to a 10GB Ethernet switch so I need to run them
> in "eth" mode rather than "ib". Unfortunately they come up in "ib" mode and I
> don't know how to change it.
>>
>> The same hardware works fine under CentOS 6.3, though I need to manually set
> the cards to 'eth' there as well (which I do using a 'connectx_port_config
> script from Mellanox that twiddles the mlx4_port1 entries under /sys (sysfs).
> Under FreeBSD I see these sysctls but I can't set them to 'eth' either via
> /boot/loader.conf or by sysctl after boot, with or without mlxen and/or mlx4ib
> loaded:
>> sys.device.mlx4_core0.mlx4_port1: ib
>> sys.device.mlx4_core1.mlx4_port1: ib
>
> So this was just fixed (finally) in HEAD in r253048. You can how use the
> sysctls to change this.
I saw the commit. Thanks! I'll give it a try at some point (whenever my schedule and hardware availability align).
> On Monday, September 24, 2012 12:37:30 pm John Nielsen wrote:
>> I have a machine running "FreeBSD 10.0-CURRENT #0 r240887" amd64 with two
> ConnectX (InfiniBand) cards. Relevant bits of dmesg and pciconf -lv below. The
> cards are connected directly to a 10GB Ethernet switch so I need to run them
> in "eth" mode rather than "ib". Unfortunately they come up in "ib" mode and I
> don't know how to change it.
>>
>> The same hardware works fine under CentOS 6.3, though I need to manually set
> the cards to 'eth' there as well (which I do using a 'connectx_port_config
> script from Mellanox that twiddles the mlx4_port1 entries under /sys (sysfs).
> Under FreeBSD I see these sysctls but I can't set them to 'eth' either via
> /boot/loader.conf or by sysctl after boot, with or without mlxen and/or mlx4ib
> loaded:
>> sys.device.mlx4_core0.mlx4_port1: ib
>> sys.device.mlx4_core1.mlx4_port1: ib
>
> So this was just fixed (finally) in HEAD in r253048. You can how use the
> sysctls to change this.
JN
Eggert, Lars
Jan 20, 2014, 7:08:52 AM1/20/14
to John Nielsen, freebsd...@freebsd.org
Hi,
On 2013-7-9, at 22:08, John Nielsen <li...@jnielsen.net> wrote:
On Jul 9, 2013, at 9:58 AM, John Baldwin <j...@freebsd.org> wrote:
>> So this was just fixed (finally) in HEAD in r253048. You can how use the
>> sysctls to change this.
>
> I saw the commit. Thanks! I'll give it a try at some point (whenever my schedule and hardware availability align).
is this supposed to work at the moment? When I try, the machine seems to crash:
root@one:~ # sysctl sys.device.mlx4_core0.mlx4_port1=eth
sys.device.mlx4_core0.mlx4_port1: auto (ib)
Write failed: Broken pipe
Shared connection to xxx closed.
Unfortunately I don't have serial console access at the moment, so I can't access any messages that may have gotten dumped.
The cards in question are:
mlx4_core0@pci0:17:0:0: class=0x028000 card=0x005015b3 chip=0x100315b3 rev=0x00 hdr=0x00
On 2013-7-9, at 22:08, John Nielsen <li...@jnielsen.net> wrote:
On Jul 9, 2013, at 9:58 AM, John Baldwin <j...@freebsd.org> wrote:
>> So this was just fixed (finally) in HEAD in r253048. You can how use the
>> sysctls to change this.
>
> I saw the commit. Thanks! I'll give it a try at some point (whenever my schedule and hardware availability align).
root@one:~ # sysctl sys.device.mlx4_core0.mlx4_port1=eth
sys.device.mlx4_core0.mlx4_port1: auto (ib)
Write failed: Broken pipe
Shared connection to xxx closed.
Unfortunately I don't have serial console access at the moment, so I can't access any messages that may have gotten dumped.
The cards in question are:
mlx4_core0@pci0:17:0:0: class=0x028000 card=0x005015b3 chip=0x100315b3 rev=0x00 hdr=0x00
vendor = 'Mellanox Technologies'
device = 'MT27500 Family [ConnectX-3]'
class = network
Lars
device = 'MT27500 Family [ConnectX-3]'
class = network
John Baldwin
Jan 20, 2014, 4:24:52 PM1/20/14
to Eggert, Lars, freebsd...@freebsd.org, John Nielsen
would be really helpful in figuring out why it isn't. Thanks.
--
John Baldwin
Eggert, Lars
Jan 21, 2014, 4:05:04 AM1/21/14
to John Baldwin, freebsd...@freebsd.org, John Nielsen
Hi,
On 2014-1-20, at 21:59, John Baldwin <j...@freebsd.org> wrote:
> I believe this should work, yes. Getting a crashdump or the panic messages
> would be really helpful in figuring out why it isn't. Thanks.
I rebuilt the kernel, and see no crashes anymore. So that's good.
But there are a bunch of other issues that maybe someone has some ideas about:
(1) Late attach
The ConnectX-3 attaches very late during the boot process, after the system is already in single-user mode. See the attached dmesg; pci17 and pci18 (there are two identical cards in this system) first show as "no driver attached" during the PCI bus enumeration. Only after the system is single-user mode does the mlx4_core attach to the cards.
That means that e.g. trying to set sysctls for these cards in /etc/sysctl.conf, or configuring their IP addresses via rc.conf is not possible. At the moment, I work around this by sleeping in rc.local and then doing assignments there, but that's a hack.
Any clues why these cards attach so late?
(2) Device numbers change
After booting, these cards show up in InfiniBand mode:
ib0: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d1.21
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib1: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d1.22
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib2: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib3: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d2
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Then I force one into Ethernet mode:
# sysctl sys.device.mlx4_core0.mlx4_port1=eth
sys.device.mlx4_core0.mlx4_port1: auto (ib) -> eth
and the device numbers on the ib devices change: ib1 is now ib4, and I have a new mlxen0 device.
ib2: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib3: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d2
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:21
inet6 fe80::f652:14ff:fe10:d121%mlxen0 prefixlen 64 scopeid 0xe
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
ib4: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.4a.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d1.22
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
When I change another port into Ethernet mode
# sysctl sys.device.mlx4_core0.mlx4_port2=eth
sys.device.mlx4_core0.mlx4_port2: auto (ib) -> eth
device numbers change again. Now mxlen0 disappears and becomes mxlen1, and I have a new mxlen2 device:
ib2: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib3: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d2
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:21
inet6 fe80::f652:14ff:fe10:d121%mlxen1 prefixlen 64 scopeid 0xe
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:22
inet6 fe80::f652:14ff:fe10:d122%mlxen2 prefixlen 64 scopeid 0xf
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
Changing the other two ports (on the second card) to Ethernet mode
# sysctl sys.device.mlx4_core1.mlx4_port1=eth
sys.device.mlx4_core1.mlx4_port1: auto (ib) -> eth
# sysctl sys.device.mlx4_core1.mlx4_port2=eth
sys.device.mlx4_core1.mlx4_port2: auto (ib) -> eth
leaves me with mlxen1, mlxen2, mlxen4 and mlxen 5:
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:21
inet6 fe80::f652:14ff:fe10:d121%mlxen1 prefixlen 64 scopeid 0xe
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:22
inet6 fe80::f652:14ff:fe10:d122%mlxen2 prefixlen 64 scopeid 0xf
inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (40Gbase-CR4 <full-duplex,rxpause,txpause>)
status: active
mlxen4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d0:d1
inet6 fe80::f652:14ff:fe10:d0d1%mlxen4 prefixlen 64 scopeid 0x10
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
mlxen5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d0:d2
inet6 fe80::f652:14ff:fe10:d0d2%mlxen5 prefixlen 64 scopeid 0x11
inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
Needless to say, having devices change numbers is problematic.
(3) 40G TCP performance
I barely get over 10G with netperf over the 40G interfaces:
root@one:~ # netperf -H two-mlxen2 -- -s512k -S512K
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to two-mlxen2.muclab () port 0 AF_INET : histogram : interval : dirty data : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
524288 512000 512000 10.07 10268.01
Any clues as to what could be limiting performance here?
Thanks,
Lars
On 2014-1-20, at 21:59, John Baldwin <j...@freebsd.org> wrote:
> I believe this should work, yes. Getting a crashdump or the panic messages
> would be really helpful in figuring out why it isn't. Thanks.
But there are a bunch of other issues that maybe someone has some ideas about:
(1) Late attach
The ConnectX-3 attaches very late during the boot process, after the system is already in single-user mode. See the attached dmesg; pci17 and pci18 (there are two identical cards in this system) first show as "no driver attached" during the PCI bus enumeration. Only after the system is single-user mode does the mlx4_core attach to the cards.
That means that e.g. trying to set sysctls for these cards in /etc/sysctl.conf, or configuring their IP addresses via rc.conf is not possible. At the moment, I work around this by sleeping in rc.local and then doing assignments there, but that's a hack.
Any clues why these cards attach so late?
(2) Device numbers change
After booting, these cards show up in InfiniBand mode:
ib0: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d1.21
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib1: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d1.22
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib2: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib3: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d2
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Then I force one into Ethernet mode:
# sysctl sys.device.mlx4_core0.mlx4_port1=eth
sys.device.mlx4_core0.mlx4_port1: auto (ib) -> eth
and the device numbers on the ib devices change: ib1 is now ib4, and I have a new mlxen0 device.
ib2: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib3: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d2
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:21
inet6 fe80::f652:14ff:fe10:d121%mlxen0 prefixlen 64 scopeid 0xe
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
ib4: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.4a.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d1.22
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
When I change another port into Ethernet mode
# sysctl sys.device.mlx4_core0.mlx4_port2=eth
sys.device.mlx4_core0.mlx4_port2: auto (ib) -> eth
device numbers change again. Now mxlen0 disappears and becomes mxlen1, and I have a new mxlen2 device:
ib2: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d1
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib3: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d2
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:21
inet6 fe80::f652:14ff:fe10:d121%mlxen1 prefixlen 64 scopeid 0xe
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:22
inet6 fe80::f652:14ff:fe10:d122%mlxen2 prefixlen 64 scopeid 0xf
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
Changing the other two ports (on the second card) to Ethernet mode
# sysctl sys.device.mlx4_core1.mlx4_port1=eth
sys.device.mlx4_core1.mlx4_port1: auto (ib) -> eth
# sysctl sys.device.mlx4_core1.mlx4_port2=eth
sys.device.mlx4_core1.mlx4_port2: auto (ib) -> eth
leaves me with mlxen1, mlxen2, mlxen4 and mlxen 5:
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:21
inet6 fe80::f652:14ff:fe10:d121%mlxen1 prefixlen 64 scopeid 0xe
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d1:22
inet6 fe80::f652:14ff:fe10:d122%mlxen2 prefixlen 64 scopeid 0xf
inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (40Gbase-CR4 <full-duplex,rxpause,txpause>)
status: active
mlxen4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d0:d1
inet6 fe80::f652:14ff:fe10:d0d1%mlxen4 prefixlen 64 scopeid 0x10
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
mlxen5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether f4:52:14:10:d0:d2
inet6 fe80::f652:14ff:fe10:d0d2%mlxen5 prefixlen 64 scopeid 0x11
inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
Needless to say, having devices change numbers is problematic.
(3) 40G TCP performance
I barely get over 10G with netperf over the 40G interfaces:
root@one:~ # netperf -H two-mlxen2 -- -s512k -S512K
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to two-mlxen2.muclab () port 0 AF_INET : histogram : interval : dirty data : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
524288 512000 512000 10.07 10268.01
Any clues as to what could be limiting performance here?
Thanks,
Lars
Eggert, Lars
Jan 21, 2014, 4:05:59 AM1/21/14
to John Baldwin, freebsd...@freebsd.org, John Nielsen
On 2014-1-21, at 10:04, Lars Eggert <la...@netapp.com> wrote:
> See the attached dmesg
which I of course forget to attach (sigh). See below.
Lars
> See the attached dmesg
which I of course forget to attach (sigh). See below.
Lars
Eggert, Lars
Jan 21, 2014, 4:31:53 AM1/21/14
to John Baldwin, freebsd...@freebsd.org, John Nielsen
Last follow-up: I just saw that there are some additional messages (errors?) on the serial console when changing the device from IB to Ethernet, maybe they mean something to someone:
root@one:~ # sysctl sys.device.mlx4_core0.mlx4_port1=eth
sys.device.mlx4_core0.mlx4_port1: auto (ib)<7>ib0: stopping interface
<7>ib0: downing ib_dev
<7>ib0: stopping multicast thread
<7>ib0: flushing multicast list
<7>"qpn 0x48: invalid attribute mask specified " "for transition 0 to 6.
qp_type 4," " attr_mask 0x1\n"<4>ib0: Failed to modify QP to ERROR state
<7>ib0: All sends and receives done.
<7>ib0: cleaning up ib_dev
<7>ib0: stopping multicast thread
<7>ib0: flushing multicast list
<7>ib0: Cleanup ipoib connected mode.
<7>ib1: stopping interface
<7>ib1: downing ib_dev
<7>ib1: stopping multicast thread
<7>ib1: flushing multicast list
<7>"qpn 0x49: invalid attribute mask specified " "for transition 0 to 6.
qp_type 4," " attr_mask 0x1\n"<4>ib1: Failed to modify QP to ERROR state
<7>ib1: All sends and receives done.
<7>ib1: cleaning up ib_dev
<7>ib1: stopping multicast thread
<7>ib1: flushing multicast list
<7>ib1: Cleanup ipoib connected mode.
<6>mlx4_en mlx4_core0: Using 5 tx rings for port:1
<6>mlx4_en mlx4_core0: Defaulting to 4 rx rings for port:1
<6>mlx4_en mlx4_core0: Activating port:1
mlxen0: Ethernet address: f4:52:14:10:d1:21
<4>mlx4_en: mlx4_core0: Port 1: Using 5 TX rings
<4>mlx4_en: mlx4_core0: Port 1: Using 4 RX rings
<6>mlx4_ib: Mellanox ConnectX InfiniBand driver v1.Jan 21 09:21:31 0
(April 4, 2008)
one kernel: mlx4_en: mlx4_core0: Port 1: Using 5 TX rings
Jan <7>ib4: max_srq_sge=31
21 09:21:31 one <7>ib4: max_cm_mtu = 0x10000, num_frags=16
kernel: mlx4_en:ib4: mlx4_core0: PorAttached to mlx4_0 port 2
t 1: Using 4 RX rings
-> eth
Lars
root@one:~ # sysctl sys.device.mlx4_core0.mlx4_port1=eth
sys.device.mlx4_core0.mlx4_port1: auto (ib)<7>ib0: stopping interface
<7>ib0: downing ib_dev
<7>ib0: stopping multicast thread
<7>ib0: flushing multicast list
<7>"qpn 0x48: invalid attribute mask specified " "for transition 0 to 6.
qp_type 4," " attr_mask 0x1\n"<4>ib0: Failed to modify QP to ERROR state
<7>ib0: All sends and receives done.
<7>ib0: cleaning up ib_dev
<7>ib0: stopping multicast thread
<7>ib0: flushing multicast list
<7>ib0: Cleanup ipoib connected mode.
<7>ib1: stopping interface
<7>ib1: downing ib_dev
<7>ib1: stopping multicast thread
<7>ib1: flushing multicast list
<7>"qpn 0x49: invalid attribute mask specified " "for transition 0 to 6.
qp_type 4," " attr_mask 0x1\n"<4>ib1: Failed to modify QP to ERROR state
<7>ib1: All sends and receives done.
<7>ib1: cleaning up ib_dev
<7>ib1: stopping multicast thread
<7>ib1: flushing multicast list
<7>ib1: Cleanup ipoib connected mode.
<6>mlx4_en mlx4_core0: Using 5 tx rings for port:1
<6>mlx4_en mlx4_core0: Defaulting to 4 rx rings for port:1
<6>mlx4_en mlx4_core0: Activating port:1
mlxen0: Ethernet address: f4:52:14:10:d1:21
<4>mlx4_en: mlx4_core0: Port 1: Using 5 TX rings
<4>mlx4_en: mlx4_core0: Port 1: Using 4 RX rings
<6>mlx4_ib: Mellanox ConnectX InfiniBand driver v1.Jan 21 09:21:31 0
(April 4, 2008)
one kernel: mlx4_en: mlx4_core0: Port 1: Using 5 TX rings
Jan <7>ib4: max_srq_sge=31
21 09:21:31 one <7>ib4: max_cm_mtu = 0x10000, num_frags=16
kernel: mlx4_en:ib4: mlx4_core0: PorAttached to mlx4_0 port 2
t 1: Using 4 RX rings
-> eth
Lars
0 new messages