On 12/2/2025 11:49 AM, Navdeep Parhar wrote:
> On 12/1/25 10:01 AM, mike tancsa wrote:
>> TL;DR. I can get 10G coming in and out on port 0, Port 1 can only
>> take traffic at ~ 2.5 Gb/s incoming, but can send out a full 10G.
>>
> Have you tried your tests with these two left to their default values?
> We don't have a 64KB allocator in the kernel (16KB is the max) so the
> first setting isn't even valid.
Hi Navdeep,
Thank you for looking!
Yes, I tried with no settings in /boot/loader.conf and same behaviour.
I was running with
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"
hw.cxgbe.pause_settings="0"
hw.cxgbe.attack_filter="1"
hw.cxgbe.drop_pkts_with_l3_errors="1"
before as this is just an nfs server, zrepl (receive zfs streams over
tcp) and inbound sftp connections. cc0 is the busiest interface and
regularly bursts close to full speed. cc0 has an mtu of 9000. cc1 has
the default MTU of 1500 on 4 different vlan interfaces. I also tried to
blast some udp packets across both links as a test and cc1 also
underperforms with some dropped packets (rx_ovflow2,rx_trunc2) where as
the same rate is 100% error free on cc0
>
> Please also provide the output of
>
> # sysctl dev.cc | egrep 'link|caps|pause'
sysctl dev.cc | egrep 'link|caps|pause'
dev.cc.1.stats.rx_pause: 0
dev.cc.1.stats.tx_pause: 0
dev.cc.1.lpacaps: 458752
dev.cc.1.acaps: 268435460
dev.cc.1.pcaps: 269418502
dev.cc.1.rcaps: 270532612
dev.cc.1.link_fec: 4<NO-FEC>
dev.cc.1.pause_settings: 0
dev.cc.1.linkdnrc: n/a
dev.cc.0.stats.rx_pause: 0
dev.cc.0.stats.tx_pause: 0
dev.cc.0.lpacaps: 458752
dev.cc.0.acaps: 268435460
dev.cc.0.pcaps: 269418502
dev.cc.0.rcaps: 270532612
dev.cc.0.link_fec: 4<NO-FEC>
dev.cc.0.pause_settings: 0
dev.cc.0.linkdnrc: n/a
>
> It is possible that the link on the good port has pause based flow
> control enabled and the other one doesn't.
On the quanta where cc0 is connected (the one without the slowdowns) the
pause counters dont seem to be incrementing. I just did a test and it was
802.3x Pause Frames Received................... 7939772
before and after.
Same with the problem port on the cisco that cc1 is on
cisco-core4#show int te1/1/1 | inc pau
0 watchdog, 31 multicast, 424569 pause input
0 lost carrier, 0 no carrier, 0 pause output
cisco-core4#show int te1/1/1 | inc pau
0 watchdog, 31 multicast, 424569 pause input
0 lost carrier, 0 no carrier, 0 pause output
cisco-core4#
cisco-core4#show int te1/1/4 | inc pau
0 watchdog, 88 multicast, 0 pause input
0 lost carrier, 0 no carrier, 0 pause output
cisco-core4#
Nothing special about the config
interface TenGigabitEthernet1/1/1
description backup4-cc1
switchport trunk allowed vlan 2-4,223
switchport mode trunk
spanning-tree portfast trunk
(cc0 interface)
(RackA-10G) #show interface ethernet 0/23
Total Packets Received (Octets)................ 164104553985987
Packets Received 64 Octets..................... 2785340519
Packets Received 65-127 Octets................. 3551885604
Packets Received 128-255 Octets................ 2720325376
Packets Received 256-511 Octets................ 2916456887
Packets Received 512-1023 Octets............... 3987305743
Packets Received 1024-1518 Octets.............. 516438495
Packets Received > 1518 Octets................. 0
Packets RX and TX 64 Octets....................
4127650421
Packets RX and TX 65-127 Octets................
2635703067
Packets RX and TX 128-255 Octets...............
2696715779
Packets RX and TX 256-511 Octets............... 1789991928
Packets RX and TX 512-1023 Octets.............. 1073841812
Packets RX and TX 1024-1518 Octets.............
3415095303
Packets RX and TX 1519-2047 Octets............. 1866698571
Packets RX and TX 2048-4095 Octets.............
2166348155
Packets RX and TX 4096-9216 Octets............. 2969664207
Total Packets Received Without Errors.......... 1204638026215
Unicast Packets Received....................... 1204629504771
Multicast Packets Received..................... 7939833
Broadcast Packets Received..................... 581611
--More-- or (q)uit
Total Packets Received with MAC Errors......... 0
Jabbers Received............................... 0
Fragments Received............................. 0
Undersize Received............................. 0
Alignment Errors............................... 0
FCS Errors..................................... 0
Overruns....................................... 0
Total Received Packets Not Forwarded........... 7939796
802.3x Pause Frames Received................... 7939772
Unacceptable Frame Type........................ 3
Total Packets Transmitted (Octets)............. 253100014214248
Packets Transmitted 64 Octets.................. 1342309902
Packets Transmitted 65-127 Octets..............
3378784759
Packets Transmitted 128-255 Octets............. 4271357699
Packets Transmitted 256-511 Octets.............
3168502337
Packets Transmitted 512-1023 Octets............ 1381503365
Packets Transmitted 1024-1518 Octets...........
2898656808
Packets Transmitted > 1518 Octets.............. 0
Max Frame Size................................. 9216
Total Packets Transmitted Successfully......... 828148377556
Unicast Packets Transmitted.................... 828055172598
Multicast Packets Transmitted.................. 92093622
Broadcast Packets Transmitted.................. 1111336
Total Transmit Errors.......................... 0
Total Transmit Packets Discarded............... 0
Single Collision Frames........................ 0
Multiple Collision Frames...................... 0
Excessive Collision Frames..................... 0
802.3x Pause Frames Transmitted................ 0
Time Since Counters Last Cleared............... 1768 day 23 hr 31 min 49
sec
>
> Regards,
> Navdeep
>