Realtek re(4) driver

[Dieter BSD]

Multiple people have found that FreeBSD's re(4) driver for Realtek
Ethernet chips has problems. Something like a rcp(1) with another
FreeBSD box merely runs slower due to the stalls, but if the other end
has buggy network code and/or if the transfer is forced to use UDP
(Unreliable Data Protocol) data is lost. :-(

Rumor has it that Realtek has a driver that works properly with FreeBSD.

FreeBSD's developers are apparently unable to:
a) Fix the existing FreeBSD re(4) device driver so that it works correctly.
b) Replace the existing FreeBSD re(4) device driver with Realtek's driver.
c) Make Realtek's driver into a port/pkg.
d) At least add a warning to the re(4) man page, with the URL for
Realtek's driver.

Therefore every user with Realtek Ethernet chips are forced to somehow
discover that Realtek has a working driver, then somehow obtain a copy
of this mythical driver, get it to compile and build a custom kernel.
They then need to write a test suite and verify that Realtek's driver
does in fact work properly for their applications.

I am currently suffering data loss due to this. Therefore I am
attempting to obtain a copy of the mythical Realtek device driver.
I managed to find

> http://www.realtek.com.tw/Downloads/downloadsView.aspx?Langid=4&PNid=13&PFid=5&Level=5&Conn=4&DownTypeID=3&GetDown=false

But it does not work for me. It has a listing for

> Description Version Update Time File Size Download Site 1
> FreeBSD 7.x and 8.0 1.94 2017/9/15 93k Global

The machine with the re ports is running 10.3, hopefully the 7&8 code
will compile and work on 10. But the real problem is that clicking on
"Global" does not work. Some browsers do nothing. Firefox creates
a new window with http://www.realtek.com.tw/downloads/
which is going backwards. I was expecting something like a tar file
containing the source code.

I also tried google, and emailing a FreeBSD user that runs the Realtek
driver, but both pointed me to the above non-working URL. I'm
told that seamonkey works, but...

So my question is: could someone please tell me the URL that points
*directly* to the mythical Realtek device driver? (without
the broken javascheist garbage)
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

[Steven Hartland]

Try:
http://12244.wpc.azureedge.net/8012244/drivers/rtdrivers/cn/nic/0007-rtl_bsd_drv_v194.01.tgz

May or many not work depending on if its a per client download URL

[Gary Jennejohn]

On Wed, 11 Apr 2018 00:05:59 +0100
Steven Hartland <kil...@multiplay.co.uk> wrote:

> Try:
> http://12244.wpc.azureedge.net/8012244/drivers/rtdrivers/cn/nic/0007-rtl_bsd_drv_v194.01.tgz
>
> May or many not work depending on if its a per client download URL
>

This is the old if_rl driver by Bill Paul from 1998. Hard to
believe it's really superior to the newer version, which is also
based on later code from Bill Paul.

[snip]

--
Gary Jennejohn

[BERTRAND Joël]

Gary Jennejohn a écrit :

> On Wed, 11 Apr 2018 00:05:59 +0100
> Steven Hartland <kil...@multiplay.co.uk> wrote:
>
>> Try:
>> http://12244.wpc.azureedge.net/8012244/drivers/rtdrivers/cn/nic/0007-rtl_bsd_drv_v194.01.tgz
>>
>> May or many not work depending on if its a per client download URL
>>
>
> This is the old if_rl driver by Bill Paul from 1998. Hard to
> believe it's really superior to the newer version, which is also
> based on later code from Bill Paul.
>
> [snip]

I use a diskless workstation. With re driver provided by FreeBSD kernel
(9/10/11.x), system randomly crashed because ethernet driver stalls (and
datarate is always less than 300mbps). With official realtek driver
(v194.01), system now runs as expected (with datarate up to 1 Gbps).

Internal re driver is broken on this ethernet adapter (maybe on several
other adapters) :

re0@pci0:2:0:0: class=0x020000 card=0x78511462 chip=0x816810ec rev=0x0c
hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

Regards,

JKB

[David Wolfskill]

On Wed, Apr 11, 2018 at 12:32:00PM +0200, BERTRAND Joël wrote:
> ...

> I use a diskless workstation. With re driver provided by FreeBSD kernel
> (9/10/11.x), system randomly crashed because ethernet driver stalls (and
> datarate is always less than 300mbps). With official realtek driver
> (v194.01), system now runs as expected (with datarate up to 1 Gbps).
>
> Internal re driver is broken on this ethernet adapter (maybe on several
> other adapters) :
>
> re0@pci0:2:0:0: class=0x020000 card=0x78511462 chip=0x816810ec rev=0x0c
> hdr=0x00
> vendor = 'Realtek Semiconductor Co., Ltd.'
> device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
> class = network
> subclass = ethernet
>
> Regards,
>
> JKB

> ....

In counterpoint, my build machine for the last few years has been a Dell
mini-tower I bought from Costco... with its original disk drive removed
(replaced by an SSD, then supplemented with 3 more for a poudriere
scratch-space). It runs a GENERIC kernel, tracking stable/11 & head
daily, using poudriere to build local packages weekly, and acting as an
NFS server for my "production" machines for their weekly updates.

Its NIC:

re0@pci0:3:0:0: class=0x020000 card=0x05b71028 chip=0x816810ec rev=0x0c hdr=0x00

vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

freebeast(11.1-S)[2] ifconfig re0
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
ether 98:90:96:d6:c9:6d
hwaddr 98:90:96:d6:c9:6d
inet 172.16.8.10 netmask 0xffffff00 broadcast 172.16.8.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex,master>)
status: active
freebeast(11.1-S)[3]

(It is connected to a "dumb" 16-port Netgear gigabit switch.)

I will admit that I avoid having the machine try to do too much
else while the production machines are using it as an NFS server
(serving /usr/src & /usr/obj, while (e.g.) "make installworld" runs
on the production machines), but other than that, I have no issues
involving performance and behavior of the NIC and its driver.

Peace,
david
--
David H. Wolfskill da...@catwhisker.org
Well, what did you EXPECT from Trump? He has a history of breaking promises.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.

[Vladimir Terziev]

Hi Dieter,

i have managed to download the driver from the link that you provided by just clicking the “Global” link.

Just let me know if you want me to send it to you.


Regards,

Vladimir

> freeb...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net...@freebsd.org"

[Alex Dupre]

David Wolfskill wrote:
>> I use a diskless workstation. With re driver provided by FreeBSD kernel
>> (9/10/11.x), system randomly crashed because ethernet driver stalls (and
>> datarate is always less than 300mbps). With official realtek driver
>> (v194.01), system now runs as expected (with datarate up to 1 Gbps).

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=166724

Anyone interested in taking the bounty or contributing to it? Perhaps
the FreeBSD Foundation should jump in?

--
Alex Dupre

[Gary Jennejohn]

On Wed, 11 Apr 2018 12:51:09 +0200
Alex Dupre <a...@FreeBSD.org> wrote:

> David Wolfskill wrote:
> >> I use a diskless workstation. With re driver provided by FreeBSD kernel
> >> (9/10/11.x), system randomly crashed because ethernet driver stalls (and
> >> datarate is always less than 300mbps). With official realtek driver
> >> (v194.01), system now runs as expected (with datarate up to 1 Gbps).
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=166724
>
> Anyone interested in taking the bounty or contributing to it? Perhaps
> the FreeBSD Foundation should jump in?
>

Reading the bugzilla shows that this is a comples problem with no
obvious error source.

One poster notes that he could transfer 300+GB of data error-free
using rsync, but with NFS he encountered problesm after a few GB.
To me this is an indictment of the NFS implementation used and
not of the re driver itself.

The same goes for some other posts. Everyone blames re, but AFAICT
no-one switched to a non-Realtek chip to run tests and prove that,
yes, re is really the cause of all the network problems.

I've been using re for many years in various incarnations - PCIe
cards and integrated on the mainboard. I've never experienced
any network instabilities. But I've never used one in a NFS
server under FBSD. I do, however, have one in a Linux NFS server
(which was at one time my primary FBSD box) and never observed
any problems.

In summary, this is one tough nut to crack.

--
Gary Jennejohn

[Rodney W. Grimes]

> David Wolfskill wrote:
> >> I use a diskless workstation. With re driver provided by FreeBSD kernel
> >> (9/10/11.x), system randomly crashed because ethernet driver stalls (and
> >> datarate is always less than 300mbps). With official realtek driver
> >> (v194.01), system now runs as expected (with datarate up to 1 Gbps).
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=166724

I have put that bug back on a public bug list (n...@freebsd.org) so
that it appears in the nag mails sent out periodically.

I think I might have some of that hardware around here, but not sure
if it is that specific chip. I do know that some "re(4)" cards
work just fine with FreeBSD, but others have issues, I suspect the
ones that have issues are ones that have hardware bugs that need
a specific software work around.

I do have this:
re0@pci0:2:0:0: class=0x020000 card=0x84321043 chip=0x816810ec rev=0x06 hdr=0x00

vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'

and I use that pretty regular, including NFS, but rarely push more than
1 or 2 gigabyte through it in any one operation. (src tree size chunks go
in and out of that box).

> Anyone interested in taking the bounty or contributing to it? Perhaps
> the FreeBSD Foundation should jump in?
>
> --
> Alex Dupre

--
Rod Grimes rgr...@freebsd.org

[Mike Tancsa]

On 4/11/2018 10:12 AM, Rodney W. Grimes wrote:
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=166724
>
> I have put that bug back on a public bug list (n...@freebsd.org) so
> that it appears in the nag mails sent out periodically.
>
> I think I might have some of that hardware around here, but not sure
> if it is that specific chip. I do know that some "re(4)" cards
> work just fine with FreeBSD, but others have issues, I suspect the
> ones that have issues are ones that have hardware bugs that need
> a specific software work around.

Over the years I have had mixed results with re nics. Some work as
expected, others with issues. I think a big part of the problem is that
there are so many varieties out there. That being said, I run into
similar issues on Linux from time to time with these NICs, so its not
just a FreeBSD problem. Just the other week, on a new MSI RYZEN board
(MS-7A34), I found I could wedge the onboard NIC without too much effort
doing some network stress tests. (kernel is 4.4.0-119). Havent tried
this board on FreeBSD however.

---Mike



--
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, mi...@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada

[Alex Dupre]

Gary Jennejohn wrote:
> Reading the bugzilla shows that this is a comples problem with no
> obvious error source.
>
> One poster notes that he could transfer 300+GB of data error-free
> using rsync, but with NFS he encountered problesm after a few GB.
> To me this is an indictment of the NFS implementation used and
> not of the re driver itself.

I agree that is a complex problem with a non obvious solution, I have
also used many re cards successfully and others not, but surely it's not
a NFS issue. I can tell you more, I've used successfully a re card for
years, and started seeing issues with the same card after upgrading my
upstream connection to gigabit. The only thing that link all bug reports
(and there are tons) is the use of re card sustaining gigabit (duplex)
transfers. And I couldn't find similar reports for other chipsets that
might lead to think it's a generic FreeBSD issue. Everything is
possible, but the main suspect is the re driver.

--
Alex Dupre

[Gary Jennejohn]

On Wed, 11 Apr 2018 17:19:18 +0200
Alex Dupre <a...@FreeBSD.org> wrote:

> Gary Jennejohn wrote:
> > Reading the bugzilla shows that this is a comples problem with no
> > obvious error source.
> >
> > One poster notes that he could transfer 300+GB of data error-free
> > using rsync, but with NFS he encountered problesm after a few GB.
> > To me this is an indictment of the NFS implementation used and
> > not of the re driver itself.
> I agree that is a complex problem with a non obvious solution, I have
> also used many re cards successfully and others not, but surely it's not
> a NFS issue. I can tell you more, I've used successfully a re card for
> years, and started seeing issues with the same card after upgrading my
> upstream connection to gigabit. The only thing that link all bug reports
> (and there are tons) is the use of re card sustaining gigabit (duplex)
> transfers. And I couldn't find similar reports for other chipsets that
> might lead to think it's a generic FreeBSD issue. Everything is
> possible, but the main suspect is the re driver.
>

One thing which did seem common was checksum offloading. My cards
don't seem to support that.

My Linux NFS server is serving clients with 1Gbps links - never
seen a problem. But the MTU is limited to 1500.

The re in my FBSD box uses a MTU of 4808. I've transferred lots
of data with a Linux device and a Windows machine also using that
setting and never saw a problem. Maybe I'm just lucky.

The Realtech Linux driver could be a place to start looking for useful
changes.

--
Gary Jennejohn

[Allan Jude]

I think it would be useful to collect the 'pciconf -lv' output of the
re(4) devices that have issues, so they can be differentiated from the
ones that work fine.

This would be the first step to getting a developer setup with a
reliable reproduction platform, so the bug could be tracked down.

--
Allan Jude

[Rodney W. Grimes]

> I think it would be useful to collect the 'pciconf -lv' output of the
> re(4) devices that have issues, so they can be differentiated from the
> ones that work fine.

IIRC one of the "issues" with re type devies is that you can have
the exact same PCI vendor and device ID, but they are NOT the same
device. You have to read some additional realtek specific hwrev
register to know which card you actuall have. That is why another
person in this thread asked for dmesg output, as the driver decodes
these bits and spits out a proper info line.

Search for RL_HWREV in both if_rl.c and if_re.c, the re gets
this info from the if_rl.h header.

SO if you want to submit your pciconf -lv,
PLEASE include the dmesg output for the device.

>
> This would be the first step to getting a developer setup with a
> reliable reproduction platform, so the bug could be tracked down.

--

Rod Grimes rgr...@freebsd.org

[BERTRAND Joël]

Rodney W. Grimes a écrit :

> SO if you want to submit your pciconf -lv,
> PLEASE include the dmesg output for the device.

I don't know if it's enough as I cannot reboot with FreeBSD officiel
driver. I use Realtek one :

Mar 24 21:35:06 pythagore kernel: re0: <Realtek PCIe GBE Family
Controller> port 0xe000-0xe0ff mem
0xf7d00000-0xf7d00fff,0xf0000000-0xf0003fff irq 19 at device 0.0 on pci2
Mar 24 21:35:06 pythagore kernel: re0: Using Memory Mapping!
Mar 24 21:35:06 pythagore kernel: re0: Using 1 MSI-X message
Mar 24 21:35:06 pythagore kernel: re0: version:1.94.01
Mar 24 21:35:06 pythagore kernel: re0: Ethernet address: d8:cb:8a:7d:10:59
Mar 24 21:35:06 pythagore kernel: re0: Ethernet address: d8:cb:8a:7d:10:59
Mar 24 21:35:06 pythagore kernel: re0: link state changed to UP

This adapter doesn't work as expected on diskless workstation with
kernel driver. No problem with Realtek driver.

re0@pci0:2:0:0: class=0x020000 card=0x78511462 chip=0x816810ec rev=0x0c

hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'

class = network
subclass = ethernet

Regards,

JKB

[Peter]

Allan Jude wrote:

> I think it would be useful to collect the 'pciconf -lv' output of the
> re(4) devices that have issues, so they can be differentiated from the
> ones that work fine.

Yes, but that would mean a structured approach, and not everybody loves
that. *vbeg*

I remember, in the old times there was a rule: on 100Mb never do
autonegotiate, on 1Gb always do autonegotiate - but that may vary
depending on device. I would think, that option, as well as respective
offloading capabilities, should also carefully checked for influences.

I have one of these pieces here, but it runs in 100Mb mode and sadly I
have no proper counterpart in reach to do full-speed testing.

[Steven Hartland]

On 11/04/2018 22:01, BERTRAND Joël wrote:
> Rodney W. Grimes a écrit :
>> SO if you want to submit your pciconf -lv,
>> PLEASE include the dmesg output for the device.
> I don't know if it's enough as I cannot reboot with FreeBSD officiel
> driver. I use Realtek one :
>
> Mar 24 21:35:06 pythagore kernel: re0: <Realtek PCIe GBE Family
> Controller> port 0xe000-0xe0ff mem
> 0xf7d00000-0xf7d00fff,0xf0000000-0xf0003fff irq 19 at device 0.0 on pci2
> Mar 24 21:35:06 pythagore kernel: re0: Using Memory Mapping!
> Mar 24 21:35:06 pythagore kernel: re0: Using 1 MSI-X message
> Mar 24 21:35:06 pythagore kernel: re0: version:1.94.01
> Mar 24 21:35:06 pythagore kernel: re0: Ethernet address: d8:cb:8a:7d:10:59
> Mar 24 21:35:06 pythagore kernel: re0: Ethernet address: d8:cb:8a:7d:10:59
> Mar 24 21:35:06 pythagore kernel: re0: link state changed to UP
>
> This adapter doesn't work as expected on diskless workstation with
> kernel driver. No problem with Realtek driver.
>
> re0@pci0:2:0:0: class=0x020000 card=0x78511462 chip=0x816810ec rev=0x0c
> hdr=0x00
> vendor = 'Realtek Semiconductor Co., Ltd.'
> device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
> class = network
> subclass = ethernet
>

I know you said you couldn't do this, but it may well be very useful to
compare this to the output when using the built in FreeBSD driver as it
may highlight a key difference.

I'm thinking given the ages of the Realtek driver we could be looking at
a different operational mode, different MSI-X count etc.

Here's some other links with potentially relevant info:
https://forums.freebsd.org/threads/replacing-realtek-re-driver.55861/
https://unixblogger.com/2011/10/18/the-pain-of-an-realtek-rtl8111rtl8168-ethernet-card/

    Regards
    Steve

[Farhan Khan]

On Wed, Apr 11, 2018 at 5:22 PM, Steven Hartland <kil...@multiplay.co.uk>
wrote:

Hi all, I would love to assist and perform testing in any way I can. I have
2 devices running Realtek devices. One is a 4-port NIC. If we can
narrow-down the issue and create a reproduceable failure, I might be able
to hunt through the code to help identify the problem (I am primarily
tapped on rtwn(4), but I would not mind lending an extra pair of eye-balls)

Laptop machine running 12.0-CURRENT:

re0@pci0:3:0:0: class=0x020000 card=0x21de103c chip=0x813610ec rev=0x08

hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'

device = 'RTL8101/2/6E PCI Express Fast Ethernet controller'

class = network
subclass = ethernet

Think Server running 11.1-RELEASE-p8:

re0@pci0:4:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x07

hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

re1@pci0:5:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x07

hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

re2@pci0:6:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x07

hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

re3@pci0:7:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x07

hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

pcib9@pci0:8:0:0: class=0x060401 card=0x30a517aa chip=0x88931283 rev=0x41
hdr=0x01
vendor = 'Integrated Technology Express, Inc.'
device = 'IT8893E PCIe to PCI Bridge'
class = bridge
subclass = PCI-PCI

[Dieter BSD]

One problem I see with the Realtek Ethernet and the stock FreeBSD
device driver is occasional pauses in Ethernet traffic.
Observe with:
netstat -w 1 -I re2

With reasonably decent network software and TCP, this is only a minor
problem. However with buggy network software (like a "black box"
with closed source firmware and maybe a transmit buffer that is *way*
too small, and/or true real time requirements), and/or with a brain dead
protocol like UDP, you can lose data.

So I decided to try and find a way to demonstrate the problem using
only 2 normal computers running FreeBSD, one with Realtek Ethernet.
The easy way seemed to be to transfer a large file using UDP and
look for errors.

On machine with Realtek Ethernet:
nc -4nul 55555 > /var/tmp/big_file_copied_via_udp
On some other machine:
nc -4u machine_with_realtek 55555 < big_file

If you don't like 55555, pick another port number. ;-)

I first tried a 2.9 GB file, which worked ok a few times,
then I tried a 28 GB file and the copy was smaller than the original.
Tried it a couple more times and also got different sizes which were
smaller than the original. Then transferred it twice with tcp,
getting the correct size both times, and the two copies passed cmp(1).

original: 28132560000 bytes
udp copy 1: 28131929216 (smaller)
udp copy 2: 28132523136 (smaller)
udp copy 3: 28132537472 (smaller)
tcp copy 1: 28132560000 (correct)
tcp copy 2: 28132560000 (correct)

Ran time(1) on nc (using tcp):
real 5m56.591s -> 78,893,073 B/s

Next, transfer the opposite direction, with Realtek transmitting
and non-Realtek receiving.

Transferred with tcp, got the correct file size, and the copy passed cmp(1)
against the original file.

Attempts to transfer using udp seem to run into some sort of flow control
problem, the nc processes hang. Sometimes a small amount of data gets through.

Both machines are amd64 with ECC memory
RTL8111F and stock FreeBSD 10.4 re(4)
The other machine has nfe(4) running FreeBSD 8.2

re0@pci0:4:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x07 hdr=0x00

re1@pci0:5:0:0: class=0x020000 card=0x012310ec chip=0x816810ec rev=0x07 hdr=0x00

re2@pci0:12:0:0: class=0x020000 card=0xe0001458 chip=0x816810ec
rev=0x0c hdr=0x00

re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet>
re0: Chip rev. 0x2c800000
re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet>
re1: Chip rev. 0x2c800000
re2: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet>
re2: Chip rev. 0x4c000000
=================================================
Steven writes:
>> http://12244.wpc.azureedge.net/8012244/drivers/rtdrivers/cn/nic/0007-rtl_bsd_drv_v194.01.tgz

Gary writes:
> This is the old if_rl driver by Bill Paul from 1998. Hard to
> believe it's really superior to the newer version, which is also
> based on later code from Bill Paul.

The URL worked for me (using wget). 96,127 bytes. Thank you, Steven!

tar tvzf 0007-rtl_bsd_drv_v194.01.tgz
drwxrwxrwx 0 0 0 0 Dec 26 2016 rtl_bsd_drv_v194.01/
-rwxrwxrwx 0 0 0 1186892 Aug 29 2017 rtl_bsd_drv_v194.01/if_re.c
-rwxrwxrwx 0 0 0 37071 Aug 25 2017 rtl_bsd_drv_v194.01/if_rereg.h
-rwxrwxrwx 0 0 0 200 Jun 14 2016 rtl_bsd_drv_v194.01/Makefile
-rwxrwxrwx 0 0 0 3172 Jan 3 2017 rtl_bsd_drv_v194.01/Readme.txt

Huh? What is this about 1998? Looks like 2017-08-29 to me.
The files are newer than FreeBSD 10.3's sources:

-r--r--r-- 1 root wheel 111335 Mar 24 2016 re/if_re.c

However, I have RTL8111E on UD5 mainboard, and 2x RTL8111F on PCIe card.
The FreeBSD sources attempt to support these, but the Realtek sources
do not mention these revisions, which seems odd when it is 1.5 years newer.

# grep -i 8111e re/* rtl_bsd_drv_v194.01/*
re/if_re.c: { RL_HWREV_8168E, RL_8169, "8168E/8111E", RL_JUMBO_MTU_9K},
re/if_re.c: { RL_HWREV_8168E_VL, RL_8169, "8168E/8111E-VL",
RL_JUMBO_MTU_6K},
re/if_re.c: { RL_HWREV_8168EP, RL_8169, "8168EP/8111EP", RL_JUMBO_MTU_9K},

# grep -i 8111f re/* rtl_bsd_drv_v194.01/*
re/if_re.c: { RL_HWREV_8168F, RL_8169, "8168F/8111F", RL_JUMBO_MTU_9K},

Which leaves me wondering if the Reaktek code supports the chips I have or not.

Readme.txt says
> for FreeBSD v4.x/5.x/6.x/7.x/8.x/9.x

No mention of 10 or newer, which, again, seems odd when it is 1.5 years
newer than 10.3. May or may not be a problem.

[Dieter BSD]

Built a kernel with Realtek's Ethernet driver.
The 8111E & 8111F work, although initial testing using nc(1) is being
a bit strange.

1st testing window:
re0 I tried the same copy a 28 GB file using nc(1) and udp.
The 1st & 3rd try gave the correct number of bytes, but
the 2nd try was short.
The other 2 ports (re1 re2) worked less well.

2nd testing window:
re0 (8111F on Syba 2 port PCIe card [1])
6 times 28 GB file using nc and udp 3 "plain" and 3 with -T reliability
1 "plain" copy worked correctly, the other 5 were short
re1 (2nd 8111F port on PCIe card)
all 5 attempts were short, 3 of 5 were very short:
28132560000 goal
27982918784 big_file_copied_via_udp_re1_0
27981835392 big_file_copied_via_udp_re1_1
587776 big_file_copied_via_udp_re1_2
608256 big_file_copied_via_udp_re1_3
606208 big_file_copied_via_udp_re1_Trel_0
re2 (8111E on UD5 mainboard)
all 6 attempts were short
32452608 big_file_copied_via_udp_re2rt_0
3932160 big_file_copied_via_udp_re2rt_1
399364096 big_file_copied_via_udp_re2_Trel_0
5963776 big_file_copied_via_udp_re2_Trel_1
4159488 big_file_copied_via_udp_re2_Trel_2
4827136 big_file_copied_via_udp_re2_2
28132560000 goal

One burst and then it fails:
input re2 output
packets errs idrops bytes packets errs bytes colls
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
3840 0 0 4078080 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0

For real world use, so far the Realtek driver is working at least as well
as the stock FreeBSD 10.3 driver, probably better. But it does lose data
with real world applications, (2 separate incidents so far) so it needs
improvement.

Some of the test results with nc(1) are kinda screwy. Maybe nc is the
wrong tool, maybe I'm using it wrong, I don't know.

Has anyone tried similar testing?
Suggestions of knobs to turn, or other things to try?

I need to receive data via UDP without dropping packets. Closed source sucks,
but I'm stuck with it, and thus with UDP.

[1] http://www.sybausa.com/index.php?route=product/product&manufacturer_id=11&product_id=130&page=25
$21.22

[Stefan Esser]

Am 16.04.18 um 23:23 schrieb Dieter BSD:

> test results with nc(1) are kinda screwy. Maybe nc is the
> wrong tool, maybe I'm using it wrong, I don't know.
>
> Has anyone tried similar testing?
> Suggestions of knobs to turn, or other things to try?
>
> I need to receive data via UDP without dropping packets. Closed source sucks,
> but I'm stuck with it, and thus with UDP.

Well, but you know that the U in UDP means unreliable?

On a non-congested link, UDP packets should arrive at the receiver,
but the application has to deal with packet loss, since it cannot
be sure that it is used on a reliable link. Packet loss will cause
reduced throughput (as with TCP), unless it can be dealt with by FEC
or other means (which also reduce throughput).

Regards, STefan

[Dieter BSD]

With several days more data, the Realtek driver is slightly different
than the stock FreeBSD 10.3 driver, but it still fails a lot, with both
TCP and UDP.

> Has anyone tried similar testing?
> Suggestions of knobs to turn, or other things to try?

> I need to receive data via UDP without dropping packets.
> Closed source sucks, but I'm stuck with it, and thus with UDP.

Gary typed:

>> Everyone blames re, but AFAICT
>> no-one switched to a non-Realtek chip to run tests and prove that,
>> yes, re is really the cause of all the network problems.

I have a box with nfe(4) and bge(4) (BCM5750) that can receive data
with UDP or TCP without error. This has been working for several years.
But a newer, faster machine with re(4) 8111E & 8111F cannot. The nfe is
in the chipset, and the BCM5750 is onboard. I had google look for an
expansion card with BCM5750 but found nothing. I'm looking into
a Broadcom 5719 card, but there is at least one open PR against it, so...

Alex typed:

> surely it's not a NFS issue

NFS is a bug. By design. "Not a File System"
My machines are NFS-free.

> The only thing that link all bug reports
> (and there are tons) is the use of re card sustaining gigabit (duplex)
> transfers.

My re run (1000baseT <full-duplex>) but the pauses happen even with
light traffic. You don't have to run them full blast with rcp/ftp/whatever
to get failures.

STefan typed:

> Well, but you know that the U in UDP means unreliable?

Well, :-) you didn't pay attention when I said exactly that:

>>> Something like a rcp(1) with another
>>> FreeBSD box merely runs slower due to the stalls, but if the other end
>>> has buggy network code and/or if the transfer is forced to use UDP
>>> (Unreliable Data Protocol) data is lost. :-(

>> With reasonably decent network software and TCP, this is only a minor
>> problem. However with buggy network software (like a "black box"
>> with closed source firmware and maybe a transmit buffer that is *way*
>> too small, and/or true real time requirements), and/or with a brain dead
>> protocol like UDP, you can lose data.

rcp(1) to 8111F using Realtek driver
Both machines basically idle.
mtu=9000 on both ends but appears to be using 1500 anyway?
netstat -w 1 -d -I re0

input re0 output
packets errs idrops bytes packets errs bytes colls drops
5 0 0 330 6 0 755 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
17454 0 0 26176844 17453 0 1152264 0 0
67883 0 0 101023187 67880 0 4481248 0 0
52133 0 0 77141014 52140 0 3444580 0 0
72410 0 0 107980584 72409 0 4780384 0 0
80736 0 0 120645177 80735 0 5328810 0 0
71898 0 0 107298972 71899 0 4745634 0 0
14452 0 0 21510681 14475 0 955577 0 0
5 0 0 330 6 0 755 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
4 0 0 264 4 0 564 0 0
65323 0 0 97533279 65321 0 4311552 0 0
28383 0 0 42398766 28385 0 1873769 0 0
80759 0 0 120315366 80756 0 5330394 0 0
80992 0 0 120693553 80995 0 5345772 0 0
80932 0 0 120692488 80930 0 5341746 0 0
78905 0 0 117912275 78922 0 5209152 0 0
80158 0 0 119592452 80159 0 5290728 0 0
input re0 output
packets errs idrops bytes packets errs bytes colls drops
80851 0 0 120590446 80852 0 5336591 0 0
80849 0 0 120589243 80847 0 5336334 0 0
80858 0 0 120598524 80860 0 5336928 0 0
15085 0 0 22529747 15107 0 997296 0 0
4 0 0 264 4 0 564 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
4 0 0 264 4 0 564 0 0
63944 0 0 95447185 63943 0 4220604 0 0
80588 0 0 120276192 80589 0 5319233 0 0
80721 0 0 120480130 80722 0 5328011 0 0
79870 0 0 119139685 79870 0 5271720 0 0
79425 0 0 118587402 79442 0 5243472 0 0
80609 0 0 120333747 80609 0 5320494 0 0
80852 0 0 120624944 80850 0 5336532 0 0
80827 0 0 120555558 80831 0 5335073 0 0
79185 0 0 118137499 79185 0 5226510 0 0
76417 0 0 114252562 76431 0 5045010 0 0
79896 0 0 119330393 79900 0 5273436 0 0
79455 0 0 118627478 79456 0 5244330 0 0
77667 0 0 116033902 77662 0 5126256 0 0
input re0 output
packets errs idrops bytes packets errs bytes colls drops
79517 0 0 118735507 79515 0 5248547 0 0
78557 0 0 117304898 78578 0 5186118 0 0
79607 0 0 118742143 79625 0 5255616 0 0
79776 0 0 119140720 79775 0 5265450 0 0
75105 0 0 112725906 75107 0 4957230 0 0
79214 0 0 118774181 79214 0 5228424 0 0
13090 0 0 19520508 13113 0 865692 0 0

Same as above except to 2nd 8111F port.

input re1 output
packets errs idrops bytes packets errs bytes colls drops
75170 0 0 112288950 75165 0 4960956 0 0
74149 0 0 110808698 74150 0 4893834 0 0
75088 0 0 112276960 75088 0 4955808 0 0
74567 0 0 111313160 74566 0 4921356 0 0
77064 0 0 115370280 77065 0 5086290 0 0
71606 0 0 107343414 71622 0 4727052 0 0
70831 0 0 105956278 70832 0 4674849 0 0
70727 0 0 106127094 70725 0 4667916 0 0
73427 0 0 110111478 73425 0 4846182 0 0
72519 0 0 108763678 72522 0 4786320 0 0
5661 0 0 8423884 5681 0 374880 0 0

0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0

2 0 0 196 0 0 0 0 0
1 0 0 98 0 0 0 0 0
563 0 0 838984 561 0 37026 0 0
29144 0 0 43754296 29142 0 1923438 0 0
73986 0 0 110909436 73987 0 4883142 0 0
39601 0 0 59343834 39621 0 2614920 0 0

0 0 0 0 0 0 0 0 0

3 0 0 201 0 0 0 0 0
input re1 output
packets errs idrops bytes packets errs bytes colls drops

0 0 0 0 0 0 0 0 0

1 0 0 98 0 0 0 0 0
567 0 0 839958 567 0 37422 0 0
1 0 0 98 0 0 0 0 0
38003 0 0 56780696 38000 0 2508066 0 0
75439 0 0 112523550 75438 0 4978908 0 0
77058 0 0 114890108 77059 0 5085894 0 0
77471 0 0 115473190 77470 0 5113020 0 0
76814 0 0 114644092 76815 0 5069790 0 0
78120 0 0 116633730 78118 0 5155788 0 0
74853 0 0 111727082 74854 0 4940298 0 0
77626 0 0 115697756 77623 0 5123250 0 0
70804 0 0 105582200 70804 0 4672998 0 0
76722 0 0 114492148 76722 0 5063652 0 0
29562 0 0 44157654 29582 0 1952346 0 0

0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0

568 0 0 840298 566 0 37356 0 0

0 0 0 0 0 0 0 0 0

input re1 output
packets errs idrops bytes packets errs bytes colls drops
43130 0 0 64590180 43128 0 2846514 0 0
29010 0 0 43374204 29032 0 1916046 0 0

0 0 0 0 0 0 0 0 0

2 0 0 134 0 0 0 0 0

0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0

515 0 0 770942 515 0 33990 0 0
17903 0 0 26853958 17902 0 1181598 0 0
74854 0 0 111699988 74851 0 4940166 0 0
77120 0 0 115061536 77117 0 5089722 0 0
76857 0 0 114664362 76855 0 5072562 0 0
75817 0 0 113174146 75820 0 5003988 0 0
76737 0 0 114435914 76737 0 5064642 0 0
78495 0 0 117128814 78493 0 5180604 0 0
75723 0 0 112970948 75724 0 4997718 0 0
76501 0 0 114196604 76500 0 5049066 0 0
76187 0 0 113779382 76189 0 5028408 0 0
77236 0 0 115269384 77235 0 5097510 0 0
75300 0 0 112389128 75300 0 4969800 0 0
76492 0 0 114455568 76492 0 5048472 0 0
75573 0 0 113011418 75572 0 4987752 0 0
input re1 output
packets errs idrops bytes packets errs bytes colls drops
73331 0 0 109541752 73330 0 4839780 0 0
76732 0 0 114631816 76733 0 5064378 0 0
74400 0 0 111067456 74399 0 4910334 0 0
75974 0 0 113463012 75975 0 5014350 0 0
76312 0 0 113999448 76311 0 5036526 0 0
43806 0 0 65415884 43844 0 2893638 0 0
1 0 0 66 0 0 0 0 0

0 0 0 0 0 0 0 0 0

Same as above except to onboard 8111E.

input re2 output
packets errs idrops bytes packets errs bytes colls drops
53122 0 0 80345556 0 0 0 0 0
53984 0 0 81332624 0 0 0 0 0
61394 0 0 92785548 0 0 0 0 0
53421 0 0 80696090 0 0 0 0 0
30857 0 0 46597698 0 0 0 0 0
76124 0 0 115239504 0 0 0 0 0
69172 0 0 104706904 0 0 0 0 0
76094 0 0 115191956 0 0 0 0 0
76849 0 0 116329874 0 0 0 0 0
68788 0 0 104120096 0 0 0 0 0
69233 0 0 104802146 0 0 0 0 0
50527 0 0 76465702 0 0 0 0 0
75916 0 0 114920192 0 0 0 0 0
66062 0 0 99990092 0 0 0 0 0
76257 0 0 115436194 0 0 0 0 0
47074 0 0 71233028 0 0 0 0 0
51788 0 0 78359104 0 0 0 0 0
44486 0 0 67316276 0 0 0 0 0
76159 0 0 115288566 0 0 0 0 0
64766 0 0 98037852 0 0 0 0 0
76786 0 0 116237604 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
50676 0 0 76688424 0 0 0 0 0
76918 0 0 116440932 0 0 0 0 0
65723 0 0 99488694 0 0 0 0 0
73756 0 0 111653288 0 0 0 0 0
71393 0 0 108070674 0 0 0 0 0
63089 0 0 95485346 0 0 0 0 0
69588 0 0 105333248 0 0 0 0 0
59215 0 0 89617670 0 0 0 0 0
57339 0 0 86752950 0 0 0 0 0
75739 0 0 114651942 0 0 0 0 0
68442 0 0 103594692 0 0 0 0 0
49990 0 0 75649100 0 0 0 0 0
27710 0 0 41940348 0 0 0 0 0
50005 0 0 75694058 0 0 0 0 0
76748 0 0 116183384 0 0 0 0 0
52088 0 0 78847168 0 0 0 0 0
76816 0 0 116287048 0 0 0 0 0
70232 0 0 106301624 0 0 0 0 0
76467 0 0 115754502 0 0 0 0 0
68979 0 0 104412918 0 0 0 0 0
76694 0 0 116101068 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
68857 0 0 104224114 0 0 0 0 0
76545 0 0 115871154 0 0 0 0 0
65996 0 0 99843912 0 0 0 0 0
76674 0 0 116070772 0 0 0 0 0
68095 0 0 103070230 0 0 0 0 0
74877 0 0 113319642 0 0 0 0 0
39228 0 0 59366824 0 0 0 0 0
53026 0 0 80203068 0 0 0 0 0

...

77799 0 0 117779926 0 0 0 0 0
72666 0 0 110007220 0 0 0 0 0
69173 0 0 104681578 0 0 0 0 0
55918 0 0 84169172 0 0 0 0 0
24127 0 0 36195390 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops

0 0 0 0 0 0 0 0 0

19706 0 0 29578468 0 0 0 0 0
53262 0 0 79974084 0 0 0 0 0
59033 0 0 88835122 0 0 0 0 0
54469 0 0 81830514 0 0 0 0 0
55005 0 0 82842706 0 0 0 0 0
47867 0 0 72331934 0 0 0 0 0
65468 0 0 99099624 0 0 0 0 0
80572 0 0 121980632 0 0 0 0 0
58201 0 0 88091802 0 0 0 0 0
51948 0 0 78614144 0 0 0 0 0
80053 0 0 121196242 0 0 0 0 0
52331 0 0 79200470 0 0 0 0 0
80123 0 0 121302582 0 0 0 0 0
43414 0 0 65698001 0 0 0 0 0
77961 0 0 117995130 0 0 0 0 0
62287 0 0 94226926 0 0 0 0 0
75558 0 0 114290892 0 0 0 0 0
64228 0 0 97158320 0 0 0 0 0
59981 0 0 90654186 0 0 0 0 0
43520 0 0 65739096 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
56570 0 0 85335700 0 0 0 0 0
50066 0 0 75384196 0 0 0 0 0
55959 0 0 84239934 0 0 0 0 0
53192 0 0 80101232 0 0 0 0 0
49002 0 0 73982980 0 0 0 0 0
49713 0 0 74993370 0 0 0 0 0
43984 0 0 66011800 0 0 0 0 0
54324 0 0 81935448 0 0 0 0 0
48502 0 0 72734828 0 0 0 0 0
26191 0 0 39411806 0 0 0 0 0

0 0 0 0 0 0 0 0 0

42582 0 0 64021036 0 0 0 0 0
47216 0 0 70880104 0 0 0 0 0
52239 0 0 78498398 0 0 0 0 0
52517 0 0 79279690 0 0 0 0 0
44212 0 0 66384032 0 0 0 0 0
57225 0 0 86126642 0 0 0 0 0
47902 0 0 72238300 0 0 0 0 0
44933 0 0 67508656 0 0 0 0 0
54967 0 0 82411976 0 0 0 0 0
50724 0 0 76288600 0 0 0 0 0

...

input re2 output
packets errs idrops bytes packets errs bytes colls drops
56426 0 0 84726988 0 0 0 0 0
53332 0 0 79840392 0 0 0 0 0
52763 0 0 78943630 0 0 0 0 0
52190 0 0 78016892 0 0 0 0 0
56889 0 0 85597554 0 0 0 0 0
55265 0 0 82915642 0 0 0 0 0
52162 0 0 78061690 0 0 0 0 0
58041 0 0 87326532 0 0 0 0 0
53244 0 0 79740088 0 0 0 0 0
51387 0 0 76791862 0 0 0 0 0
54187 0 0 81106878 0 0 0 0 0
51846 0 0 77541604 0 0 0 0 0
42316 0 0 63282584 0 0 0 0 0

0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0

590 0 0 890908 0 0 0 0 0
40253 0 0 60411290 0 0 0 0 0
56653 0 0 85264434 0 0 0 0 0
53499 0 0 80125838 0 0 0 0 0
[ ... ]
input re2 output
packets errs idrops bytes packets errs bytes colls drops
52839 0 0 78935238 0 0 0 0 0
53098 0 0 79307900 0 0 0 0 0
49559 0 0 74231830 0 0 0 0 0
79905 0 0 120972890 0 0 0 0 0
66836 0 0 101174008 0 0 0 0 0
75308 0 0 113998016 0 0 0 0 0
47258 0 0 71543932 0 0 0 0 0
32 0 0 48448 0 0 0 0 0
70026 0 0 106002764 0 0 0 0 0
63937 0 0 96604786 0 0 0 0 0
60947 0 0 92062662 0 0 0 0 0
63772 0 0 96410208 0 0 0 0 0
58129 0 0 87582154 0 0 0 0 0
59204 0 0 89190072 0 0 0 0 0
61375 0 0 92741230 0 0 0 0 0
69159 0 0 104622734 0 0 0 0 0
75748 0 0 114661800 0 0 0 0 0
75478 0 0 114253980 0 0 0 0 0
76814 0 0 116282228 0 0 0 0 0
76931 0 0 116458670 0 0 0 0 0
75299 0 0 113979718 0 0 0 0 0

[Borja Marcos]

> On 11 Apr 2018, at 17:19, Alex Dupre <a...@FreeBSD.org> wrote:
> I agree that is a complex problem with a non obvious solution, I have
> also used many re cards successfully and others not, but surely it's not
> a NFS issue.

I have this interface in an Intel NUC,

re0@pci0:3:0:0: class=0x020000 card=0x20678086 chip=0x816810ec rev=0x15 hdr=0x00

vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

and I have had serious problems running an iSCSI initiator with some work load. I tried both the
builtin “re” driver in FreeBSD and the 1.94.01 version driver available from the Realtek website
with similar results.

More or less the same workload on a different machine with Intel “em” cards and
the same FreeBSD version works.

Borja.

[Borja Marcos]

> On 11 Apr 2018, at 19:34, Allan Jude <alla...@freebsd.org> wrote:
>
> I think it would be useful to collect the 'pciconf -lv' output of the
> re(4) devices that have issues, so they can be differentiated from the
> ones that work fine.

There we go. The machine is an Intel NUC6CAYB. Yep, sounds stupid. Intel using Realtek Ethernet chips. Sigh.

re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe
0ff mem 0x91104000-0x91104fff,0x91100000-0x91103fff at device 0.0 on pci3

re0: Using 1 MSI-X message

re0: turning off MSI enable bit.
re0: Chip rev. 0x54000000
re0: MAC rev. 0x00100000
miibus0: <MII bus> on re0
rgephy0: <RTL8251 1000BASE-T media interface> PHY 1 on miibus0
rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 94:c6:91::

re0@pci0:3:0:0: class=0x020000 card=0x20678086 chip=0x816810ec rev=0x15 hdr=0x00
vendor = 'Realtek Semiconductor Co., Ltd.'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

What I’ve observed is crazy packet loss and round trip times especially when running an iSCSI initiator with a lot
of filesystem activity.







Borja.

[YongHyeon PYUN]

On Sun, Apr 22, 2018 at 10:57:07PM -0700, Dieter BSD wrote:
> With several days more data, the Realtek driver is slightly different
> than the stock FreeBSD 10.3 driver, but it still fails a lot, with both
> TCP and UDP.
>

[...]

> My re run (1000baseT <full-duplex>) but the pauses happen even with
> light traffic. You don't have to run them full blast with rcp/ftp/whatever
> to get failures.
>

I guess benchmarks/netperf or benchmarks/iperf is better tool to
measure performance of ethernet driver. rcp/ftp involves other IOs
with network IO so you may measure performance of other subsystem.
If you see intermittent pauses during the test, try enabling or
disabling ethernet flow control of the controller.

# Disable flowcontrol
#ifconfig re0 media auto -mediaopt flow


# Enable flowcontrol
#ifconfig re0 media auto mediaopt flow

[...]

> rcp(1) to 8111F using Realtek driver
> Both machines basically idle.
> mtu=9000 on both ends but appears to be using 1500 anyway?

Cached route may cache the MTU of the interface. You may have to
down and re-up the interface to reflect the change.

> netstat -w 1 -d -I re0
>
> input re0 output
> packets errs idrops bytes packets errs bytes colls drops
> 5 0 0 330 6 0 755 0 0
> 4 0 0 264 4 0 564 0 0
> 5 0 0 331 5 0 630 0 0
> 17454 0 0 26176844 17453 0 1152264 0 0
> 67883 0 0 101023187 67880 0 4481248 0 0
> 52133 0 0 77141014 52140 0 3444580 0 0
> 72410 0 0 107980584 72409 0 4780384 0 0
> 80736 0 0 120645177 80735 0 5328810 0 0
> 71898 0 0 107298972 71899 0 4745634 0 0
> 14452 0 0 21510681 14475 0 955577 0 0
> 5 0 0 330 6 0 755 0 0
> 4 0 0 264 4 0 564 0 0
> 5 0 0 331 5 0 630 0 0
> 4 0 0 264 4 0 564 0 0
> 65323 0 0 97533279 65321 0 4311552 0 0
> 28383 0 0 42398766 28385 0 1873769 0 0
> 80759 0 0 120315366 80756 0 5330394 0 0
> 80992 0 0 120693553 80995 0 5345772 0 0
> 80932 0 0 120692488 80930 0 5341746 0 0
> 78905 0 0 117912275 78922 0 5209152 0 0
> 80158 0 0 119592452 80159 0 5290728 0 0

These counters are maintained in driver but it's not from MAC of
the controller. So you can't tell how much RX packets were dropped
before driver sees them. There is an undocumented sysctl variable
which reads H/W MAC counters and output them on console.

#sysctl dev.re.0.stats=1

[Joerg Sonnenberger]

On Wed, Apr 11, 2018 at 12:51:09PM +0200, Alex Dupre wrote:
> David Wolfskill wrote:
> >> I use a diskless workstation. With re driver provided by FreeBSD kernel
> >> (9/10/11.x), system randomly crashed because ethernet driver stalls (and
> >> datarate is always less than 300mbps). With official realtek driver
> >> (v194.01), system now runs as expected (with datarate up to 1 Gbps).
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=166724
>
> Anyone interested in taking the bounty or contributing to it? Perhaps
> the FreeBSD Foundation should jump in?

A starting point would be:
(1) what hardware offload features are enabled
(2) whether a specific direction is the problem
(3) correlate traffic on both ends, i.e. what packets are dropped "on
the wire"

The most likely cause of any problem here is a missing workaround for a
hardware bug or Realtek deciding that some specific tuning is necessary
for certain chips. It is nearly impossible to fix this kind of bugs
without having access to the hardware and a way to reproduce it.
Comparing with a working driver might be possible, but it is extremely
tideous.

Joerg

[Alex Dupre]

Joerg Sonnenberger wrote:
> The most likely cause of any problem here is a missing workaround for a
> hardware bug or Realtek deciding that some specific tuning is necessary
> for certain chips. It is nearly impossible to fix this kind of bugs
> without having access to the hardware and a way to reproduce it.
> Comparing with a working driver might be possible, but it is extremely
> tideous.

I've just setup two machines with re interfaces (one onboard and one
with an external pci-ex card), connected point to point, but I was
unable to reproduce the issue. I can easily reproduce it on another
server, but that's a semi-production one, so I can do limited testing
(ie replace the if_re.ko kernel module, reboot, and do something for a
few minutes, then I have to revert to the working kernel module).

--
Alex Dupre

[Warner Losh]

On Tue, Apr 24, 2018 at 7:48 AM, Alex Dupre <sysa...@alexdupre.com> wrote:

> Joerg Sonnenberger wrote:
> > The most likely cause of any problem here is a missing workaround for a
> > hardware bug or Realtek deciding that some specific tuning is necessary
> > for certain chips. It is nearly impossible to fix this kind of bugs
> > without having access to the hardware and a way to reproduce it.
> > Comparing with a working driver might be possible, but it is extremely
> > tideous.
>
> I've just setup two machines with re interfaces (one onboard and one
> with an external pci-ex card), connected point to point, but I was
> unable to reproduce the issue. I can easily reproduce it on another
> server, but that's a semi-production one, so I can do limited testing
> (ie replace the if_re.ko kernel module, reboot, and do something for a
> few minutes, then I have to revert to the working kernel module).
>

The rl/re cards have some code in them to cope with cable length to the
hub, since on the older cards this can greatly increase reliability.
Perhaps there's signal issue here? Also, the re drivers are known to have
some revs that have hardware offload features, but that said features don't
work. Ideally, if you had a completely reproducible scenario you could turn
off all the hardware offload and see if that fixes the problem.

Warner