Bug#960702: ethtool -m values change when output is redirected
Yannis Aribaud
Version: 1:4.19-1
Severity: important
I'm facing a very strange behavior. The command ethtool -m <interface> report the transceiver DOM values correctly, but when the command output is redirected to an other program, values change to somthing else.
Here is a transcript:
root@localhost:~# ethtool -m enp10s0; echo -e '\n\n\n'; ethtool -m enp10s0 | cat
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 80m
Length (62.5um) : 20m
Length (Copper) : 0m
Length (OM3) : 300m
Laser wavelength : 850nm
Vendor name : Pureoptics
Vendor OUI : 00:00:00
Vendor PN : EX-SFP-10GE-SR
Vendor rev : B4
Option values : 0x00 0x1a
Option : RX_LOS implemented
Option : TX_FAULT implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : M4351243
Date code : 190515
Optical diagnostics support : Yes
Laser bias current : 16.480 mA
Laser output power : 3.0768 mW / 4.88 dBm
Receiver signal average optical power : 1.2298 mW / 0.90 dBm
Module temperature : 48.47 degrees C / 119.24 degrees F
Module voltage : 1.2336 V
Alarm/warning flags implemented : Yes
Laser bias current high alarm : Off
Laser bias current low alarm : Off
Laser bias current high warning : Off
Laser bias current low warning : Off
Laser output power high alarm : Off
Laser output power low alarm : Off
Laser output power high warning : Off
Laser output power low warning : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : Off
Module voltage low alarm : Off
Module voltage high warning : Off
Module voltage low warning : Off
Laser rx power high alarm : Off
Laser rx power low alarm : Off
Laser rx power high warning : Off
Laser rx power low warning : Off
Laser bias current high alarm threshold : 4.744 mA
Laser bias current low alarm threshold : 49.896 mA
Laser bias current high warning threshold : 51.776 mA
Laser bias current low warning threshold : 50.910 mA
Laser output power high alarm threshold : 2.5701 mW / 4.10 dBm
Laser output power low alarm threshold : 0.8224 mW / -0.85 dBm
Laser output power high warning threshold : 0.8224 mW / -0.85 dBm
Laser output power low warning threshold : 0.8224 mW / -0.85 dBm
Module temperature high alarm threshold : 0.00 degrees C / 32.00 degrees F
Module temperature low alarm threshold : 0.00 degrees C / 32.00 degrees F
Module temperature high warning threshold : 0.00 degrees C / 32.00 degrees F
Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
Module voltage high alarm threshold : 0.4356 V
Module voltage low alarm threshold : 0.0000 V
Module voltage high warning threshold : 0.0000 V
Module voltage low warning threshold : 0.0000 V
Laser rx power high alarm threshold : 0.8224 mW / -0.85 dBm
Laser rx power low alarm threshold : 0.8224 mW / -0.85 dBm
Laser rx power high warning threshold : 0.8224 mW / -0.85 dBm
Laser rx power low warning threshold : 0.8224 mW / -0.85 dBm
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 80m
Length (62.5um) : 20m
Length (Copper) : 0m
Length (OM3) : 300m
Laser wavelength : 850nm
Vendor name : Pureoptics
Vendor OUI : 00:00:00
Vendor PN : EX-SFP-10GE-SR
Vendor rev : B4
Option values : 0x00 0x1a
Option : RX_LOS implemented
Option : TX_FAULT implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : M4351243
Date code : 190515
Optical diagnostics support : Yes
Laser bias current : 16.448 mA
Laser output power : 0.8224 mW / -0.85 dBm
Receiver signal average optical power : 0.8224 mW / -0.85 dBm
Module temperature : 32.12 degrees C / 89.83 degrees F
Module voltage : 0.8224 V
Alarm/warning flags implemented : Yes
Laser bias current high alarm : Off
Laser bias current low alarm : Off
Laser bias current high warning : Off
Laser bias current low warning : Off
Laser output power high alarm : Off
Laser output power low alarm : Off
Laser output power high warning : Off
Laser output power low warning : Off
Module temperature high alarm : Off
Module temperature low alarm : Off
Module temperature high warning : Off
Module temperature low warning : Off
Module voltage high alarm : On
Module voltage low alarm : Off
Module voltage high warning : On
Module voltage low warning : Off
Laser rx power high alarm : Off
Laser rx power low alarm : Off
Laser rx power high warning : Off
Laser rx power low warning : Off
Laser bias current high alarm threshold : 4.754 mA
Laser bias current low alarm threshold : 51.402 mA
Laser bias current high warning threshold : 56.552 mA
Laser bias current low warning threshold : 53.964 mA
Laser output power high alarm threshold : 2.6981 mW / 4.31 dBm
Laser output power low alarm threshold : 2.9216 mW / 4.66 dBm
Laser output power high warning threshold : 0.8224 mW / -0.85 dBm
Laser output power low warning threshold : 0.8224 mW / -0.85 dBm
Module temperature high alarm threshold : 0.00 degrees C / 32.00 degrees F
Module temperature low alarm threshold : 0.00 degrees C / 32.00 degrees F
Module temperature high warning threshold : 0.00 degrees C / 32.00 degrees F
Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
Module voltage high alarm threshold : 0.4368 V
Module voltage low alarm threshold : 0.0000 V
Module voltage high warning threshold : 0.0000 V
Module voltage low warning threshold : 0.0000 V
Laser rx power high alarm threshold : 0.8224 mW / -0.85 dBm
Laser rx power low alarm threshold : 0.8224 mW / -0.85 dBm
Laser rx power high warning threshold : 0.8224 mW / -0.85 dBm
Laser rx power low warning threshold : 0.8224 mW / -0.85 dBm
As you can see alsmost all mesuring values (C, V, mA and dBm) change. The values when output is redirected are unreliable which makes it impossible to use in a script.
I am using Debian GNU/Linux 10 (buster), kernel 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux and libc6 2.28-10
FYI, I could not reproduce the bug on Debian Stretch (ethtool 1:4.7-1+b1, kernel 4.9.0-11-amd64 and libc6 2.24-11+deb9u4).
--
Yannis Aribaud
Bjørn Mork
> Package: ethtool
> Version: 1:4.19-1
> Severity: important
AFAICS, your SFP+ is reporting strange values in either case. I do not
think any of these are correct. Looking at the non-redirected one:
> Laser output power : 3.0768 mW / 4.88 dBm
> Receiver signal average optical power : 1.2298 mW / 0.90 dBm
> Module temperature : 48.47 degrees C / 119.24 degrees F
> Module voltage : 1.2336 V
> Laser bias current high alarm threshold : 4.744 mA
> Laser bias current low alarm threshold : 49.896 mA
> Laser output power high alarm threshold : 2.5701 mW / 4.10 dBm
than your current output.
> Laser output power low alarm threshold : 0.8224 mW / -0.85 dBm
> Laser output power high warning threshold : 0.8224 mW / -0.85 dBm
> Laser output power low warning threshold : 0.8224 mW / -0.85 dBm
> Module temperature high alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low alarm threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature high warning threshold : 0.00 degrees C / 32.00 degrees F
> Module temperature low warning threshold : 0.00 degrees C / 32.00 degrees F
> Module voltage high alarm threshold : 0.4356 V
> Module voltage low alarm threshold : 0.0000 V
> Module voltage high warning threshold : 0.0000 V
> Module voltage low warning threshold : 0.0000 V
> Laser rx power high alarm threshold : 0.8224 mW / -0.85 dBm
> Laser rx power low alarm threshold : 0.8224 mW / -0.85 dBm
> Laser rx power high warning threshold : 0.8224 mW / -0.85 dBm
> Laser rx power low warning threshold : 0.8224 mW / -0.85 dBm
To me it looks like you are just reading arbitrary numbers from the
SFP+. Try replacing it and see if the results are more reliable.
Bjørn
Yannis Aribaud
> "Yannis Aribaud" <bu...@d6bell.net> writes:
>
> [...]
>
> To me it looks like you are just reading arbitrary numbers from the
> SFP+. Try replacing it and see if the results are more reliable.
I will try, but I am seeing the same behavior and similar values on 2 servers running the same hardware and the same software. The SFP+ modules model is commonly used on our equipements (servers, routers, switchs) and we never had such issues.
Values are clearly unreliable but not random at all. Values doesn't change between several runs of the command output redirected or not.
I mean values obviously change between redirected output and non redirected output runs, but not much between several runs redirected and neither between several runs non redirected.
Thus even if the values provided by the tranceivers are crazy, I see no reason seeing differences like those in values due to output redirect... I could not find any clue in the source code of ethtool, but I am no good developper at all.
Regards,
--
Yannis Aribaud
Ben Hutchings
On Fri, 2020-05-15 at 15:02 +0000, Yannis Aribaud wrote:
> Package: ethtool
> Version: 1:4.19-1
> Severity: important
> redirected to an other program, values change to somthing else.
> The values when output is redirected are unreliable which makes it
> impossible to use in a script.
driver.
> I am using Debian GNU/Linux 10 (buster), kernel 4.19.0-9-amd64 #1 SMP
> Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux and libc6 2.28-10
Ben.
--
Ben Hutchings
It's easier to fight for one's principles than to live up to them.
Yannis Aribaud
> Control: tag -1 moreinfo
>
> [...]
>
>> I am using Debian GNU/Linux 10 (buster), kernel 4.19.0-9-amd64 #1 SMP
>> Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux and libc6 2.28-10
>
> And which network driver are you using?
root@localhost:~# ethtool -i enp10s0
driver: mlx4_en
version: 4.0-0
firmware-version: 2.42.5000
expansion-rom-version:
bus-info: 0000:0a:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
root@localhost:~# modinfo mlx4_en
filename: /lib/modules/4.19.0-9-amd64/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
version: 4.0-0
license: Dual BSD/GPL
description: Mellanox ConnectX HCA Ethernet driver
author: Liran Liss, Yevgeny Petrilin
srcversion: 5E7CBA2E401F495296DC544
depends: mlx4_core,devlink
retpoline: Y
intree: Y
name: mlx4_en
vermagic: 4.19.0-9-amd64 SMP mod_unload modversions
sig_id: PKCS#7
signer: Debian Secure Boot CA
sig_key: A7:46:8D:EF
sig_hashalgo: sha256
signature: 27:8D:17:13:F5:5B:EA:8E:DC:88:30:FD:2C:89:50:64:E8:2A:C6:7A:
25:F5:67:A9:DB:82:CD:5E:46:0E:F6:2A:2F:3F:EF:6B:BE:EF:49:AA:
27:1D:4D:95:28:AA:6C:88:5A:B1:C7:75:5F:C2:78:71:7F:1B:FC:CC:
E0:F4:33:DE:9E:51:99:C2:FE:D1:2A:9B:EF:17:63:5C:1E:29:9A:4F:
00:56:6F:3E:ED:C5:4A:72:63:8A:4C:EE:33:FD:FE:7C:73:2B:D0:0D:
80:80:A0:91:36:0D:7E:B0:C2:AC:2D:5C:A3:55:BF:03:94:52:64:30:
DF:14:BC:FB:AF:3B:60:D3:4C:02:78:4D:51:61:D8:F2:E4:02:9C:D5:
A4:05:81:D9:47:92:5A:C2:0D:D5:00:71:37:85:9B:F4:E5:54:3F:22:
CD:4C:67:12:63:15:C2:1D:87:C8:03:3C:4E:23:6C:A5:F3:C6:12:99:
0E:D1:40:AA:47:45:64:0F:AB:C2:E0:71:64:03:AD:4B:F2:0C:02:F3:
D3:D5:E4:4C:26:63:14:93:46:5F:31:3B:4C:9B:4B:5D:CD:30:C0:DF:
28:8A:4D:9A:0E:A8:5A:87:F4:7F:DD:F6:0E:A3:3A:B2:99:C0:75:F9:
00:13:B0:44:06:D1:1C:FE:34:A1:77:36:2A:15:29:F7
parm: udp_rss:Enable RSS for incoming UDP traffic or disabled (0) (uint)
parm: pfctx:Priority based Flow Control policy on TX[7:0]. Per priority bit mask (uint)
parm: pfcrx:Priority based Flow Control policy on RX[7:0]. Per priority bit mask (uint)
parm: inline_thold:Threshold for using inline data (range: 17-104, default: 104) (uint)
Latest firmware available on Mellanox website for this NIC and default kernel driver.
Regards,
--
Yannis Aribaud
Ben Hutchings
Control: reassign -1 src:linux
Control: found -1 4.9.210-1
Control: found -1 4.19.118-2
Control: found -1 5.6.7-1
Control: tag -1 security
On Sun, 2020-05-17 at 14:41 +0000, Yannis Aribaud wrote:
> 16 mai 2020 01:29 "Ben Hutchings" <b...@decadent.org.uk> a écrit:
>
> > Control: tag -1 moreinfo
> >
> > [...]
> >
> > > I am using Debian GNU/Linux 10 (buster), kernel 4.19.0-9-amd64 #1 SMP
> > > Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux and libc6 2.28-10
> >
> > And which network driver are you using?
>
> root@localhost:~# ethtool -i enp10s0
> driver: mlx4_en
> version: 4.0-0
> firmware-version: 2.42.5000
> expansion-rom-version:
> bus-info: 0000:0a:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: yes
>
> root@localhost:~# modinfo mlx4_en
> filename: /lib/modules/4.19.0-9-amd64/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
Right, this turns to have exactly the bug I suspected. :-/
Tagged security because this is leaking kernel memory.
Ben.
--
Ben Hutchings
Logic doesn't apply to the real world. - Marvin Minsky
Ben Hutchings
in copying an uninitialised (or partly initialised) buffer back to
user-space.
Change it so that:
* In the special case that the DOM turns out not to be readable, the
remaining part of the buffer is cleared. This should avoid a
regression when reading modules with this problem.
* In other error cases, the error code is propagated.
Reported-by: Yannis Aribaud <bu...@d6bell.net>
References: https://bugs.debian.org/960702
Fixes: 7202da8b7f71 ("ethtool, net/mlx4_en: Cable info, get_module_info/...")
Signed-off-by: Ben Hutchings <b...@decadent.org.uk>
---
This is compile-tested only. It should go to stable, if it is a
correct fix.
Ben.
drivers/net/ethernet/mellanox/mlx4/en_ethtool.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c
index 8a5ea2543670..6edc3177af1c 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c
@@ -2078,14 +2078,17 @@ static int mlx4_en_get_module_eeprom(struct net_device *dev,
ret = mlx4_get_module_info(mdev->dev, priv->port,
offset, ee->len - i, data + i);
- if (!ret) /* Done reading */
+ if (!ret) {
+ /* DOM was not readable after all */
+ memset(data + i, 0, ee->len - i);
return 0;
+ }
if (ret < 0) {
en_err(priv,
"mlx4_get_module_info i(%d) offset(%d) bytes_to_read(%d) - FAILED (0x%x)\n",
i, offset, ee->len - i, ret);
- return 0;
+ return ret;
}
i += ret;
Yannis Aribaud
> [...]
Just in case that could be usefull. We change the transceiver for an other model also known to work correctly.
root@localhost:~# ethtool -m enp10s0
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x10 0x00 0x00 0x00 0x40 0x00 0x0c 0x00 0x00
Transceiver type : 10G Ethernet: 10G Base-SR
Transceiver type : FC: short distance (S)
Transceiver type : FC: Multimode, 62.5um (M6)
Transceiver type : FC: Multimode, 50um (M5)
Encoding : 0x06 (64B/66B)
BR, Nominal : 10300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 300m
Length (62.5um) : 150m
Length (Copper) : 0m
Length (OM3) : 0m
Laser wavelength : 850nm
Vendor name : FiberStore
Vendor OUI : 00:90:65
Vendor PN : SFP-10GSR-85
Vendor rev : A
Option values : 0x00 0x1a
Option : RX_LOS implemented
Option : TX_FAULT implemented
Option : TX_DISABLE implemented
BR margin, max : 0%
BR margin, min : 0%
Vendor SN : N3612030002
Date code : 161203
If I understood correctly, the driver is unable to read the DOM values correctly and return uninitialized values, which is a security issue (leaking kernel memory).
The fact that reading the DOM fails is an other issue (maybe Mellanox NIC firmware related ?).
But why are the values changing when output is redirected ? Does it change the variables memory addresses or pointers values ? What mechanism is involved there ?
Best Regards,
--
Yannis Aribaud
Ben Hutchings
> > mlx4_en_get_module_eeprom() returns 0 even if it fails. This results
> > in copying an uninitialised (or partly initialised) buffer back to
> > user-space.
> I am not sure i see the issue in here, and why we need the partial
> memset ?
> first thing in this function we do:
> memset(data, 0, ee->len);
>
> and then mlx4_get_module_info() will only copy valid data only on
> success.
Wow, sorry, I don't know how I missed that. So this is not the bug I
was looking for.
>
> > - if (!ret) /* Done reading */
> > + if (!ret) {
> > + /* DOM was not readable after all */
>
> means how much data was read, so if it returns 0, it means that this
> was the last iteration and we are done reading the eeprom..
>
> so i would remove the above comment and the memset below is redundant
> since we already memset the whole buffer before the while loop.
Right.
> > + memset(data + i, 0, ee->len - i);
> > return 0;
> > + }
> >
> > if (ret < 0) {
> > en_err(priv,
> > "mlx4_get_module_info i(%d) offset(%d)
> > bytes_to_read(%d) - FAILED (0x%x)\n",
> > i, offset, ee->len - i, ret);
> > - return 0;
> > + return ret;
>
> problem. you can verify by looking in the kernel log and verify you see
> the log message.
The original bug report (https://bugs.debian.org/960702) says that
ethtool reports different values depending on whether its output is
redirected. Although returning all-zeroes for the unreadable part
might be wrong, it doesn't explain that behaviour.
Perhaps if the timing of the I²C reads is marginal, varying numbers of
bytes of DOM information might be readable? But I don't see how
redirection of ethtool's output would affect that. It uses a single
ioctl to read everything, and the kernel controls timing within that.
So I am mystified about what is going on here. Maybe there is a bug in
ethtool, but I'm not seeing it.
Ben.
> > }
> >
> > i += ret;
--
Ben Hutchings
The two most common things in the universe are hydrogen and stupidity.