Safe vs. unsafe handling of uncorrectable ECC errors

[Mark Seaborn]

Dan Kaminsky was asking what happens when an uncorrectable ECC error occurs [1].

On x86 systems, the most common mechanism is for an uncorrectable ECC error to generate an MCE (Machine Check Exception), which is handled by the kernel.  (There are some other possible mechanisms, which I'll mention below.)

Linux provides an "MCE tolerance level" option for specifying how MCEs should be handled.  It can be set via the "mce=N" boot option or via sysfs.  It is documented here:

https://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt

https://www.kernel.org/doc/Documentation/x86/x86_64/machinecheck

The available MCE tolerance levels are:

"0: always panic on uncorrected errors, log corrected errors

1: panic or SIGBUS on uncorrected errors, log corrected errors

2: SIGBUS or log uncorrected errors, log corrected errors

3: never panic or SIGBUS, log all errors (for testing only)

Default is 1"

If you have a machine that is susceptible to rowhammer and uses ECC, you should make sure that you're using at least "mce=0" or "mce=1".  Make sure you're not using "mce=2" or "mce=3", otherwise the ECC won't be protecting you from rowhammer-induced bit flips (which could lead to privilege escalation).

You can check your settings with "cat /sys/devices/system/machinecheck/machinecheck*/tolerant".  For the default setting, that will print all "1"s.

Based on the documentation above, it's possible there could be some risks from using the default of "mce=1" instead of the stricter "mce=0".  For "mce=1", the docs imply that the kernel *could* just deliver SIGBUS to a process that gets an uncorrectable ECC error when accessing memory.  There are two considerations here:

• In what cases does Linux deliver a SIGBUS instead of panicking and rebooting?  I'm told that this depends on hardware support for "uncorrectable [but] recoverable" (UCR) MCEs, which is only present in higher-end CPUs, such as those with the "-EX" suffix (e.g. Xeon-EX).
  
  That is covered by section 15.6, "Recovery of uncorrected recoverable (UCR) errors", in Intel's docs (http://download.intel.com/products/processor/manual/325462.pdf).  It should be possible to check programmatically whether this feature is available by reading MSRs.  The docs say that bit 24 of IA32_MCG_CAP (MCG_SER_P) reports whether this feature is available.
  
  
• How might an attacker abuse this?
  
  The biggest risk is that getting a SIGBUS instead of a panic+reboot allows an attacker to quickly continue running after triggering an uncorrectable 2-bit error.  The attacker could carry on hammering until they find a memory location that produces an undetectable 3-bit error, if the DRAM is bad enough.  The attacker could then exploit the 3-bit error.  Yoongu Kim et al's paper shows that they've seen DRAM with some 3-bit errors (see Table 5 in [2]).
  
  A possible, smaller risk is that there are programs that have unsafe behaviour when they get a SIGBUS from a memory access where getting a SIGBUS would normally be "impossible" (because the page being accessed is always mapped correctly).  If so, then if an attacker can find a memory location that generates a SIGBUS when accessed, they might be able to deallocate this page and cause this page to be reused by the unsafe program.

While MCEs appear to be the most common mechanism for reporting uncorrectable ECC errors, I'm told that some systems use other mechanisms:

• Some systems use EDAC for reporting errors.  See https://www.kernel.org/doc/Documentation/edac.txt.  That mentions Nehalem CPUs specifically, though those are quite old now.  There's a kernel option for setting whether uncorrectable errors produce a kernel panic:  see /sys/module/edac_core/parameters/edac_mc_panic_on_ue.
  
  
• I'm told that there are some systems which enforce a hardware reboot based on registers configured by the BIOS, and which don't have a mechanism for reporting errors to the kernel (either synchronously or asynchronously).

Cheers,

Mark

[1] https://twitter.com/dakami/status/577992096944025600 - "A definitive answer to what modern operating systems do given an uncorrectable ECC error is surprisingly elusive. #RowHammer"

[2] http://users.ece.cmu.edu/~yoonguk/papers/kim-isca14.pdf