[mm/kasan] 80a9201a59 BUG: kernel reboot-without-warning in early-boot stage, last printk: Booting the kernel.

[Fengguang Wu]

Hi Alexander,

FYI, we find an old bug that's still alive in linux-next. The attached
reproduce-* script may help debug the problem.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

commit 80a9201a5965f4715d5c09790862e0df84ce0614
Author: Alexander Potapenko <gli...@google.com>
AuthorDate: Thu Jul 28 15:49:07 2016 -0700
Commit: Linus Torvalds <torv...@linux-foundation.org>
CommitDate: Thu Jul 28 16:07:41 2016 -0700

mm, kasan: switch SLUB to stackdepot, enable memory quarantine for SLUB

For KASAN builds:
- switch SLUB allocator to using stackdepot instead of storing the
allocation/deallocation stacks in the objects;
- change the freelist hook so that parts of the freelist can be put
into the quarantine.

[arya...@virtuozzo.com: fixes]
Link: http://lkml.kernel.org/r/1468601423-28676-1-gi...@virtuozzo.com
Link: http://lkml.kernel.org/r/1468347165-41906-3-g...@google.com
Signed-off-by: Alexander Potapenko <gli...@google.com>
Cc: Andrey Konovalov <adec...@gmail.com>
Cc: Christoph Lameter <c...@linux.com>
Cc: Dmitry Vyukov <dvy...@google.com>
Cc: Steven Rostedt (Red Hat) <ros...@goodmis.org>
Cc: Joonsoo Kim <iamjoon...@lge.com>
Cc: Kostya Serebryany <k...@google.com>
Cc: Andrey Ryabinin <arya...@virtuozzo.com>
Cc: Kuthonuzo Luruo <kuthonu...@hpe.com>
Signed-off-by: Andrew Morton <ak...@linux-foundation.org>
Signed-off-by: Linus Torvalds <torv...@linux-foundation.org>

c146a2b98e mm, kasan: account for object redzone in SLUB's nearest_obj()
80a9201a59 mm, kasan: switch SLUB to stackdepot, enable memory quarantine for SLUB
+--------------------------------------------------------------------------------------+------------+------------+
| | c146a2b98e | 80a9201a59 |
+--------------------------------------------------------------------------------------+------------+------------+
| boot_successes | 740 | 48 |
| boot_failures | 0 | 142 |
| BUG:kernel_reboot-without-warning_in_early-boot_stage,last_printk:Booting_the_kernel | 0 | 131 |
| BUG:kernel_in_stage | 0 | 11 |
+--------------------------------------------------------------------------------------+------------+------------+


Decompressing Linux... Parsing ELF... done.
Booting the kernel.


git bisect start v4.8 v4.7 --
git bisect bad e6e7214fbbdab1f90254af68e0927bdb24708d22 # 20:07 0- 1 Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad ba929b6646c5b87c7bb15cd8d3e51617725c983b # 21:11 0- 2 Merge branch 'for-linus-4.8' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good 5f22004ba9b4cf740773777ea7b74586743f6051 # 22:41 190+ 0 Merge branch 'x86-timers-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 124a3d88fa20e1869fc229d7d8c740cc81944264 # 23:01 182+ 0 Disable "frame-address" warning
git bisect bad 20d00ee829428ea6aab77fa3acca048a6f57d3bc # 23:35 0- 1 Revert "vfs: add lookup_hash() helper"
git bisect good 6039b80eb50a893476fea7d56e86ed2d19290054 # 00:20 183+ 0 Merge tag 'dmaengine-4.8-rc1' of git://git.infradead.org/users/vkoul/slave-dma
git bisect bad e55884d2c6ac3ae50e49a1f6fe38601a91181719 # 00:53 0- 3 Merge tag 'vfio-v4.8-rc1' of git://github.com/awilliam/linux-vfio
git bisect bad d94ba9e7d8d5c821d0442f13b30b0140c1109c38 # 01:46 0- 2 Merge tag 'pinctrl-v4.8-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl
git bisect bad 1c88e19b0f6a8471ee50d5062721ba30b8fd4ba9 # 01:58 0- 1 Merge branch 'akpm' (patches from Andrew)
git bisect good bca6759258dbef378bcf5b872177bcd2259ceb68 # 03:16 181+ 0 mm, vmstat: remove zone and node double accounting by approximating retries
git bisect good efdc94907977d2db84b4b00cb9bd98ca011f6819 # 08:58 190+ 0 mm: fix memcg stack accounting for sub-page stacks
git bisect good fb399b4854d2159a4d23fbfbd7daaed914fd54fa # 11:50 183+ 0 mm/memblock.c: fix index adjustment error in __next_mem_range_rev()
git bisect bad 31a6c1909f51dbe9bf08eb40dc64e3db90cf6f79 # 12:09 0- 2 mm, page_alloc: set alloc_flags only once in slowpath
git bisect good c146a2b98eb5898eb0fab15a332257a4102ecae9 # 12:51 180+ 0 mm, kasan: account for object redzone in SLUB's nearest_obj()
git bisect bad 87cc271d5e4320d705cfdf59f68d4d037b3511b2 # 13:19 0- 1 lib/stackdepot.c: use __GFP_NOWARN for stack allocations
git bisect bad 80a9201a5965f4715d5c09790862e0df84ce0614 # 13:34 0- 1 mm, kasan: switch SLUB to stackdepot, enable memory quarantine for SLUB
# first bad commit: [80a9201a5965f4715d5c09790862e0df84ce0614] mm, kasan: switch SLUB to stackdepot, enable memory quarantine for SLUB
git bisect good c146a2b98eb5898eb0fab15a332257a4102ecae9 # 15:16 550+ 0 mm, kasan: account for object redzone in SLUB's nearest_obj()
# extra tests on HEAD of linux-devel/devel-spot-201702260211
git bisect bad 494b6947f72e0d28eb229387a0dc27e95d79b605 # 15:16 0- 15 0day head guard for 'devel-spot-201702260211'
# extra tests on tree/branch linus/master
git bisect bad e5d56efc97f8240d0b5d66c03949382b6d7e5570 # 15:28 0- 1 Merge tag 'watchdog-for-linus-v4.11' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging
# extra tests on tree/branch linux-next/master
git bisect bad ed7b11e565c736828f0b793f596a4ca20efee747 # 15:40 0- 3 Add linux-next specific files for 20170227

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/lkp Intel Corporation

[Dmitry Vyukov]

On Tue, Feb 28, 2017 at 4:12 AM, Fengguang Wu <fenggu...@intel.com> wrote:
> Hi Alexander,
>
> FYI, we find an old bug that's still alive in linux-next. The attached
> reproduce-* script may help debug the problem.

Hi Fengguang,

KASAN works fine for us all that time in qemu and on real machines. Do
you have any idea as to what's relevant to the hang in all these qemu
flags and command line flags? One idea is that 512MB may not be enough
for KASAN. Does increasing amount of memory help?

> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.
> To post to this group, send email to kasa...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20170228031227.tm7flsxl7t7klspf%40wfg-t540p.sh.intel.com.
> For more options, visit https://groups.google.com/d/optout.

[Fengguang Wu]

Hi Alexander,

FYI, here is another bisect result.

434fd6353b Merge tag 'tty-4.11-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
5be4921c99 Add linux-next specific files for 20170310
+------------------------------+------------+------------+------------+---------------+
| | c146a2b98e | 80a9201a59 | 434fd6353b | next-20170310 |
+------------------------------+------------+------------+------------+---------------+
| boot_successes | 31 | 0 | 0 | 0 |
| boot_failures | 0 | 11 | 13 | 11 |
| BUG:KASAN:slab-out-of-bounds | 0 | 11 | 13 | 11 |
| calltrace:SyS_read | 0 | 11 | | |
| calltrace:SyS_linkat | 0 | 11 | | |
| calltrace:SyS_link | 0 | 11 | | |
| calltrace:SyS_unlink | 0 | 11 | | |
| calltrace:SyS_write | 0 | 11 | | |
| calltrace:SyS_getdents | 0 | 9 | | |
| calltrace:sock_init | 0 | 9 | | |
| calltrace:ide_cdrom_init | 0 | 9 | | |
| calltrace:md_init | 0 | 9 | | |
| calltrace:init_scsi | 0 | 9 | | |
| calltrace:init_xfs_fs | 0 | 7 | | |
| calltrace:init_devpts_fs | 0 | 7 | | |
| calltrace:sysctl_core_init | 0 | 3 | | |
| calltrace:af_unix_init | 0 | 3 | | |
+------------------------------+------------+------------+------------+---------------+

[ 22.974867] debug: unmapping init [mem 0xffff8800023f5000-0xffff8800023fffff]
[ 40.729584] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[ 40.743879] random: init: uninitialized urandom read (12 bytes read)
[ 40.754136] hostname (177) used greatest stack depth: 29632 bytes left
[ 40.791170] ==================================================================
[ 40.792751] BUG: KASAN: slab-out-of-bounds in inotify_read+0x1ac/0x2c6 at addr ffff88001539780c
[ 40.794614] Read of size 5 by task init/1
[ 40.795491] CPU: 0 PID: 1 Comm: init Not tainted 4.7.0-05999-g80a9201 #1
[ 40.796933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014
[ 40.798606] ffffed0002a72f02 ffff88000004fcb8 ffffffff813fbc56 ffff88000004fd48
[ 40.799906] ffffffff81125e14 ffff880000000000 ffff880000041300 0000000000000246
[ 40.801214] 0000000000000282 ffff880011331b00 0000000000000010 0000000000000246
[ 40.802505] Call Trace:
[ 40.802934] [<ffffffff813fbc56>] dump_stack+0x19/0x1b
[ 40.803791] [<ffffffff81125e14>] kasan_report+0x316/0x552
[ 40.804670] [<ffffffff81124ca6>] check_memory_region+0x10b/0x10d
[ 40.805674] [<ffffffff81124d7b>] kasan_check_read+0x11/0x13
[ 40.806623] [<ffffffff81171647>] inotify_read+0x1ac/0x2c6
[ 40.807535] [<ffffffff8108cda1>] ? wait_woken+0x76/0x76
[ 40.808425] [<ffffffff811382b0>] __vfs_read+0x23/0xe3
[ 40.809270] [<ffffffff813a372f>] ? security_file_permission+0x93/0x9c
[ 40.810351] [<ffffffff81138406>] vfs_read+0x96/0x102
[ 40.811181] [<ffffffff811387cb>] SyS_read+0x4e/0x94
[ 40.812010] [<ffffffff81d379bd>] entry_SYSCALL_64_fastpath+0x23/0xc1
[ 40.813058] Object at ffff8800153977e0, in cache kmalloc-64
[ 40.813979] Object allocated with size 54 bytes.
[ 40.814697] Allocation:
[ 40.815123] PID = 189
[ 40.815514] [<ffffffff81010c9f>] save_stack_trace+0x27/0x45
[ 40.816473] [<ffffffff8112530e>] kasan_kmalloc+0xe5/0x16c
[ 40.817399] [<ffffffff81123d1d>] __kmalloc+0x16c/0x17e
[ 40.818289] [<ffffffff8117106e>] inotify_handle_event+0x80/0x10e
[ 40.819323] [<ffffffff8116f8b0>] fsnotify+0x3c5/0x4f4
[ 40.820200] [<ffffffff81145c5b>] vfs_link+0x1d8/0x210
[ 40.821070] [<ffffffff81145dfb>] SyS_linkat+0x168/0x22c
[ 40.821981] [<ffffffff81145ed8>] SyS_link+0x19/0x1b
[ 40.822805] [<ffffffff81d379bd>] entry_SYSCALL_64_fastpath+0x23/0xc1
[ 40.823902] Memory state around the buggy address:
[ 40.824664] ffff880015397700: fc fc fc fc 00 00 00 00 00 00 00 fc fc fc fc fc

# HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD

git bisect start v4.8 v4.7 --

git bisect bad e6e7214fbbdab1f90254af68e0927bdb24708d22 # 17:23 B 0 7 17 0 Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad ba929b6646c5b87c7bb15cd8d3e51617725c983b # 17:31 B 0 2 12 0 Merge branch 'for-linus-4.8' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect good 468fc7ed5537615efe671d94248446ac24679773 # 17:44 G 11 0 0 0 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
git bisect bad e55884d2c6ac3ae50e49a1f6fe38601a91181719 # 17:59 B 0 5 15 0 Merge tag 'vfio-v4.8-rc1' of git://github.com/awilliam/linux-vfio
git bisect good 554828ee0db41618d101d9549db8808af9fd9d65 # 18:16 G 10 0 0 0 Merge branch 'salted-string-hash'
git bisect good ce8c891c3496d3ea4a72ec40beac9a7b7f6649bf # 18:30 G 11 0 0 0 Merge tag 'rproc-v4.8' of git://github.com/andersson/remoteproc
git bisect bad 1c88e19b0f6a8471ee50d5062721ba30b8fd4ba9 # 18:39 B 0 11 21 0 Merge branch 'akpm' (patches from Andrew)
git bisect good c9b011a87dd49bac1632311811c974bb7cd33c25 # 18:51 G 11 0 0 0 Merge tag 'hwlock-v4.8' of git://github.com/andersson/remoteproc
git bisect good 6039b80eb50a893476fea7d56e86ed2d19290054 # 19:06 G 11 0 0 0 Merge tag 'dmaengine-4.8-rc1' of git://git.infradead.org/users/vkoul/slave-dma
git bisect good bca6759258dbef378bcf5b872177bcd2259ceb68 # 19:17 G 11 0 0 0 mm, vmstat: remove zone and node double accounting by approximating retries
git bisect good efdc94907977d2db84b4b00cb9bd98ca011f6819 # 19:32 G 11 0 0 0 mm: fix memcg stack accounting for sub-page stacks
git bisect good fb399b4854d2159a4d23fbfbd7daaed914fd54fa # 19:42 G 10 0 0 0 mm/memblock.c: fix index adjustment error in __next_mem_range_rev()
git bisect bad 31a6c1909f51dbe9bf08eb40dc64e3db90cf6f79 # 19:50 B 0 2 12 0 mm, page_alloc: set alloc_flags only once in slowpath
git bisect good c146a2b98eb5898eb0fab15a332257a4102ecae9 # 20:04 G 10 0 0 0 mm, kasan: account for object redzone in SLUB's nearest_obj()
git bisect bad 87cc271d5e4320d705cfdf59f68d4d037b3511b2 # 20:11 B 0 4 14 0 lib/stackdepot.c: use __GFP_NOWARN for stack allocations
git bisect bad 80a9201a5965f4715d5c09790862e0df84ce0614 # 20:25 B 0 4 14 0 mm, kasan: switch SLUB to stackdepot, enable memory quarantine for SLUB

# first bad commit: [80a9201a5965f4715d5c09790862e0df84ce0614] mm, kasan: switch SLUB to stackdepot, enable memory quarantine for SLUB

git bisect good c146a2b98eb5898eb0fab15a332257a4102ecae9 # 20:34 G 31 0 0 0 mm, kasan: account for object redzone in SLUB's nearest_obj()
# extra tests with CONFIG_DEBUG_INFO_REDUCED
git bisect bad 80a9201a5965f4715d5c09790862e0df84ce0614 # 20:47 B 0 10 20 0 mm, kasan: switch SLUB to stackdepot, enable memory quarantine for SLUB
# extra tests on HEAD of linux-devel/devel-spot-201703111328
git bisect bad f5cfbd2efb09391768ad494ec6cab7395c6835fe # 20:48 B 0 15 30 2 0day head guard for 'devel-spot-201703111328'

# extra tests on tree/branch linus/master

git bisect bad 434fd6353b4c83938029ca6ea7dfa4fc82d602bd # 20:59 B 0 2 12 0 Merge tag 'tty-4.11-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty

# extra tests on tree/branch linux-next/master

git bisect bad 5be4921c9958ec02a67506bd6f7a52fce663c201 # 21:15 B 0 11 21 0 Add linux-next specific files for 20170310

[Andrey Ryabinin]

On 02/28/2017 06:12 AM, Fengguang Wu wrote:
> Hi Alexander,
>
> FYI, we find an old bug that's still alive in linux-next. The attached
> reproduce-* script may help debug the problem.
>

...

> +--------------------------------------------------------------------------------------+------------+------------+
> | | c146a2b98e | 80a9201a59 |
> +--------------------------------------------------------------------------------------+------------+------------+
> | boot_successes | 740 | 48 |
> | boot_failures | 0 | 142 |
> | BUG:kernel_reboot-without-warning_in_early-boot_stage,last_printk:Booting_the_kernel | 0 | 131 |
> | BUG:kernel_in_stage | 0 | 11 |
> +--------------------------------------------------------------------------------------+------------+------------+
>

Indeed it is an old bug, I'll send a fix shortly. But the bisection result is not correct. This bug is actually much older.
Note that commit 80a9201a596 changes Kconfig dependency - it removes depends on SLUB_DEBUG from config KASAN section.
And yours config has:
# CONFIG_SLUB_DEBUG is not set

So you simply test c146a2b98e with CONFIG_KASAN=n and 80a9201a59 with CONFIG_KASAN=y

[Andrey Ryabinin]

On 03/11/2017 04:54 PM, Fengguang Wu wrote:
> Hi Alexander,
>
> FYI, here is another bisect result.
>

Also wrong for the same reason as before.

> [ 22.974867] debug: unmapping init [mem 0xffff8800023f5000-0xffff8800023fffff]
> [ 40.729584] x86/mm: Checked W+X mappings: passed, no W+X pages found.
> [ 40.743879] random: init: uninitialized urandom read (12 bytes read)
> [ 40.754136] hostname (177) used greatest stack depth: 29632 bytes left
> [ 40.791170] ==================================================================
> [ 40.792751] BUG: KASAN: slab-out-of-bounds in inotify_read+0x1ac/0x2c6 at addr ffff88001539780c
> [ 40.794614] Read of size 5 by task init/1

This is false-positive. According to dmesg this kernel was built with "gcc version 4.6.4 (Debian 4.6.4-7)".
As we recently discovered here - http://lkml.kernel.org/r/<1eb0b1ba-3847-9bdc...@gmail.com>
some old gcc versions such as 4.7.4 and now apparently 4.6.4 as well cause false-positives reports.
I'm guessing that old gcc miss-compile something in check_memory_region().

Given that kasan is fully supported only since gcc 5, could you teach the bot use only supported gcc
for the runtime testing with kasan?

[Andrey Ryabinin]

The kernel doesn't boot with both PROFILE_ANNOTATED_BRANCHES=y and KASAN=y
options selected. With branch profiling enabled we end up calling
ftrace_likely_update() before kasan_early_init(). ftrace_likely_update()
built with KASAN instrumentation, so calling it before kasan has been
initialized leads to crash.

Use DISABLE_BRANCH_PROFILING define to make sure that we don't call
ftrace_likely_update() from early code before kasan_early_init().

Fixes: ef7f0d6a6ca8 ("x86_64: add KASan support")
Reported-by: Fengguang Wu <fenggu...@intel.com>
Signed-off-by: Andrey Ryabinin <arya...@virtuozzo.com>
Cc: <sta...@vger.kernel.org>
---
arch/x86/kernel/head64.c | 1 +
arch/x86/mm/kasan_init_64.c | 1 +
2 files changed, 2 insertions(+)

diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 54a2372..b5785c1 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -4,6 +4,7 @@
* Copyright (C) 2000 Andrea Arcangeli <and...@suse.de> SuSE
*/

+#define DISABLE_BRANCH_PROFILING
#include <linux/init.h>
#include <linux/linkage.h>
#include <linux/types.h>
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 8d63d7a..4c90cfd 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -1,3 +1,4 @@
+#define DISABLE_BRANCH_PROFILING
#define pr_fmt(fmt) "kasan: " fmt
#include <linux/bootmem.h>
#include <linux/kasan.h>
--
2.10.2

[Greg Kroah-Hartman]

4.4-stable review patch. If anyone has any objections, please let me know.

------------------

From: Andrey Ryabinin <arya...@virtuozzo.com>

commit be3606ff739d1c1be36389f8737c577ad87e1f57 upstream.

The kernel doesn't boot with both PROFILE_ANNOTATED_BRANCHES=y and KASAN=y
options selected. With branch profiling enabled we end up calling

ftrace_likely_update() before kasan_early_init(). ftrace_likely_update() is

built with KASAN instrumentation, so calling it before kasan has been
initialized leads to crash.

Use DISABLE_BRANCH_PROFILING define to make sure that we don't call
ftrace_likely_update() from early code before kasan_early_init().

Fixes: ef7f0d6a6ca8 ("x86_64: add KASan support")
Reported-by: Fengguang Wu <fenggu...@intel.com>
Signed-off-by: Andrey Ryabinin <arya...@virtuozzo.com>

Cc: kasa...@googlegroups.com
Cc: Alexander Potapenko <gli...@google.com>
Cc: Andrew Morton <ak...@linux-foundation.org>
Cc: l...@01.org
Cc: Dmitry Vyukov <dvy...@google.com>
Link: http://lkml.kernel.org/r/20170313163337....@virtuozzo.com
Signed-off-by: Thomas Gleixner <tg...@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gre...@linuxfoundation.org>

---
arch/x86/kernel/head64.c | 1 +
arch/x86/mm/kasan_init_64.c | 1 +
2 files changed, 2 insertions(+)

--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -4,6 +4,7 @@
* Copyright (C) 2000 Andrea Arcangeli <and...@suse.de> SuSE
*/

+#define DISABLE_BRANCH_PROFILING
#include <linux/init.h>
#include <linux/linkage.h>
#include <linux/types.h>

[Greg Kroah-Hartman]

4.9-stable review patch. If anyone has any objections, please let me know.

[Greg Kroah-Hartman]

4.10-stable review patch. If anyone has any objections, please let me know.