[console_unlock] BUG: KASAN: use-after-scope in console_unlock+0x9cd/0xd10

8 views
Skip to first unread message

Fengguang Wu

unread,
Apr 18, 2018, 10:18:09 PM4/18/18
to Andrey Ryabinin, Pavel Tatashin, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Linus Torvalds, Nicolas Pitre, Nikitas Angelinas, Matt Redfearn, linux-...@vger.kernel.org, l...@01.org, Dmitry Vyukov, Steven Sistare, Daniel Jordan, Bob Picco, Linux Memory Management List, kasa...@googlegroups.com
Hello,

FYI this happens in mainline kernel 4.17.0-rc1.
It at least dates back to v4.15-rc1 .

The regression was reported before

https://lkml.org/lkml/2017/11/30/33

Where the last message from Dmitry mentions that use-after-scope has
known false positives with CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL=y
If so, what would be the best way to workaround such false positives
in boot testing? Disable the above config?

0day bisects produce diverged results, with 2 of them converge to
commit d17a1d97dc ("x86/mm/kasan: don't use vmemmap_populate() to
initialize shadow") and 1 bisected to the earlier a4a3ede213 ("mm:
zero reserved and unavailable struct pages"). I'll send the bisect
reports in follow up emails.

This occurs in 6 out of 6 boots.

[ 0.001000] RCU CPU stall warnings timeout set to 100 (rcu_cpu_stall_timeout).
[ 0.001000] Tasks RCU enabled.
[ 0.001000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.001000] NR_IRQS: 4352, nr_irqs: 48, preallocated irqs: 16
[ 0.001000] ==================================================================
[ 0.001000] BUG: KASAN: use-after-scope in console_unlock+0x9cd/0xd10:
console_unlock at kernel/printk/printk.c:2396
[ 0.001000] Write of size 1 at addr ffffffff84c07998 by task swapper/0
[ 0.001000]
[ 0.001000] CPU: 0 PID: 0 Comm: swapper Tainted: G T 4.17.0-rc1 #1
[ 0.001000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
[ 0.001000] Call Trace:
[ 0.001000] ? dump_stack+0x11a/0x20b:
dump_stack at lib/dump_stack.c:115
[ 0.001000] ? show_regs_print_info+0x5/0x5:
dump_stack at lib/dump_stack.c:89
[ 0.001000] ? do_raw_spin_trylock+0xee/0x190:
arch_atomic_cmpxchg at arch/x86/include/asm/atomic.h:191
(inlined by) atomic_cmpxchg at include/asm-generic/atomic-instrumented.h:58
(inlined by) queued_spin_trylock at include/asm-generic/qspinlock.h:72
(inlined by) do_raw_spin_trylock at kernel/locking/spinlock_debug.c:119
[ 0.001000] ? print_address_description+0x24/0x330:
print_address_description at mm/kasan/report.c:257
[ 0.001000] ? kasan_report+0x22a/0x380:
kasan_report_error at mm/kasan/report.c:355
(inlined by) kasan_report at mm/kasan/report.c:412
[ 0.001000] ? console_unlock+0x9cd/0xd10:
console_unlock at kernel/printk/printk.c:2396
[ 0.001000] ? lock_acquire+0x4d0/0x560:
get_current at arch/x86/include/asm/current.h:15
(inlined by) lock_acquire at kernel/locking/lockdep.c:3922
[ 0.001000] ? vprintk_emit+0x555/0x880:
console_trylock_spinning at kernel/printk/printk.c:1643
(inlined by) vprintk_emit at kernel/printk/printk.c:1906
[ 0.001000] ? wake_up_klogd+0x110/0x110:
console_unlock at kernel/printk/printk.c:2289
[ 0.001000] ? lock_release+0xf80/0xf80:
lock_acquire at kernel/locking/lockdep.c:3909
[ 0.001000] ? do_raw_spin_trylock+0x190/0x190:
do_raw_spin_unlock at kernel/locking/spinlock_debug.c:133
[ 0.001000] ? trace_hardirqs_on+0x3f0/0x400:
trace_hardirqs_on at kernel/trace/trace_irqsoff.c:795
[ 0.001000] ? vprintk_emit+0x555/0x880:
console_trylock_spinning at kernel/printk/printk.c:1643
(inlined by) vprintk_emit at kernel/printk/printk.c:1906
[ 0.001000] ? vprintk_emit+0x813/0x880:
__preempt_count_sub at arch/x86/include/asm/preempt.h:81
(inlined by) vprintk_emit at kernel/printk/printk.c:1908
[ 0.001000] ? console_unlock+0xd10/0xd10:
vprintk_emit at kernel/printk/printk.c:1830
[ 0.001000] ? memblock_add+0x163/0x163:
memblock_reserve at mm/memblock.c:716
[ 0.001000] ? lock_release+0xf23/0xf80:
lock_release at kernel/locking/lockdep.c:3929
[ 0.001000] ? memblock_virt_alloc_internal+0x191/0x2ef:
memblock_virt_alloc_internal at mm/memblock.c:1277 (discriminator 1)
[ 0.001000] ? memset+0x1f/0x40:
memset at mm/kasan/kasan.c:287
[ 0.001000] ? zero_pud_populate+0x5b1/0x936:
set_pmd at arch/x86/include/asm/paravirt.h:468
(inlined by) pmd_populate_kernel at arch/x86/include/asm/pgalloc.h:80
(inlined by) zero_pmd_populate at mm/kasan/kasan_init.c:76
(inlined by) zero_pud_populate at mm/kasan/kasan_init.c:109
[ 0.001000] ? printk+0x9c/0xc3:
printk at kernel/printk/printk.c:1975
[ 0.001000] ? kmsg_dump_rewind+0x134/0x134:
printk at kernel/printk/printk.c:1975
[ 0.001000] ? kasan_init+0x413/0x4af:
__flush_tlb_global at arch/x86/include/asm/paravirt.h:299
(inlined by) __flush_tlb_all at arch/x86/include/asm/tlbflush.h:433
(inlined by) kasan_init at arch/x86/mm/kasan_init_64.c:390
[ 0.001000] ? setup_arch+0x1fdf/0x225a:
setup_arch at arch/x86/kernel/setup.c:1216
[ 0.001000] ? reserve_standard_io_resources+0x88/0x88:
setup_arch at arch/x86/kernel/setup.c:816
[ 0.001000] ? debug_check_no_locks_freed+0x241/0x280:
debug_check_no_locks_freed at kernel/locking/lockdep.c:4422 (discriminator 1)
[ 0.001000] ? printk+0x9c/0xc3:
printk at kernel/printk/printk.c:1975
[ 0.001000] ? kmsg_dump_rewind+0x134/0x134:
printk at kernel/printk/printk.c:1975
[ 0.001000] ? do_device_not_available+0x60/0x60:
idt_setup_from_table at arch/x86/kernel/idt.c:220
[ 0.001000] ? start_kernel+0xf3/0xfd4:
add_latent_entropy at include/linux/random.h:26
(inlined by) start_kernel at init/main.c:556
[ 0.001000] ? early_idt_handler_common+0x3b/0x52:
early_idt_handler_common at arch/x86/kernel/head_64.S:327
[ 0.001000] ? mem_encrypt_init+0x33/0x33
[ 0.001000] ? memcpy_orig+0x54/0x110:
memcpy_orig at arch/x86/lib/memcpy_64.S:106
[ 0.001000] ? secondary_startup_64+0xa5/0xb0:
secondary_startup_64 at arch/x86/kernel/head_64.S:242
[ 0.001000]
[ 0.001000] Memory state around the buggy address:
[ 0.001000] ffffffff84c07880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 0.001000] ffffffff84c07900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1
[ 0.001000] >ffffffff84c07980: f1 f1 f1 f8 f2 f2 f2 f2 f2 f2 f2 01 f2 f2 f2 f2

Attached the full dmesg, kconfig and reproduce scripts.

Thanks,
Fengguang
dmesg-yocto-lkp-nhm-dp1-1:20180417164158:x86_64-randconfig-b0-04141112:4.17.0-rc1:1
.config
reproduce-yocto-lkp-nhm-dp1-1:20180417164158:x86_64-randconfig-b0-04141112:4.17.0-rc1:1

Fengguang Wu

unread,
Apr 18, 2018, 10:22:13 PM4/18/18
to Pavel Tatashin, Andrey Ryabinin, Steven Sistare, Daniel Jordan, Bob Picco, Linux Memory Management List, linux-...@vger.kernel.org, LKP, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Linus Torvalds, Nicolas Pitre, Nikitas Angelinas, Matt Redfearn, Dmitry Vyukov, kasa...@googlegroups.com
On Thu, Apr 19, 2018 at 10:17:57AM +0800, Fengguang Wu wrote:
>Hello,
>
>FYI this happens in mainline kernel 4.17.0-rc1.
>It at least dates back to v4.15-rc1 .
>
>The regression was reported before
>
> https://lkml.org/lkml/2017/11/30/33
>
>Where the last message from Dmitry mentions that use-after-scope has
>known false positives with CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL=y
>If so, what would be the best way to workaround such false positives
>in boot testing? Disable the above config?
>
>0day bisects produce diverged results, with 2 of them converge to
>commit d17a1d97dc ("x86/mm/kasan: don't use vmemmap_populate() to
>initialize shadow") and 1 bisected to the earlier a4a3ede213 ("mm:
>zero reserved and unavailable struct pages"). I'll send the bisect
>reports in follow up emails.

Here is the bisect report for

commit a4a3ede2132ae0863e2d43e06f9b5697c51a7a3b
Author: Pavel Tatashin <pasha.t...@oracle.com>
AuthorDate: Wed Nov 15 17:36:31 2017 -0800
Commit: Linus Torvalds <torv...@linux-foundation.org>
CommitDate: Wed Nov 15 18:21:05 2017 -0800

mm: zero reserved and unavailable struct pages

Some memory is reserved but unavailable: not present in memblock.memory
(because not backed by physical pages), but present in memblock.reserved.
Such memory has backing struct pages, but they are not initialized by
going through __init_single_page().

In some cases these struct pages are accessed even if they do not
contain any data. One example is page_to_pfn() might access page->flags
if this is where section information is stored (CONFIG_SPARSEMEM,
SECTION_IN_PAGE_FLAGS).

One example of such memory: trim_low_memory_range() unconditionally
reserves from pfn 0, but e820__memblock_setup() might provide the
exiting memory from pfn 1 (i.e. KVM).

Since struct pages are zeroed in __init_single_page(), and not during
allocation time, we must zero such struct pages explicitly.

The patch involves adding a new memblock iterator:
for_each_resv_unavail_range(i, p_start, p_end)

Which iterates through reserved && !memory lists, and we zero struct pages
explicitly by calling mm_zero_struct_page().

===

Here is more detailed example of problem that this patch is addressing:

Run tested on qemu with the following arguments:

-enable-kvm -cpu kvm64 -m 512 -smp 2

This patch reports that there are 98 unavailable pages.

They are: pfn 0 and pfns in range [159, 255].

Note, trim_low_memory_range() reserves only pfns in range [0, 15], it does
not reserve [159, 255] ones.

e820__memblock_setup() reports linux that the following physical ranges are
available:
[1 , 158]
[256, 130783]

Notice, that exactly unavailable pfns are missing!

Now, lets check what we have in zone 0: [1, 131039]

pfn 0, is not part of the zone, but pfns [1, 158], are.

However, the bigger problem we have if we do not initialize these struct
pages is with memory hotplug. Because, that path operates at 2M
boundaries (section_nr). And checks if 2M range of pages is hot
removable. It starts with first pfn from zone, rounds it down to 2M
boundary (sturct pages are allocated at 2M boundaries when vmemmap is
created), and checks if that section is hot removable. In this case
start with pfn 1 and convert it down to pfn 0. Later pfn is converted
to struct page, and some fields are checked. Now, if we do not zero
struct pages, we get unpredictable results.

In fact when CONFIG_VM_DEBUG is enabled, and we explicitly set all
vmemmap memory to ones, the following panic is observed with kernel test
without this patch applied:

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: is_pageblock_removable_nolock+0x35/0x90
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT
...
task: ffff88001f4e2900 task.stack: ffffc90000314000
RIP: 0010:is_pageblock_removable_nolock+0x35/0x90
Call Trace:
? is_mem_section_removable+0x5a/0xd0
show_mem_removable+0x6b/0xa0
dev_attr_show+0x1b/0x50
sysfs_kf_seq_show+0xa1/0x100
kernfs_seq_show+0x22/0x30
seq_read+0x1ac/0x3a0
kernfs_fop_read+0x36/0x190
? security_file_permission+0x90/0xb0
__vfs_read+0x16/0x30
vfs_read+0x81/0x130
SyS_read+0x44/0xa0
entry_SYSCALL_64_fastpath+0x1f/0xbd

Link: http://lkml.kernel.org/r/20171013173214.273...@oracle.com
Signed-off-by: Pavel Tatashin <pasha.t...@oracle.com>
Reviewed-by: Steven Sistare <steven....@oracle.com>
Reviewed-by: Daniel Jordan <daniel....@oracle.com>
Reviewed-by: Bob Picco <bob....@oracle.com>
Tested-by: Bob Picco <bob....@oracle.com>
Acked-by: Michal Hocko <mho...@suse.com>
Cc: Alexander Potapenko <gli...@google.com>
Cc: Andrey Ryabinin <arya...@virtuozzo.com>
Cc: Ard Biesheuvel <ard.bie...@linaro.org>
Cc: Catalin Marinas <catalin...@arm.com>
Cc: Christian Borntraeger <bornt...@de.ibm.com>
Cc: David S. Miller <da...@davemloft.net>
Cc: Dmitry Vyukov <dvy...@google.com>
Cc: Heiko Carstens <heiko.c...@de.ibm.com>
Cc: "H. Peter Anvin" <h...@zytor.com>
Cc: Ingo Molnar <mi...@redhat.com>
Cc: Mark Rutland <mark.r...@arm.com>
Cc: Matthew Wilcox <wi...@infradead.org>
Cc: Mel Gorman <mgo...@techsingularity.net>
Cc: Michal Hocko <mho...@kernel.org>
Cc: Sam Ravnborg <s...@ravnborg.org>
Cc: Thomas Gleixner <tg...@linutronix.de>
Cc: Will Deacon <will....@arm.com>
Signed-off-by: Andrew Morton <ak...@linux-foundation.org>
Signed-off-by: Linus Torvalds <torv...@linux-foundation.org>

ea1f5f3712 mm: define memblock_virt_alloc_try_nid_raw
a4a3ede213 mm: zero reserved and unavailable struct pages
0b412605ef Merge tag 'drm-fixes-for-v4.16-rc8' of git://people.freedesktop.org/~airlied/linux
7373fc81da Add linux-next specific files for 20180328
+--------------------------------------------------+------------+------------+------------+---------------+
| | ea1f5f3712 | a4a3ede213 | 0b412605ef | next-20180328 |
+--------------------------------------------------+------------+------------+------------+---------------+
| boot_successes | 0 | 0 | 0 | 0 |
| boot_failures | 44 | 11 | 17 | 11 |
| BUG:KASAN:use-after-scope_in__free_pages_bootmem | 44 | | | |
| BUG:KASAN:use-after-scope_in_c | 0 | 11 | 17 | 11 |
+--------------------------------------------------+------------+------------+------------+---------------+

[ 0.000000] ** **
[ 0.000000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
[ 0.000000] **********************************************************
[ 0.010000] NR_IRQS: 4352, nr_irqs: 256, preallocated irqs: 16
[ 0.010000] ==================================================================
[ 0.010000] BUG: KASAN: use-after-scope in console_unlock+0x303/0x645
[ 0.010000] Write of size 4 at addr ffffffff83607ac0 by task swapper/0
[ 0.010000]
[ 0.010000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0-04318-ga4a3ede #1
[ 0.010000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
[ 0.010000] Call Trace:
[ 0.010000] ? dump_stack+0x2a/0x39
[ 0.010000] ? print_address_description+0xb2/0x397
[ 0.010000] ? console_unlock+0x303/0x645
[ 0.010000] ? kasan_report+0x31b/0x36d
[ 0.010000] ? __asan_store4+0xe0/0xe8
[ 0.010000] ? console_unlock+0x303/0x645
[ 0.010000] ? wake_up_klogd+0x112/0x112
[ 0.010000] ? do_raw_spin_unlock+0x10d/0x118
[ 0.010000] ? arch_local_irq_restore+0xe/0x16
[ 0.010000] ? vprintk_emit+0x364/0x393
[ 0.010000] ? __down_trylock_console_sem+0x88/0x9e
[ 0.010000] ? vprintk_emit+0x364/0x393
[ 0.010000] ? vprintk_emit+0x37b/0x393
[ 0.010000] ? vprintk_default+0x20/0x28
[ 0.010000] ? vprintk_func+0x9a/0xa2
[ 0.010000] ? printk+0xa2/0xcc
[ 0.010000] ? show_regs_print_info+0x4e/0x4e
[ 0.010000] ? pte_offset_kernel+0x29/0x71
[ 0.010000] ? kasan_populate_zero_shadow+0x696/0x7a9
[ 0.010000] ? kasan_init+0x303/0x375
[ 0.010000] ? setup_arch+0x1d33/0x1efb
[ 0.010000] ? reserve_standard_io_resources+0x9d/0x9d
[ 0.010000] ? vprintk_emit+0x37b/0x393
[ 0.010000] ? vprintk_default+0x20/0x28
[ 0.010000] ? vprintk_func+0x9a/0xa2
[ 0.010000] ? printk+0xa2/0xcc
[ 0.010000] ? show_regs_print_info+0x4e/0x4e
[ 0.010000] ? cgroup_wq_init+0x8d/0x8d
[ 0.010000] ? load_idt+0x16/0x16
[ 0.010000] ? start_kernel+0x10e/0xb00
[ 0.010000] ? mem_encrypt_init+0x3a/0x3a
[ 0.010000] ? early_idt_handler_common+0x3b/0x52
[ 0.010000] ? x86_64_start_reservations+0x71/0x99
[ 0.010000] ? x86_64_start_kernel+0xeb/0x115
[ 0.010000] ? secondary_startup_64+0xa5/0xb0
[ 0.010000]
[ 0.010000] Memory state around the buggy address:
[ 0.010000] ffffffff83607980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 0.010000] ffffffff83607a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 0.010000] >ffffffff83607a80: 00 00 00 00 f1 f1 f1 f1 f8 f2 f2 f2 f2 f2 f2 f2

# HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD
git bisect start v4.15 v4.14 --
git bisect bad e017b4db26d03c1a6531f814ecc5ab41bcb889e9 # 09:33 B 0 11 25 0 Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad e0bcb42e602816415f6fe07313b6fc84932244b7 # 09:51 B 0 11 25 0 Merge tag 'ecryptfs-4.15-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tyhicks/ecryptfs
git bisect good 23c258763ba992f6a95a4b8980ffa7c1890bc8d8 # 10:13 G 11 0 11 11 Merge tag 'dmaengine-4.15-rc1' of git://git.infradead.org/users/vkoul/slave-dma
git bisect bad 93ea0eb7d77afab34657715630d692a78b8cea6a # 10:35 B 0 11 25 0 Merge tag 'leaks-4.15-rc1' of git://github.com/tcharding/linux
git bisect good 373c4557d2aa362702c4c2d41288fb1e54990b7c # 10:56 G 10 0 10 10 mm/pagewalk.c: report holes in hugetlb ranges
git bisect good 1bc03573e1c9024d4e4be97df4a1e0931edbae2c # 11:17 G 11 0 11 11 Merge branch 'for-4.15' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/libata
git bisect good ad0835a93008e5901415a0a27847d6a27649aa3a # 11:37 G 11 0 11 11 Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma
git bisect good 6363b3f3ac5be096d08c8c504128befa0c033529 # 11:55 G 11 0 11 11 Merge tag 'ipmi-for-4.15' of git://github.com/cminyard/linux-ipmi
git bisect bad 7c225c69f86c934e3be9be63ecde754e286838d7 # 12:11 B 0 11 25 0 Merge branch 'akpm' (patches from Andrew)
git bisect good 4be90299a1693c2112edb20ca78d6cc9f2183326 # 12:26 G 11 0 11 11 ceph: use pagevec_lookup_range_nr_tag()
git bisect bad 76253fbc8fbf6018401755fc5c07814a837cc832 # 12:47 B 0 1 15 0 mm: move accounting updates before page_cache_tree_delete()
git bisect good 353b1e7b5859e98860f984d8894fa7ddc242a90e # 13:24 G 11 0 11 11 x86/mm: set fields in deferred pages
git bisect bad 78c943662f4b1d53ddbfc515e427827915781377 # 13:44 B 0 11 25 0 sparc64: optimize struct page zeroing
git bisect bad a4a3ede2132ae0863e2d43e06f9b5697c51a7a3b # 13:59 B 0 11 25 0 mm: zero reserved and unavailable struct pages
git bisect good df8ee578894ebb591c2995cce422e6189c8bb757 # 14:18 G 11 0 11 11 sparc64: simplify vmemmap_populate
git bisect good ea1f5f3712afe895dfa4176ec87376b4a9ac23be # 14:31 G 11 0 11 11 mm: define memblock_virt_alloc_try_nid_raw
# first bad commit: [a4a3ede2132ae0863e2d43e06f9b5697c51a7a3b] mm: zero reserved and unavailable struct pages
git bisect good ea1f5f3712afe895dfa4176ec87376b4a9ac23be # 14:32 G 33 0 33 44 mm: define memblock_virt_alloc_try_nid_raw
# extra tests with debug options
git bisect bad a4a3ede2132ae0863e2d43e06f9b5697c51a7a3b # 14:45 B 0 11 25 0 mm: zero reserved and unavailable struct pages
# extra tests on HEAD of linux-devel/devel-catchup-201803290753
git bisect bad 5551da2e0f5324564f98bf5ac7d66449740c4934 # 14:46 B 0 13 30 0 0day head guard for 'devel-catchup-201803290753'
# extra tests on tree/branch linus/master
git bisect bad 0b412605ef5f5c64b31f19e2910b1d5eba9929c3 # 15:01 B 0 11 25 0 Merge tag 'drm-fixes-for-v4.16-rc8' of git://people.freedesktop.org/~airlied/linux
# extra tests on tree/branch linux-next/master
git bisect bad 7373fc81dadd068a8f2ea26011774f00f1f156bd # 15:24 B 0 11 25 0 Add linux-next specific files for 20180328

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/lkp Intel Corporation
dmesg-yocto-vp-17:20180329135720:x86_64-randconfig-s5-03290803:4.14.0-04318-ga4a3ede:1.gz
dmesg-yocto-vp-10:20180329143211:x86_64-randconfig-s5-03290803:4.14.0-04317-gea1f5f3:1.gz
reproduce-yocto-vp-17:20180329135720:x86_64-randconfig-s5-03290803:4.14.0-04318-ga4a3ede:1
config-4.14.0-04318-ga4a3ede

Fengguang Wu

unread,
Apr 18, 2018, 10:26:21 PM4/18/18
to Andrey Ryabinin, Pavel Tatashin, Steven Sistare, Daniel Jordan, Bob Picco, Linux Memory Management List, linux-...@vger.kernel.org, LKP, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Linus Torvalds, Nicolas Pitre, Nikitas Angelinas, Matt Redfearn, Dmitry Vyukov, kasa...@googlegroups.com
On Thu, Apr 19, 2018 at 10:17:57AM +0800, Fengguang Wu wrote:
>Hello,
>
>FYI this happens in mainline kernel 4.17.0-rc1.
>It at least dates back to v4.15-rc1 .
>
>The regression was reported before
>
> https://lkml.org/lkml/2017/11/30/33
>
>Where the last message from Dmitry mentions that use-after-scope has
>known false positives with CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL=y
>If so, what would be the best way to workaround such false positives
>in boot testing? Disable the above config?
>
>0day bisects produce diverged results, with 2 of them converge to
>commit d17a1d97dc ("x86/mm/kasan: don't use vmemmap_populate() to
>initialize shadow") and 1 bisected to the earlier a4a3ede213 ("mm:
>zero reserved and unavailable struct pages"). I'll send the bisect
>reports in follow up emails.

Here is the bisect report for

commit d17a1d97dc208d664c91cc387ffb752c7f85dc61
Author: Andrey Ryabinin <arya...@virtuozzo.com>
AuthorDate: Wed Nov 15 17:36:35 2017 -0800
Commit: Linus Torvalds <torv...@linux-foundation.org>
CommitDate: Wed Nov 15 18:21:05 2017 -0800

x86/mm/kasan: don't use vmemmap_populate() to initialize shadow

The kasan shadow is currently mapped using vmemmap_populate() since that
provides a semi-convenient way to map pages into init_top_pgt. However,
since that no longer zeroes the mapped pages, it is not suitable for
kasan, which requires zeroed shadow memory.

Add kasan_populate_shadow() interface and use it instead of
vmemmap_populate(). Besides, this allows us to take advantage of
gigantic pages and use them to populate the shadow, which should save us
some memory wasted on page tables and reduce TLB pressure.

Link: http://lkml.kernel.org/r/20171103185147.268...@oracle.com
Signed-off-by: Andrey Ryabinin <arya...@virtuozzo.com>
Signed-off-by: Pavel Tatashin <pasha.t...@oracle.com>
Cc: Steven Sistare <steven....@oracle.com>
Cc: Daniel Jordan <daniel....@oracle.com>
Cc: Bob Picco <bob....@oracle.com>
Cc: Michal Hocko <mho...@suse.com>
Cc: Alexander Potapenko <gli...@google.com>
Cc: Ard Biesheuvel <ard.bie...@linaro.org>
Cc: Catalin Marinas <catalin...@arm.com>
Cc: Christian Borntraeger <bornt...@de.ibm.com>
Cc: David S. Miller <da...@davemloft.net>
Cc: Dmitry Vyukov <dvy...@google.com>
Cc: Heiko Carstens <heiko.c...@de.ibm.com>
Cc: "H. Peter Anvin" <h...@zytor.com>
Cc: Ingo Molnar <mi...@redhat.com>
Cc: Mark Rutland <mark.r...@arm.com>
Cc: Matthew Wilcox <wi...@infradead.org>
Cc: Mel Gorman <mgo...@techsingularity.net>
Cc: Michal Hocko <mho...@kernel.org>
Cc: Sam Ravnborg <s...@ravnborg.org>
Cc: Thomas Gleixner <tg...@linutronix.de>
Cc: Will Deacon <will....@arm.com>
Signed-off-by: Andrew Morton <ak...@linux-foundation.org>
Signed-off-by: Linus Torvalds <torv...@linux-foundation.org>

a4a3ede213 mm: zero reserved and unavailable struct pages
d17a1d97dc x86/mm/kasan: don't use vmemmap_populate() to initialize shadow
d6bbd51587 Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace
73005e1a35 Add linux-next specific files for 20180103
+--------------------------------+------------+------------+------------+---------------+
| | a4a3ede213 | d17a1d97dc | d6bbd51587 | next-20180103 |
+--------------------------------+------------+------------+------------+---------------+
| boot_successes | 35 | 0 | 0 | 10 |
| boot_failures | 0 | 15 | 17 | |
| BUG:KASAN:use-after-scope_in_c | 0 | 15 | 17 | |
+--------------------------------+------------+------------+------------+---------------+

[ 0.004000] Tasks RCU enabled.
[ 0.004000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 0.004000] NR_IRQS: 4352, nr_irqs: 440, preallocated irqs: 16
[ 0.004000] Offload RCU callbacks from CPUs: .
[ 0.004000] ==================================================================
[ 0.004000] BUG: KASAN: use-after-scope in console_unlock+0x516/0x7bf
[ 0.004000] Write of size 4 at addr ffffffffaf207aa0 by task swapper/0
[ 0.004000]
[ 0.004000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0-04319-gd17a1d9 #2
[ 0.004000] Call Trace:
[ 0.004000] ? dump_stack+0xd1/0x178
[ 0.004000] ? _atomic_dec_and_lock+0x11a/0x11a
[ 0.004000] ? show_regs_print_info+0x51/0x51
[ 0.004000] ? do_raw_spin_unlock+0x223/0x247
[ 0.004000] ? print_address_description+0x94/0x2d9
[ 0.004000] ? console_unlock+0x516/0x7bf
[ 0.004000] ? kasan_report+0x21e/0x244
[ 0.004000] ? console_unlock+0x516/0x7bf
[ 0.004000] ? wake_up_klogd+0xe6/0xe6
[ 0.004000] ? vprintk_emit+0x3ee/0x426
[ 0.004000] ? __down_trylock_console_sem+0x5d/0x6c
[ 0.004000] ? vprintk_emit+0x3f7/0x426
[ 0.004000] ? console_unlock+0x7bf/0x7bf
[ 0.004000] ? memblock_virt_alloc_try_nid+0xd9/0x107
[ 0.004000] ? zero_pud_populate+0x7f1/0x8e8
[ 0.004000] ? printk+0x8f/0xab
[ 0.004000] ? show_regs_print_info+0x51/0x51
[ 0.004000] ? native_flush_tlb_global+0x71/0x7d
[ 0.004000] ? setup_arch+0x2427/0x2770
[ 0.004000] ? reserve_standard_io_resources+0x83/0x83
[ 0.004000] ? debug_check_no_locks_freed+0x20b/0x21a
[ 0.004000] ? __lockdep_init_map+0x20f/0x4d5
[ 0.004000] ? printk+0x8f/0xab
[ 0.004000] ? show_regs_print_info+0x51/0x51
[ 0.004000] ? cgroup_init_early+0xad/0x16e
[ 0.004000] ? do_device_not_available+0x4f/0x4f
[ 0.004000] ? start_kernel+0xe1/0x10ce
[ 0.004000] ? early_idt_handler_common+0x3b/0x60
[ 0.004000] ? thread_stack_cache_init+0x2e/0x2e
[ 0.004000] ? memcpy_orig+0x16/0x110
[ 0.004000] ? load_ucode_bsp+0x69/0x2fe
[ 0.004000] ? secondary_startup_64+0xa5/0xb0
[ 0.004000]
[ 0.004000] Memory state around the buggy address:
[ 0.004000] ffffffffaf207980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 0.004000] ffffffffaf207a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 0.004000] >ffffffffaf207a80: f1 f1 f1 f1 f8 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2

# HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD
git bisect start 30a7acd573899fd8b8ac39236eff6468b195ac7d v4.14 --
git bisect bad 4fbd8d194f06c8a3fd2af1ce560ddb31f7ec8323 # 02:00 B 0 11 25 0 Linux 4.15-rc1
git bisect bad 93ea0eb7d77afab34657715630d692a78b8cea6a # 02:21 B 0 11 25 0 Merge tag 'leaks-4.15-rc1' of git://github.com/tcharding/linux
git bisect good 32190f0afbf4f1c0a9142e5a886a078ee0b794fd # 02:39 G 11 0 0 0 Merge tag 'fscrypt-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/fscrypt
git bisect good 37cb8e1f8e10c6e9bd2a1b95cdda0620a21b0551 # 02:52 G 11 0 0 0 Merge tag 'devicetree-for-4.15' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux
git bisect good 6c4ba00c40d5acb17f32d4b7e02dbcd21f336d9f # 03:10 G 11 0 0 0 Merge tag 'hsi-for-4.15' of git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-hsi
git bisect good 766ec76a27aa9dfdfee3a80f29ddc1f7539c71f9 # 03:30 G 11 0 0 0 Merge branch 'for-4.15' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu
git bisect good 1b6115fbe3b3db746d7baa11399dd617fc75e1c4 # 03:54 G 11 0 0 0 Merge tag 'pci-v4.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
git bisect good 6363b3f3ac5be096d08c8c504128befa0c033529 # 04:09 G 11 0 0 0 Merge tag 'ipmi-for-4.15' of git://github.com/cminyard/linux-ipmi
git bisect bad 7c225c69f86c934e3be9be63ecde754e286838d7 # 04:25 B 0 1 15 0 Merge branch 'akpm' (patches from Andrew)
git bisect good 4be90299a1693c2112edb20ca78d6cc9f2183326 # 04:47 G 11 0 0 0 ceph: use pagevec_lookup_range_nr_tag()
git bisect bad 76253fbc8fbf6018401755fc5c07814a837cc832 # 05:07 B 0 2 16 0 mm: move accounting updates before page_cache_tree_delete()
git bisect good 353b1e7b5859e98860f984d8894fa7ddc242a90e # 05:29 G 11 0 0 0 x86/mm: set fields in deferred pages
git bisect bad 78c943662f4b1d53ddbfc515e427827915781377 # 05:51 B 0 3 17 0 sparc64: optimize struct page zeroing
git bisect good a4a3ede2132ae0863e2d43e06f9b5697c51a7a3b # 06:16 G 11 0 0 0 mm: zero reserved and unavailable struct pages
git bisect bad e17d8025f07e4fd9d73b137a8bcab04548126b83 # 06:29 B 0 10 24 0 arm64/mm/kasan: don't use vmemmap_populate() to initialize shadow
git bisect bad d17a1d97dc208d664c91cc387ffb752c7f85dc61 # 06:42 B 0 5 19 0 x86/mm/kasan: don't use vmemmap_populate() to initialize shadow
# first bad commit: [d17a1d97dc208d664c91cc387ffb752c7f85dc61] x86/mm/kasan: don't use vmemmap_populate() to initialize shadow
git bisect good a4a3ede2132ae0863e2d43e06f9b5697c51a7a3b # 06:51 G 31 0 0 0 mm: zero reserved and unavailable struct pages
# extra tests with debug options
git bisect bad d17a1d97dc208d664c91cc387ffb752c7f85dc61 # 07:06 B 0 10 24 0 x86/mm/kasan: don't use vmemmap_populate() to initialize shadow
# extra tests on HEAD of linux-devel/devel-hourly-2018010321
git bisect bad d23305f3c66383c30bc6a65b33dbdde7cabcf2e1 # 07:07 B 0 13 30 0 0day head guard for 'devel-hourly-2018010321'
# extra tests on tree/branch linus/master
git bisect bad d6bbd51587ecd173958453969964fb41140b1540 # 07:25 B 0 6 20 0 Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace
# extra tests on tree/branch linux-next/master
git bisect good 73005e1a35fd67c644b0645c9e4c1efabd0fe62c # 07:48 G 11 0 1 1 Add linux-next specific files for 20180103
dmesg-quantal-ivb41-10:20180104064259:x86_64-randconfig-s4-01040103:4.14.0-04319-gd17a1d9:2.gz
reproduce-quantal-ivb41-10:20180104064259:x86_64-randconfig-s4-01040103:4.14.0-04319-gd17a1d9:2
config-4.14.0-04319-gd17a1d9

Dmitry Vyukov

unread,
Apr 19, 2018, 2:04:37 AM4/19/18
to Fengguang Wu, Andrey Ryabinin, Pavel Tatashin, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Linus Torvalds, Nicolas Pitre, Nikitas Angelinas, Matt Redfearn, LKML, LKP, Steven Sistare, Daniel Jordan, Bob Picco, Linux Memory Management List, kasan-dev
On Thu, Apr 19, 2018 at 4:17 AM, Fengguang Wu <fenggu...@intel.com> wrote:
> Hello,
>
> FYI this happens in mainline kernel 4.17.0-rc1.
> It at least dates back to v4.15-rc1 .
>
> The regression was reported before
>
> https://lkml.org/lkml/2017/11/30/33
>
> Where the last message from Dmitry mentions that use-after-scope has
> known false positives with CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL=y
> If so, what would be the best way to workaround such false positives
> in boot testing? Disable the above config?

Hi Fengguang,

As a short-term solution, yes, disable one of them.
But we need a bug filed on structleak plugin, it inserts zeroing at a
wrong place.
We could also make them mutually exclusive in config to prevent people
from hitting these false positives again and again.
> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.
> To post to this group, send email to kasa...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20180419021757.66xxs5fgvlrusiup%40wfg-t540p.sh.intel.com.
> For more options, visit https://groups.google.com/d/optout.

Sergey Senozhatsky

unread,
Apr 19, 2018, 2:20:55 AM4/19/18
to Dmitry Vyukov, Fengguang Wu, Andrey Ryabinin, Pavel Tatashin, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Linus Torvalds, Nicolas Pitre, Nikitas Angelinas, Matt Redfearn, LKML, LKP, Steven Sistare, Daniel Jordan, Bob Picco, Linux Memory Management List, kasan-dev
On (04/19/18 08:04), Dmitry Vyukov wrote:
[..]
> We could also make them mutually exclusive in config to prevent people
> from hitting these false positives again and again.

Let's do it. Ard and Kees agreed on making them mutually exclusive [1][2].
Dmitry, could send out a patch?

[1] lkml.kernel.org/r/CAKv+Gu8HN-t2om8sCfjxCWbs...@mail.gmail.com
[2] lkml.kernel.org/r/CAGXu5j+mcfo4aB3PM1We6O62bFBJcMFX-9obJE4jFU1Dp=gN...@mail.gmail.com

-ss

Fengguang Wu

unread,
Apr 19, 2018, 2:55:39 AM4/19/18
to Sergey Senozhatsky, Dmitry Vyukov, Andrey Ryabinin, Pavel Tatashin, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Linus Torvalds, Nicolas Pitre, Nikitas Angelinas, Matt Redfearn, LKML, LKP, Steven Sistare, Daniel Jordan, Bob Picco, Linux Memory Management List, kasan-dev
That'd be great, thank you very much!

Cheers,
Fengguang

Dmitry Vyukov

unread,
Apr 19, 2018, 5:50:41 AM4/19/18
to Fengguang Wu, Sergey Senozhatsky, Andrey Ryabinin, Pavel Tatashin, Petr Mladek, Sergey Senozhatsky, Steven Rostedt, Linus Torvalds, Nicolas Pitre, Nikitas Angelinas, Matt Redfearn, LKML, LKP, Steven Sistare, Daniel Jordan, Bob Picco, Linux Memory Management List, kasan-dev
Just mailed "KASAN: prohibit KASAN+STRUCTLEAK combination":
https://groups.google.com/d/msg/kasan-dev/Y1TEh7ZlHTQ/wR36C8uMCgAJ
Reply all
Reply to author
Forward
0 new messages