Re: [lkp-robot] [x86/cpu_entry_area] 10043e02db: kernel_BUG_at_arch/x86/mm/physaddr.c

[Thomas Gleixner]

On Tue, 26 Dec 2017, kernel test robot wrote:

>
> FYI, we noticed the following commit (built with gcc-6):
>
> commit: 10043e02db7f8a4161f76434931051e7d797a5f6 ("x86/cpu_entry_area: Add debugstore entries to cpu_entry_area")
> https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git WIP.x86/pti

...

> [ 0.000000] kernel BUG at arch/x86/mm/physaddr.c:27!
> PANIC: early exception 0x06 IP 10:ffffffff8115586f error 0 cr2 0xffff88000e468000
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.0-00160-g10043e02 #1
> [ 0.000000] task: ffffffff8a4683c0 task.stack: ffffffff8a400000
> [ 0.000000] RIP: 0010:__phys_addr+0x268/0x276
> [ 0.000000] RSP: 0000:ffffffff8a407bd8 EFLAGS: 00010002 ORIG_RAX: 0000000000000000
> [ 0.000000] RAX: 0000000000000000 RBX: 0000780000000000 RCX: 1ffffffff17a9a01
> [ 0.000000] RDX: dffffc0000000000 RSI: 0000000000000003 RDI: ffffffff8bd4d340
> [ 0.000000] RBP: ffffffff8a407bf8 R08: 0000000000000001 R09: ffffffff8a407a48
> [ 0.000000] R10: ffff880000010000 R11: ffff880000010fff R12: 0000000000000001
> [ 0.000000] R13: 0000000000000001 R14: 0000000000000000 R15: fffffbd00c401000
> [ 0.000000] FS: 0000000000000000(0000) GS:ffffffff8cb4d000(0000) knlGS:0000000000000000
> [ 0.000000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 0.000000] CR2: ffff88000e468000 CR3: 000000000cde8000 CR4: 00000000000406b0
> [ 0.000000] Call Trace:
> [ 0.000000] kasan_populate_shadow+0x3f2/0x497

So this dies simply because kasan_populate_shadow() runs out of memory and
has no sanity check whatsoever.

static __init void *early_alloc(size_t size, int nid)
{
return memblock_virt_alloc_try_nid_nopanic(size, size,
__pa(MAX_DMA_ADDRESS), BOOTMEM_ALLOC_ACCESSIBLE, nid);
}

kasan_populate_pmd()
{
.....

p = early_alloc(PAGE_SIZE, nid);
entry = pfn_pte(PFN_DOWN(__pa(p)), PAGE_KERNEL);

I've instrumented the whole thing and early_alloc() returns NULL at some
point and then __pa(NULL) dies in the VIRTUAL_DEBUG code. Well, it would
die with VIRTUAL_DEBUG=n as well at some other place.

Not really a problem caused by the patch above, it's merily exposing a code
path which relies blindly on "enough memory available" assumptions.

Throwing more memory at the VM makes the problem go away...

Thanks,

tglx

[Dmitry Vyukov]

Hi Thomas,

We just need a check inside of early_alloc() to properly diagnose such
situation, right?
KASAN allocates 1/8 of physical memory, so the code has _some_ logic
behind it. But I guess it runs into a corner case where 1/8 is somehow
larger than 1 :)

[Thomas Gleixner]

At least you want to panic with a proper out of memory message. But letting
the thing die at a random place is a bad idea.

Thanks,

tglx

[Dmitry Vyukov]

Thanks. I will cook a patch (if Andrey won't beat me to it).

[Andrey Ryabinin]

On 12/27/2017 09:12 PM, Dmitry Vyukov wrote:

>>
>> Not really a problem caused by the patch above, it's merily exposing a code
>> path which relies blindly on "enough memory available" assumptions.
>>
>> Throwing more memory at the VM makes the problem go away...
>
> Hi Thomas,
>
> We just need a check inside of early_alloc() to properly diagnose such
> situation, right?
> KASAN allocates 1/8 of physical memory, so the code has _some_ logic
> behind it. But I guess it runs into a corner case where 1/8 is somehow
> larger than 1 :)
>

We may simply don't have 1/8 available. And we actually need slightly more
than 1/8. Kernel image is present in 2 locations (direct mapping/kernel mapping)
so at least ram/8 + kernel_size/8. The same for cpu_entry_area. Plus some memory
to build page tables.
And note that kernel_size could be hundreds of megabytes.

[Andrey Ryabinin]

We probably should panic only if PAGE_SIZE allocation failed. PUD_SIZE,PMD_SIZE allocations have
failure fallback. I would suggest add 'bool panic' param to early_alloc() and call
memblock_virt_alloc_try_nid() if it's true.

[Dmitry Vyukov]

FTR, I filed https://bugzilla.kernel.org/show_bug.cgi?id=198427 for
this so it's not get lost.

[Andrey Ryabinin]

Currently KASAN doesn't panic in case it don't have enough memory
to boot. Instead, it crashes in some random place:

kernel BUG at arch/x86/mm/physaddr.c:27!

RIP: 0010:__phys_addr+0x268/0x276
Call Trace:
kasan_populate_shadow+0x3f2/0x497
kasan_init+0x12e/0x2b2
setup_arch+0x2825/0x2a2c
start_kernel+0xc8/0x15f4
x86_64_start_reservations+0x2a/0x2c
x86_64_start_kernel+0x72/0x75
secondary_startup_64+0xa5/0xb0

Use memblock_virt_alloc_try_nid() for allocations without failure
fallback. It will panic with an out of memory message.

Reported-by: kernel test robot <xiaol...@intel.com>
Signed-off-by: Andrey Ryabinin <arya...@virtuozzo.com>
---
arch/x86/mm/kasan_init_64.c | 24 ++++++++++++++----------
1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index 47388f0c0e59..af6f2f9c6a26 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -21,10 +21,14 @@ extern struct range pfn_mapped[E820_MAX_ENTRIES];

static p4d_t tmp_p4d_table[PTRS_PER_P4D] __initdata __aligned(PAGE_SIZE);

-static __init void *early_alloc(size_t size, int nid)
+static __init void *early_alloc(size_t size, int nid, bool panic)
{
- return memblock_virt_alloc_try_nid_nopanic(size, size,
- __pa(MAX_DMA_ADDRESS), BOOTMEM_ALLOC_ACCESSIBLE, nid);
+ if (panic)
+ return memblock_virt_alloc_try_nid(size, size,
+ __pa(MAX_DMA_ADDRESS), BOOTMEM_ALLOC_ACCESSIBLE, nid);
+ else
+ return memblock_virt_alloc_try_nid_nopanic(size, size,
+ __pa(MAX_DMA_ADDRESS), BOOTMEM_ALLOC_ACCESSIBLE, nid);
}

static void __init kasan_populate_pmd(pmd_t *pmd, unsigned long addr,
@@ -38,14 +42,14 @@ static void __init kasan_populate_pmd(pmd_t *pmd, unsigned long addr,
if (boot_cpu_has(X86_FEATURE_PSE) &&
((end - addr) == PMD_SIZE) &&
IS_ALIGNED(addr, PMD_SIZE)) {
- p = early_alloc(PMD_SIZE, nid);
+ p = early_alloc(PMD_SIZE, nid, false);
if (p && pmd_set_huge(pmd, __pa(p), PAGE_KERNEL))
return;
else if (p)
memblock_free(__pa(p), PMD_SIZE);
}

- p = early_alloc(PAGE_SIZE, nid);
+ p = early_alloc(PAGE_SIZE, nid, true);
pmd_populate_kernel(&init_mm, pmd, p);
}

@@ -57,7 +61,7 @@ static void __init kasan_populate_pmd(pmd_t *pmd, unsigned long addr,
if (!pte_none(*pte))
continue;

- p = early_alloc(PAGE_SIZE, nid);
+ p = early_alloc(PAGE_SIZE, nid, true);
entry = pfn_pte(PFN_DOWN(__pa(p)), PAGE_KERNEL);
set_pte_at(&init_mm, addr, pte, entry);
} while (pte++, addr += PAGE_SIZE, addr != end);
@@ -75,14 +79,14 @@ static void __init kasan_populate_pud(pud_t *pud, unsigned long addr,
if (boot_cpu_has(X86_FEATURE_GBPAGES) &&
((end - addr) == PUD_SIZE) &&
IS_ALIGNED(addr, PUD_SIZE)) {
- p = early_alloc(PUD_SIZE, nid);
+ p = early_alloc(PUD_SIZE, nid, false);
if (p && pud_set_huge(pud, __pa(p), PAGE_KERNEL))
return;
else if (p)
memblock_free(__pa(p), PUD_SIZE);
}

- p = early_alloc(PAGE_SIZE, nid);
+ p = early_alloc(PAGE_SIZE, nid, true);
pud_populate(&init_mm, pud, p);
}

@@ -101,7 +105,7 @@ static void __init kasan_populate_p4d(p4d_t *p4d, unsigned long addr,
unsigned long next;

if (p4d_none(*p4d)) {
- void *p = early_alloc(PAGE_SIZE, nid);
+ void *p = early_alloc(PAGE_SIZE, nid, true);

p4d_populate(&init_mm, p4d, p);
}
@@ -122,7 +126,7 @@ static void __init kasan_populate_pgd(pgd_t *pgd, unsigned long addr,
unsigned long next;

if (pgd_none(*pgd)) {
- p = early_alloc(PAGE_SIZE, nid);
+ p = early_alloc(PAGE_SIZE, nid, true);
pgd_populate(&init_mm, pgd, p);
}

--
2.13.6

[Dmitry Vyukov]

Acked-by: Dmitry Vyukov <dvy...@google.com>

Thanks!