v4.14-rc3/arm64 DABT exception in atomic_inc() / __skb_clone()

[Wei Wei]

Hi all,

I have fuzzed v4.14-rc3 using syzkaller and found a bug similar to that one [1].
But the call trace isn’t the same. The atomic_inc() might handle a corrupted
skb_buff.

The logs and config have been uploaded to my github repo [2].

[1] https://lkml.org/lkml/2017/10/2/216
[2] https://github.com/dotweiba/skb_clone_atomic_inc_bug

Thanks,
Wei

Unable to handle kernel paging request at virtual address ffff80005bfb81ed
Mem abort info:
Exception class = DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000033
CM = 0, WnR = 0
swapper pgtable: 4k pages, 48-bit VAs, pgd = ffff20000b366000
[ffff80005bfb81ed] *pgd=00000000beff7003, *pud=00e8000080000711
Internal error: Oops: 96000021 [#1] PREEMPT SMP
Modules linked in:
CPU: 3 PID: 4725 Comm: syz-executor0 Not tainted 4.14.0-rc3 #3
Hardware name: linux,dummy-virt (DT)
task: ffff800074409e00 task.stack: ffff800033db0000
PC is at __skb_clone+0x430/0x5b0
LR is at __skb_clone+0x1dc/0x5b0
pc : [<ffff200009705f50>] lr : [<ffff200009705cfc>] pstate: 10000145
sp : ffff800033db33d0
x29: ffff800033db33d0 x28: ffff2000098ac378
x27: ffff100006a860e1 x26: 1ffff000067b66b6
x25: ffff8000743340a0 x24: ffff800035430708
x23: ffff80005bfb80c9 x22: ffff800035430710
x21: 0000000000000380 x20: ffff800035430640
x19: ffff8000354312c0 x18: 0000000000000000
x17: 00000000004af000 x16: ffff20000845e8c8
x15: 000000001e518060 x14: 0000ffffd8316070
x13: 0000ffffd8316090 x12: ffffffffffffffff
x11: 1ffff00006a8626f x10: ffff100006a8626f
x9 : dfff200000000000 x8 : 0082009000900608
x7 : 0000000000000000 x6 : ffff800035431380
x5 : ffff100006a86270 x4 : 0000000000000000
x3 : 1ffff00006a86273 x2 : 0000000000000000
x1 : 0000000000000100 x0 : ffff80005bfb81ed
Process syz-executor0 (pid: 4725, stack limit = 0xffff800033db0000)
Call trace:
Exception stack(0xffff800033db3290 to 0xffff800033db33d0)
3280: ffff80005bfb81ed 0000000000000100
32a0: 0000000000000000 1ffff00006a86273 0000000000000000 ffff100006a86270
32c0: ffff800035431380 0000000000000000 0082009000900608 dfff200000000000
32e0: ffff100006a8626f 1ffff00006a8626f ffffffffffffffff 0000ffffd8316090
3300: 0000ffffd8316070 000000001e518060 ffff20000845e8c8 00000000004af000
3320: 0000000000000000 ffff8000354312c0 ffff800035430640 0000000000000380
3340: ffff800035430710 ffff80005bfb80c9 ffff800035430708 ffff8000743340a0
3360: 1ffff000067b66b6 ffff100006a860e1 ffff2000098ac378 ffff800033db33d0
3380: ffff200009705cfc ffff800033db33d0 ffff200009705f50 0000000010000145
33a0: ffff8000354312c0 ffff800035430640 0001000000000000 ffff800074334000
33c0: ffff800033db33d0 ffff200009705f50
[<ffff200009705f50>] __skb_clone+0x430/0x5b0
[<ffff20000971520c>] skb_clone+0x164/0x2c8
[<ffff2000098ac498>] arp_rcv+0x120/0x488
[<ffff200009741878>] __netif_receive_skb_core+0x11e8/0x18c8
[<ffff2000097479b0>] __netif_receive_skb+0x30/0x198
[<ffff200009751fd8>] netif_receive_skb_internal+0x98/0x370
[<ffff2000097522cc>] netif_receive_skb+0x1c/0x28
[<ffff2000090730e0>] tun_get_user+0x12f0/0x2e40
[<ffff200009074ddc>] tun_chr_write_iter+0xbc/0x140
[<ffff200008457284>] do_iter_readv_writev+0x2d4/0x468
[<ffff20000845a5a0>] do_iter_write+0x148/0x498
[<ffff20000845aac0>] vfs_writev+0x118/0x250
[<ffff20000845acbc>] do_writev+0xc4/0x1e8
[<ffff20000845e8fc>] SyS_writev+0x34/0x48
Exception stack(0xffff800033db3ec0 to 0xffff800033db4000)
3ec0: 0000000000000015 0000ffff829985e0 0000000000000001 0000ffff8299851c
3ee0: 0000ffff82999068 0000ffff82998f60 0000ffff82999650 0000000000000000
3f00: 0000000000000042 0000000000000036 0000000000406608 0000ffff82998400
3f20: 0000ffff82998f60 0000ffffd8316090 0000ffffd8316070 000000001e518060
3f40: 0000000000000000 00000000004af000 0000000000000000 0000000000000036
3f60: 0000000020004fca 0000000020000000 000000000046ccf0 0000000000000530
3f80: 000000000046cce8 00000000004ade98 0000000000000000 00000000395fa6f0
3fa0: 0000ffff82998f60 0000ffff82998560 0000000000431448 0000ffff82998520
3fc0: 000000000043145c 0000000080000000 0000000000000015 0000000000000042
3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[<ffff200008083ef0>] el0_svc_naked+0x24/0x28
Code: f9406680 8b010000 91009000 f9800011 (885f7c01)
---[ end trace 261e7ac1458ccc0a ]---

[Eric Dumazet]

Please provide proper file:line information in this trace.

You can use scripts/decode_stacktrace.sh

Thanks.

[Wei Wei]

Sry. Here it is.

Unable to handle kernel paging request at virtual address ffff80005bfb81ed
Mem abort info:
Exception class = DABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
Data abort info:
ISV = 0, ISS = 0x00000033
CM = 0, WnR = 0
swapper pgtable: 4k pages, 48-bit VAs, pgd = ffff20000b366000
[ffff80005bfb81ed] *pgd=00000000beff7003, *pud=00e8000080000711
Internal error: Oops: 96000021 [#1] PREEMPT SMP
Modules linked in:
CPU: 3 PID: 4725 Comm: syz-executor0 Not tainted 4.14.0-rc3 #3
Hardware name: linux,dummy-virt (DT)
task: ffff800074409e00 task.stack: ffff800033db0000

PC is at __skb_clone (/./arch/arm64/include/asm/atomic_ll_sc.h:113 (discriminator 4) /net/core/skbuff.c:873 (discriminator 4))
LR is at __skb_clone (/net/core/skbuff.c:861 (discriminator 4))
pc : lr : pstate: 10000145

__skb_clone (/./arch/arm64/include/asm/atomic_ll_sc.h:113 (discriminator 4) /net/core/skbuff.c:873 (discriminator 4))
skb_clone (/net/core/skbuff.c:1286)
arp_rcv (/./include/linux/skbuff.h:1518 /net/ipv4/arp.c:946)
__netif_receive_skb_core (/net/core/dev.c:1859 /net/core/dev.c:1874 /net/core/dev.c:4416)
__netif_receive_skb (/net/core/dev.c:4466)
netif_receive_skb_internal (/net/core/dev.c:4539)
netif_receive_skb (/net/core/dev.c:4564)
tun_get_user (/./include/linux/bottom_half.h:31 /drivers/net/tun.c:1219 /drivers/net/tun.c:1553)
tun_chr_write_iter (/drivers/net/tun.c:1579)
do_iter_readv_writev (/./include/linux/fs.h:1770 /fs/read_write.c:673)
do_iter_write (/fs/read_write.c:952)
vfs_writev (/fs/read_write.c:997)
do_writev (/fs/read_write.c:1032)
SyS_writev (/fs/read_write.c:1102)

Exception stack(0xffff800033db3ec0 to 0xffff800033db4000)
3ec0: 0000000000000015 0000ffff829985e0 0000000000000001 0000ffff8299851c
3ee0: 0000ffff82999068 0000ffff82998f60 0000ffff82999650 0000000000000000
3f00: 0000000000000042 0000000000000036 0000000000406608 0000ffff82998400
3f20: 0000ffff82998f60 0000ffffd8316090 0000ffffd8316070 000000001e518060
3f40: 0000000000000000 00000000004af000 0000000000000000 0000000000000036
3f60: 0000000020004fca 0000000020000000 000000000046ccf0 0000000000000530
3f80: 000000000046cce8 00000000004ade98 0000000000000000 00000000395fa6f0
3fa0: 0000ffff82998f60 0000ffff82998560 0000000000431448 0000ffff82998520
3fc0: 000000000043145c 0000000080000000 0000000000000015 0000000000000042
3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000

el0_svc_naked (/arch/arm64/kernel/entry.S:853)

Code: f9406680 8b010000 91009000 f9800011 (885f7c01)

All code
========
0: 80 66 40 f9 andb $0xf9,0x40(%rsi)
4: 00 00 add %al,(%rax)
6: 01 8b 00 90 00 91 add %ecx,-0x6eff7000(%rbx)
c: 11 00 adc %eax,(%rax)
e: 80 f9 01 cmp $0x1,%cl
11: 7c 5f jl 0x72
13:* 88 00 mov %al,(%rax) <-- trapping instruction
15: 00 00 add %al,(%rax)
...

Code starting with the faulting instruction
===========================================
0: 01 7c 5f 88 add %edi,-0x78(%rdi,%rbx,2)
4: 00 00 add %al,(%rax)
...
—[ end trace 261e7ac1458ccc0a ]---

Thanks,
Wei

[Eric Dumazet]

I thought it was happening on arm64 ?

This is x86_64 disassembly :/

Thanks.

[Will Deacon]

On Thu, Oct 19, 2017 at 10:34:54PM -0700, Eric Dumazet wrote:
> On Thu, Oct 19, 2017 at 8:13 PM, Wei Wei <dotw...@gmail.com> wrote:
> > Code: f9406680 8b010000 91009000 f9800011 (885f7c01)
> > All code
> > ========
> > 0: 80 66 40 f9 andb $0xf9,0x40(%rsi)
> > 4: 00 00 add %al,(%rax)
> > 6: 01 8b 00 90 00 91 add %ecx,-0x6eff7000(%rbx)
> > c: 11 00 adc %eax,(%rax)
> > e: 80 f9 01 cmp $0x1,%cl
> > 11: 7c 5f jl 0x72
> > 13:* 88 00 mov %al,(%rax) <-- trapping instruction
> > 15: 00 00 add %al,(%rax)
> > ...
> >
> > Code starting with the faulting instruction
> > ===========================================
> > 0: 01 7c 5f 88 add %edi,-0x78(%rdi,%rbx,2)
> > 4: 00 00 add %al,(%rax)
> > ...
> > —[ end trace 261e7ac1458ccc0a ]---
> >
>
> I thought it was happening on arm64 ?
>
> This is x86_64 disassembly :/

I guess they forgot the ARCH/CROSS_COMPILE env vars for decodecode. here
you go:

Code: f9406680 8b010000 91009000 f9800011 (885f7c01)
All code
========

0: f9406680 ldr x0, [x20,#200]
4: 8b010000 add x0, x0, x1
8: 91009000 add x0, x0, #0x24
c: f9800011 prfm pstl1strm, [x0]
10:* 885f7c01 ldxr w1, [x0] <-- trapping instruction

Code starting with the faulting instruction
===========================================

0: 885f7c01 ldxr w1, [x0]

so it's faulting on the load part of an atomic rmw.

Will

[Mark Rutland]

On Thu, Oct 19, 2017 at 10:16:08PM -0400, Wei Wei wrote:
> Hi all,

Hi,

> I have fuzzed v4.14-rc3 using syzkaller and found a bug similar to that one [1].
> But the call trace isn’t the same. The atomic_inc() might handle a corrupted
> skb_buff.
>
> The logs and config have been uploaded to my github repo [2].
>
> [1] https://lkml.org/lkml/2017/10/2/216
> [2] https://github.com/dotweiba/skb_clone_atomic_inc_bug

These do look very similar to what I was hitting; all appear to be
misaligned atomics in the same path.

I see that you have some empty repro files in [2]. If you have any
reproducers, would you mind sharing them?

If any of those are smaller or more reliable than the one I was able to
generate [3], it might make it more obvious what's going on, and/or make
it simpler to come up with a plain C reproducer.

Thanks,
Mark.

[3] https://www.kernel.org/pub/linux/kernel/people/mark/bugs/20171002-skb_clone-misaligned-atomic/syzkaller.repro

[Wei Wei]

Sadly, the syzkaller characterized it as a non-reproducible bug and there were empty
repro files. But if manually executing in VM like this “./syz-execprog -executor=
./syz-executor -repeat=0 -procs=16 -cover=0 crash-log”, it crashed when executing exactly
program 1056 using log0 provided.

I failed to generate the C reproducer with syz-repro as it said “no target compiler”
in the final step. I would appreciate if you could give some hints.

Thanks,
Wei

[Mark Rutland]

On Fri, Oct 20, 2017 at 10:40:38AM -0400, Wei Wei wrote:
> Sadly, the syzkaller characterized it as a non-reproducible bug and there were empty
> repro files. But if manually executing in VM like this “./syz-execprog -executor=
> ./syz-executor -repeat=0 -procs=16 -cover=0 crash-log”, it crashed when executing exactly
> program 1056 using log0 provided.
>
> I failed to generate the C reproducer with syz-repro as it said “no target compiler”
> in the final step. I would appreciate if you could give some hints.

syz-repro should produce a smaller syzkaller log before it tries to
generate a C file.

I use:

$ syz-repro -config qemu.cfg logN

... and in most cases it will eventually print a smaller log to the
console.

Thanks,
Mark.

[Dmitry Vyukov]

On Fri, Oct 20, 2017 at 4:40 PM, Wei Wei <dotw...@gmail.com> wrote:
> Sadly, the syzkaller characterized it as a non-reproducible bug and there were empty
> repro files. But if manually executing in VM like this “./syz-execprog -executor=
> ./syz-executor -repeat=0 -procs=16 -cover=0 crash-log”, it crashed when executing exactly
> program 1056 using log0 provided.
>
> I failed to generate the C reproducer with syz-repro as it said “no target compiler”
> in the final step. I would appreciate if you could give some hints.

syzkaller tries to use aarch64-linux-gnu-gcc when cross-compiling to arm64:
https://github.com/google/syzkaller/blob/master/sys/targets/targets.go#L62
Try to install g++-aarch64-linux-gnu.
Or how should it be done on your system?

> Thanks,
> Wei
>> On 20 Oct 2017, at 7:14 AM, Mark Rutland <mark.r...@arm.com> wrote:
>>
>> On Thu, Oct 19, 2017 at 10:16:08PM -0400, Wei Wei wrote:
>>> Hi all,
>>
>> Hi,
>>
>>> I have fuzzed v4.14-rc3 using syzkaller and found a bug similar to that one [1].
>>> But the call trace isn’t the same. The atomic_inc() might handle a corrupted
>>> skb_buff.
>>>
>>> The logs and config have been uploaded to my github repo [2].
>>>
>>> [1] https://lkml.org/lkml/2017/10/2/216
>>> [2] https://github.com/dotweiba/skb_clone_atomic_inc_bug
>>
>> These do look very similar to what I was hitting; all appear to be
>> misaligned atomics in the same path.
>>
>> I see that you have some empty repro files in [2]. If you have any
>> reproducers, would you mind sharing them?
>>
>> If any of those are smaller or more reliable than the one I was able to
>> generate [3], it might make it more obvious what's going on, and/or make
>> it simpler to come up with a plain C reproducer.
>>
>> Thanks,
>> Mark.
>>
>> [3] https://www.kernel.org/pub/linux/kernel/people/mark/bugs/20171002-skb_clone-misaligned-atomic/syzkaller.repro
>

> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Willem de Bruijn]

On Fri, Oct 20, 2017 at 11:14 AM, Dmitry Vyukov <dvy...@google.com> wrote:
> On Fri, Oct 20, 2017 at 4:40 PM, Wei Wei <dotw...@gmail.com> wrote:
>> Sadly, the syzkaller characterized it as a non-reproducible bug and there were empty
>> repro files. But if manually executing in VM like this “./syz-execprog -executor=
>> ./syz-executor -repeat=0 -procs=16 -cover=0 crash-log”, it crashed when executing exactly
>> program 1056 using log0 provided.
>>
>> I failed to generate the C reproducer with syz-repro as it said “no target compiler”
>> in the final step. I would appreciate if you could give some hints.
>
> syzkaller tries to use aarch64-linux-gnu-gcc when cross-compiling to arm64:
> https://github.com/google/syzkaller/blob/master/sys/targets/targets.go#L62
> Try to install g++-aarch64-linux-gnu.
> Or how should it be done on your system?

A core dump would also be helpful to root around in and inspect
what those registers point to. Thanks for posting the various reports
on github, btw.

[Dmitry Vyukov]

On Fri, Oct 20, 2017 at 8:34 PM, Evan Wei <dotw...@gmail.com> wrote:
> Sorry, didn't make it quite clear. I have manually set up the toolchain. The
> point is that it could not find the
> toolchain. How can I refer to that aarch64-linux-gnu-g++ when executing
> syz-repro? I'd apprecite your advice.

Add it to PATH. Or a link to it. Note: it's gcc, not g++.




> thanks,
> wei

>
> On Fri, Oct 20, 2017 at 11:14 AM, Dmitry Vyukov <dvy...@google.com> wrote:
>>

[Wei Wei]

I have uploaded the VM core dump [1]. And I don’t know if these logs are helpful in the case of
failing to get the C reproducer currently.

[1] https://github.com/dotweiba/skb_clone_atomic_inc_bug/blob/master/vmcore.gz

2017/10/21 20:24:32 reproducing crash 'unable to handle kernel paging request in __skb_clone': testing program (duration=24s, {Threaded:true Collide:true Repeat:true Procs:8 Sandb
ox:setuid Fault:false FaultCall:-1 FaultNth:0 EnableTun:true UseTmpDir:true HandleSegv:true WaitRepeat:true Debug:false Repro:true}): mmap-socket$inet_tcp-bind$inet-sendto$inet-se
ndto$inet-syz_emit_ethernet
2017/10/21 20:24:49 reproducing crash 'unable to handle kernel paging request in __skb_clone': program crashed: unable to handle kernel paging request in __skb_clone
2017/10/21 20:24:49 reproducing crash 'unable to handle kernel paging request in __skb_clone': extracting C reproducer
2017/10/21 20:24:49 reproducing crash 'unable to handle kernel paging request in __skb_clone': reproducing took 1h47m5.070207729s
2017/10/21 20:24:49 reproduction failed: no target compiler

Thanks,
Wei

[Willem de Bruijn]

On Sat, Oct 21, 2017 at 9:56 PM, Wei Wei <dotw...@gmail.com> wrote:
> I have uploaded the VM core dump [1]. And I don’t know if these logs are helpful in the case of
> failing to get the C reproducer currently.
>
> [1] https://github.com/dotweiba/skb_clone_atomic_inc_bug/blob/master/vmcore.gz

Thanks. So this would be the atomic_inc on shb_shinfo(skb)->dataref, which
matches the __ll_sc_atomic_add in Mark's trace.

Debugging with crash shows 0xffff800071bb3180 and 0xffff800071bb2c80
to be valid skbuffs of len 40, no sk, both pointing to the same head.

That is indeed unaligned: head = 0xffff8000327c80c9 "", end = 256, giving
skb_shared_info at 0xffff8000327c81c9 and &skb_shared_info(skb)->dataref
at 0xffff8000327c81c9 + 36 == 0xffff8000327c81ed

[Willem de Bruijn]

From skb->dev and netdev_priv, the tun device has flags 0x1002 ==
IFF_TAP | IFF_NO_PI. This kernel precedes the recent support for
IFF_NAPI and IFF_NAPI_FRAGS. The allocation most likely happened
in tun_build_skb from current->task_frag. It would be a previous
allocation that left alloc_frag->offset unaligned. But perhaps this code
needs to perform alignment before setting skb->head. At least on
platforms where atomic on dataref must be aligned.

[Eric Dumazet]

+1

Bug added in commit 66ccbc9c87c2 ("tap: use build_skb() for small packet")

[Jason Wang]

Thanks, will post a patch.

[David Laight]

From: Willem de Bruijn
> Sent: 25 October 2017 19:50
...

> From skb->dev and netdev_priv, the tun device has flags 0x1002 ==
> IFF_TAP | IFF_NO_PI. This kernel precedes the recent support for
> IFF_NAPI and IFF_NAPI_FRAGS. The allocation most likely happened
> in tun_build_skb from current->task_frag. It would be a previous
> allocation that left alloc_frag->offset unaligned. But perhaps this code
> needs to perform alignment before setting skb->head.
>
> At least on platforms where atomic on dataref must be aligned.

Isn't that true of almost everything?
I'm not even sure x86 always (ever?) manages locked cycles on
misaligned addresses.

David