WARNING in task_participate_group_stop

[syzbot]

Hello,

syzkaller hit the following crash on
d95e159cd1da1ed4dbf76bf203e8ffaf231395e4
git://git.cmpxchg.org/linux-mmots.git/master
compiler: gcc (GCC) 7.1.1 20170620
.config is attached
Raw console output is attached.
C reproducer is attached
syzkaller reproducer is attached. See https://goo.gl/kgGztJ
for information about syzkaller reproducers


WARNING: CPU: 0 PID: 1 at kernel/signal.c:340
task_participate_group_stop+0x1ce/0x230 kernel/signal.c:340
Kernel panic - not syncing: panic_on_warn set ...

CPU: 0 PID: 1 Comm: init Not tainted 4.13.0-mm1+ #5
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:16 [inline]
dump_stack+0x194/0x257 lib/dump_stack.c:52
panic+0x1e4/0x417 kernel/panic.c:181
__warn+0x1c4/0x1d9 kernel/panic.c:542
report_bug+0x211/0x2d0 lib/bug.c:183
fixup_bug+0x40/0x90 arch/x86/kernel/traps.c:178
do_trap_no_signal arch/x86/kernel/traps.c:212 [inline]
do_trap+0x260/0x390 arch/x86/kernel/traps.c:261
do_error_trap+0x120/0x390 arch/x86/kernel/traps.c:298
do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:311
invalid_op+0x18/0x20 arch/x86/entry/entry_64.S:905
RIP: 0010:task_participate_group_stop+0x1ce/0x230 kernel/signal.c:340
RSP: 0018:ffff8801d9ee77f0 EFLAGS: 00010097
RAX: ffff8801d9ed8040 RBX: ffff8801d9ed8040 RCX: ffff8801d9edb2c0
RDX: 0000000000000000 RSI: 0000000000060013 RDI: ffff8801d9ed84d0
RBP: ffff8801d9ee7808 R08: ffff8801d9ee7180 R09: ffff8801d9ee7178
R10: ffff8801d9ee70f0 R11: 1ffff1003b3db29b R12: ffff8801d9ee9740
R13: 0000000000000000 R14: dffffc0000000000 R15: ffff8801d9ed85c8
do_signal_stop+0x217/0x900 kernel/signal.c:2042
get_signal+0x61c/0x17e0 kernel/signal.c:2297
do_signal+0x94/0x1ee0 arch/x86/kernel/signal.c:808
exit_to_usermode_loop+0x224/0x300 arch/x86/entry/common.c:158
prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline]
syscall_return_slowpath+0x42f/0x500 arch/x86/entry/common.c:266
entry_SYSCALL_64_fastpath+0xbc/0xbe
RIP: 0033:0x7f33f723fdd3
RSP: 002b:00007fffb5303398 EFLAGS: 00000246 ORIG_RAX: 0000000000000017
RAX: fffffffffffffdfe RBX: 00007fffb5303540 RCX: 00007f33f723fdd3
RDX: 0000000000000000 RSI: 00007fffb53036f0 RDI: 000000000000000b
RBP: 00007fffb53036f0 R08: 00007fffb5303770 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00007fffb5303ad0 R14: 0000000000000000 R15: 0000000000000000


---
This bug is generated by a dumb bot. It may contain errors.
See https://goo.gl/tpsmEJ for details.
Direct all questions to syzk...@googlegroups.com.

syzbot will keep track of this bug report.
Once a fix for this bug is committed, please reply to this email with:
#syz fix: exact-commit-title
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug
report.

[Dmitry Vyukov]

On Mon, Oct 30, 2017 at 10:12 PM, syzbot
<bot+c9f0eb0d2a5576ece3...@syzkaller.appspotmail.com>
wrote:

> Hello,
>
> syzkaller hit the following crash on
> d95e159cd1da1ed4dbf76bf203e8ffaf231395e4
> git://git.cmpxchg.org/linux-mmots.git/master
> compiler: gcc (GCC) 7.1.1 20170620
> .config is attached
> Raw console output is attached.
> C reproducer is attached
> syzkaller reproducer is attached. See https://goo.gl/kgGztJ
> for information about syzkaller reproducers

This also happens on more recent commits, including linux-next
36ef71cae353f88fd6e095e2aaa3e5953af1685d (Oct 19) and upstream
3e0cc09a3a2c40ec1ffb6b4e12da86e98feccb11 (Oct 18).

> --
> You received this message because you are subscribed to the Google Groups
> "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to syzkaller-bug...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/syzkaller-bugs/94eb2c058c80ea49ed055cc8695e%40google.com.
> For more options, visit https://groups.google.com/d/optout.

[Oleg Nesterov]

On 10/30, Dmitry Vyukov wrote:
>
> On Mon, Oct 30, 2017 at 10:12 PM, syzbot
> <bot+c9f0eb0d2a5576ece3...@syzkaller.appspotmail.com>
> wrote:
> > Hello,
> >
> > syzkaller hit the following crash on
> > d95e159cd1da1ed4dbf76bf203e8ffaf231395e4
> > git://git.cmpxchg.org/linux-mmots.git/master
> > compiler: gcc (GCC) 7.1.1 20170620
> > .config is attached
> > Raw console output is attached.
> > C reproducer is attached

Hmm. I do not see reproducer in this email...

> > syzkaller reproducer is attached. See https://goo.gl/kgGztJ
> > for information about syzkaller reproducers
>
> This also happens on more recent commits, including linux-next
> 36ef71cae353f88fd6e095e2aaa3e5953af1685d (Oct 19) and upstream
> 3e0cc09a3a2c40ec1ffb6b4e12da86e98feccb11 (Oct 18).
>
> > WARNING: CPU: 0 PID: 1 at kernel/signal.c:340
> > task_participate_group_stop+0x1ce/0x230 kernel/signal.c:340
> > Kernel panic - not syncing: panic_on_warn set ...
> >
> > CPU: 0 PID: 1 Comm: init Not tainted 4.13.0-mm1+ #5

Looks familiar... I need some time to recall the details, will try to send
the fix(es) this week.

So this is init process with SIGNAL_UNKILLABLE flag set. And I hope it has
the pending SIGKILL, otherwise there is something else.

IIRC the problem is that complete_signal(SIGKILL) does nothing if
SIGNAL_UNKILLABLE is set, in particular it doesn't set SIGNAL_GROUP_EXIT.
This fools the signal_group_exit() check in do_signal_stop().

Actually there are more problems with SIGNAL_UNKILLABLE && SIGKILL, we need
some nasty cleanups.

Oleg.

[Dmitry Vyukov]

On Tue, Oct 31, 2017 at 7:34 PM, Oleg Nesterov <ol...@redhat.com> wrote:
> On 10/30, Dmitry Vyukov wrote:
>>
>> On Mon, Oct 30, 2017 at 10:12 PM, syzbot
>> <bot+c9f0eb0d2a5576ece3...@syzkaller.appspotmail.com>
>> wrote:
>> > Hello,
>> >
>> > syzkaller hit the following crash on
>> > d95e159cd1da1ed4dbf76bf203e8ffaf231395e4
>> > git://git.cmpxchg.org/linux-mmots.git/master
>> > compiler: gcc (GCC) 7.1.1 20170620
>> > .config is attached
>> > Raw console output is attached.
>> > C reproducer is attached
>
> Hmm. I do not see reproducer in this email...

Ah, sorry. You can see full thread with attachments here:
https://groups.google.com/forum/#!topic/syzkaller-bugs/EUmYZU4m5gU

[Oleg Nesterov]

On 11/01, Dmitry Vyukov wrote:
>
> On Tue, Oct 31, 2017 at 7:34 PM, Oleg Nesterov <ol...@redhat.com> wrote:
> > Hmm. I do not see reproducer in this email...
>
> Ah, sorry. You can see full thread with attachments here:
> https://groups.google.com/forum/#!topic/syzkaller-bugs/EUmYZU4m5gU

Heh. I can't say I enjoyed reading the reproducer ;)

> >> > WARNING: CPU: 0 PID: 1 at kernel/signal.c:340
> >> > task_participate_group_stop+0x1ce/0x230 kernel/signal.c:340
> >> > Kernel panic - not syncing: panic_on_warn set ...
> >> >
> >> > CPU: 0 PID: 1 Comm: init Not tainted 4.13.0-mm1+ #5
> >

> > So this is init process with SIGNAL_UNKILLABLE flag set. And I hope it has
> > the pending SIGKILL, otherwise there is something else.

From repro.c

line 111 r[8] = syscall(__NR_ptrace, 0x10ul, r[7]);

this is PTRACE_ATTACH

line 115 syscall(__NR_ptrace, 0x4200ul, r[7], 0x40000012ul, 0x100012ul);

this is PTRACE_SETOPTIONS and "data" includes PTRACE_O_EXITKILL.

r[7] is initialized at

line 110 r[7] = *(uint32_t*)0x20f9cffc;

so if it is eq to 1 then it can attach to init and in this case the problem
can be explained by the wrong SIGNAL_UNKILLABLE/SIGKILL logic.

But how *(uint32_t*)0x20f9cffc can be 1 ?

line 108 r[6] = syscall(__NR_fcntl, r[1], 0x10ul, 0x20f9cff8ul);

this is F_GETOWN_EX, addr = 0x20f9cff8 == 0x20f9cffc + 4, so if fcntl()
actually succeeds then r[7] == f_owner_ex->pid.

It _can_ be 1, but the reproducer doesn't work for me. If you can reproduce,
could you try the patch below?

Oleg.


diff --git a/kernel/signal.c b/kernel/signal.c
index 800a18f..7e15b56 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -78,7 +78,7 @@ static int sig_task_ignored(struct task_struct *t, int sig, bool force)
handler = sig_handler(t, sig);

if (unlikely(t->signal->flags & SIGNAL_UNKILLABLE) &&
- handler == SIG_DFL && !force)
+ handler == SIG_DFL && !(force && sig_kernel_only(sig)))
return 1;

return sig_handler_ignored(handler, sig);
@@ -94,13 +94,15 @@ static int sig_ignored(struct task_struct *t, int sig, bool force)
if (sigismember(&t->blocked, sig) || sigismember(&t->real_blocked, sig))
return 0;

- if (!sig_task_ignored(t, sig, force))
- return 0;
-
/*
- * Tracers may want to know about even ignored signals.
+ * Tracers may want to know about even ignored signal unless it
+ * is SIGKILL which can't be reported anyway but can be ignored
+ * by SIGNAL_UNKILLABLE task.
*/
- return !t->ptrace;
+ if (t->ptrace && sig != SIGKILL)
+ return 0;
+
+ return sig_task_ignored(t, sig, force);
}

/*
@@ -929,9 +931,9 @@ static void complete_signal(int sig, struct task_struct *p, int group)
* then start taking the whole group down immediately.
*/
if (sig_fatal(p, sig) &&
- !(signal->flags & (SIGNAL_UNKILLABLE | SIGNAL_GROUP_EXIT)) &&
+ !(signal->flags & SIGNAL_GROUP_EXIT) &&
!sigismember(&t->real_blocked, sig) &&
- (sig == SIGKILL || !t->ptrace)) {
+ (sig == SIGKILL || !p->ptrace)) {
/*
* This signal will be fatal to the whole group.
*/

[Dmitry Vyukov]

Hi,

I would like to understand why you were not able to reproduce it. I
won't be sitting here all the time, and we are tracking hundreds of
bugs across different linux kernels and other OSes, so it's
problematic to do any extensive work on all of them. That's why we try
to provide reproducers.

I've just tried the repro on the latest upstream
(39dae59d66acd86d1de24294bd2f343fd5e7a625) and it triggered the
WARNING within a second.
Did you use the config provided? Did you use qemu or real hardware?
Can you try in qemu (with -smp>1)?

[Jamie Iles]

Hi Dmitry,

I'm unable to reproduce the warning in qemu with SMP (on a 32 CPU VM).
Instead I get the following instant traceback which is different to what
you report when run as root:

[ 45.018469] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000013
[ 45.018469]
[ 45.019669] CPU: 19 PID: 1 Comm: systemd Not tainted 4.14.0-rc8 #7
[ 45.021094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.1-1ubuntu1 04/01/2014
[ 45.022768] Call Trace:
[ 45.023076] dump_stack+0x12e/0x188
[ 45.023481] panic+0x1e4/0x417
[ 45.023821] ? __warn+0x1d9/0x1d9
[ 45.024206] ? _raw_write_unlock_irq+0x27/0x70
[ 45.024705] do_exit+0x27ac/0x2f60
[ 45.025101] ? trace_hardirqs_on+0xd/0x10
[ 45.025551] ? _raw_spin_unlock_irq+0x27/0x70
[ 45.026034] ? mm_update_next_owner+0x640/0x640
[ 45.026540] ? get_signal+0x675/0x1520
[ 45.026971] ? recalc_sigpending+0x72/0x90
[ 45.027464] ? lock_downgrade+0x820/0x820
[ 45.027916] ? __dequeue_signal+0x640/0x640
[ 45.028388] ? _raw_spin_unlock_irq+0x27/0x70
[ 45.028877] do_group_exit+0x108/0x330
[ 45.029297] get_signal+0x61a/0x1520
[ 45.031144] do_signal+0x8d/0x1a10
[ 45.031531] ? trace_hardirqs_on_caller+0x442/0x5c0
[ 45.032105] ? trace_hardirqs_on+0xd/0x10
[ 45.032571] ? setup_sigcontext+0x7d0/0x7d0
[ 45.033071] ? ep_poll_readyevents_proc+0xa0/0xa0
[ 45.033619] ? rw_verify_area+0xe5/0x2b0
[ 45.034063] ? SyS_timerfd_settime+0xe5/0x140
[ 45.034551] ? exit_to_usermode_loop+0x45/0x230
[ 45.035065] exit_to_usermode_loop+0x16a/0x230
[ 45.035599] ? trace_hardirqs_on_caller+0x442/0x5c0
[ 45.036833] syscall_return_slowpath+0x310/0x3d0
[ 45.038547] entry_SYSCALL_64_fastpath+0xbc/0xbe
[ 45.039779] RIP: 0033:0x7fd80a914133
[ 45.040215] RSP: 002b:00007fff313d0858 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[ 45.041683] RAX: fffffffffffffffc RBX: 000055f47338c050 RCX: 00007fd80a914133
[ 45.042451] RDX: 000000000000003d RSI: 00007fff313d0860 RDI: 0000000000000004
[ 45.043307] RBP: 00007fff313d0c50 R08: 00007fff313d0860 R09: 8258efee6555c1f9
[ 45.044107] R10: 00000000ffffffff R11: 0000000000000246 R12: 00007fff313d0860
[ 45.045011] R13: ffffffffffffffff R14: 00007fff313d0c70 R15: 0000000000000001
[ 45.046217] Kernel Offset: disabled
[ 45.046615] Rebooting in 86400 seconds..

Running the same reproducer as an unprivileged user does not have any
effect - the system continues to run fine without any warning or panic.

Thanks,

Jamie

[Dmitry Vyukov]

Uh, it seems to be racy. I am getting either the WARNING or "attempt
to kill init" in ~1/5 proportion.
Please try this simplified program, it triggers the WARNING all the time for me:


// autogenerated by syzkaller (http://github.com/google/syzkaller)

#define _GNU_SOURCE
#include <sys/syscall.h>
#include <unistd.h>
#include <stdint.h>
#include <string.h>

int main()
{
long r[11];
memset(r, -1, sizeof(r));
r[0] = syscall(__NR_mmap, 0x20000000ul, 0xfec000ul, 0x3ul, 0x32ul,
0xfffffffffffffffful, 0x0ul);
r[1] = syscall(__NR_inotify_init1, 0x80000ul);
*(uint32_t*)0x20feb000 = (uint32_t)0xc;
r[3] = syscall(__NR_getsockopt, 0xfffffffffffffffful, 0x1ul, 0x11ul,
0x2003cff4ul, 0x20feb000ul);
if (r[3] != -1)
r[4] = *(uint32_t*)0x2003cff4;
r[5] = syscall(__NR_fcntl, r[1], 0x8ul, r[4]);

r[6] = syscall(__NR_fcntl, r[1], 0x10ul, 0x20f9cff8ul);

if (r[6] != -1)
r[7] = *(uint32_t*)0x20f9cffc;

r[8] = syscall(__NR_ptrace, 0x10ul, r[7]);

r[9] =
syscall(__NR_ioctl, 0xfffffffffffffffful, 0x4b6aul, 0x20f9e000ul);
r[10] =

syscall(__NR_ptrace, 0x4200ul, r[7], 0x40000012ul, 0x100012ul);

return 0;
}

[Jamie Iles]

I can't reproduce the warning with that reproducer, but I do see that it
runs and exits once and init is left in state T then the second run ends
up killing init and the VM crashes.

If I run once and then 'kill -CONT 1' then init goes back to sleeping
and won't accept a SIGSTOP so it looks like SIGNAL_UNKILLABLE isn't gone
though.

Jamie

[Oleg Nesterov]

On 11/06, Jamie Iles wrote:
>
> I'm unable to reproduce the warning in qemu with SMP (on a 32 CPU VM).

Neither me. Perhaps because I tried this test-case on the minimal system
with /bin/sh running as init process.

> Instead I get the following instant traceback which is different to what
> you report when run as root:
>
> [ 45.018469] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000013
> [ 45.018469]
> [ 45.019669] CPU: 19 PID: 1 Comm: systemd Not tainted 4.14.0-rc8 #7
> [ 45.021094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.1-1ubuntu1 04/01/2014
> [ 45.022768] Call Trace:
> [ 45.023076] dump_stack+0x12e/0x188
> [ 45.023481] panic+0x1e4/0x417

This is fine and hopefully confirms the theory. let me quote my previous email:

line 111 r[8] = syscall(__NR_ptrace, 0x10ul, r[7]);

this is PTRACE_ATTACH

line 115 syscall(__NR_ptrace, 0x4200ul, r[7], 0x40000012ul, 0x100012ul);

this is PTRACE_SETOPTIONS and "data" includes PTRACE_O_EXITKILL.

r[7] is initialized at

line 110 r[7] = *(uint32_t*)0x20f9cffc;

so if it is eq to 1 then it can attach to init and in this case the problem
can be explained by the wrong SIGNAL_UNKILLABLE/SIGKILL logic.

So, if it is eq to 1 then init will be killed after the child process created
by loop() function exits (see PTRACE_O_EXITKILL above).

This is correct, only the warning is not.

For example, this command does ptrace(PTRACE_SEIZE, 1,0, PTRACE_O_EXITKILL)

# perl -e 'syscall 101, 0x4206, 1, 0, 0x100000'

and crashes the kernel the same way, this is correct.

Oleg.

[Dmitry Vyukov]

Oleg, I've tested the patch and I don't see the WARNING with it. Only
attempt to kill init, which is fine, we test inside of pid namespace
and test process is not able to reach init.

Tested-by: Dmitry Vyukov <dvy...@google.com>

[Eric Biggers]

On Mon, Oct 30, 2017 at 10:12 PM, syzbot <bot+c9f0eb0d2a5576ece3...@syzkaller.appspotmail.com> wrote:

No longer occurring, apparently fixed by commit 426915796cc:

#syz fix: kernel/signal.c: remove the no longer needed SIGNAL_UNKILLABLE check in complete_signal()

- Eric