KASAN: use-after-free Read in inet_create

[syzbot]

Hello,

syzkaller hit the following crash on
82bcf1def3b5f1251177ad47c44f7e17af039b4b
git://git.cmpxchg.org/linux-mmots.git/master
compiler: gcc (GCC) 7.1.1 20170620
.config is attached
Raw console output is attached.

Unfortunately, I don't have any reproducer for this bug yet.


==================================================================
BUG: KASAN: use-after-free in inet_create+0xda0/0xf50 net/ipv4/af_inet.c:338
Read of size 4 at addr ffff8801bde28554 by task kworker/u4:5/3492

CPU: 0 PID: 3492 Comm: kworker/u4:5 Not tainted 4.15.0-rc2-mm1+ #39
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Workqueue: krdsd rds_connect_worker
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x194/0x257 lib/dump_stack.c:53
print_address_description+0x73/0x250 mm/kasan/report.c:252
kasan_report_error mm/kasan/report.c:351 [inline]
kasan_report+0x25b/0x340 mm/kasan/report.c:409
__asan_report_load4_noabort+0x14/0x20 mm/kasan/report.c:429
inet_create+0xda0/0xf50 net/ipv4/af_inet.c:338
__sock_create+0x4d4/0x850 net/socket.c:1265
sock_create_kern+0x3f/0x50 net/socket.c:1311
rds_tcp_conn_path_connect+0x26f/0x920 net/rds/tcp_connect.c:108
rds_connect_worker+0x156/0x1f0 net/rds/threads.c:165
process_one_work+0xbfd/0x1bc0 kernel/workqueue.c:2113
worker_thread+0x223/0x1990 kernel/workqueue.c:2247
kthread+0x37a/0x440 kernel/kthread.c:238
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:524

Allocated by task 3362:
save_stack+0x43/0xd0 mm/kasan/kasan.c:447
set_track mm/kasan/kasan.c:459 [inline]
kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:551
kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:489
kmem_cache_alloc+0x12e/0x760 mm/slab.c:3548
kmem_cache_zalloc include/linux/slab.h:695 [inline]
net_alloc net/core/net_namespace.c:362 [inline]
copy_net_ns+0x196/0x580 net/core/net_namespace.c:402
create_new_namespaces+0x425/0x880 kernel/nsproxy.c:107
unshare_nsproxy_namespaces+0xae/0x1e0 kernel/nsproxy.c:206
SYSC_unshare kernel/fork.c:2421 [inline]
SyS_unshare+0x653/0xfa0 kernel/fork.c:2371
entry_SYSCALL_64_fastpath+0x1f/0x96

Freed by task 35:
save_stack+0x43/0xd0 mm/kasan/kasan.c:447
set_track mm/kasan/kasan.c:459 [inline]
kasan_slab_free+0x71/0xc0 mm/kasan/kasan.c:524
__cache_free mm/slab.c:3492 [inline]
kmem_cache_free+0x77/0x280 mm/slab.c:3750
net_free+0xca/0x110 net/core/net_namespace.c:378
net_drop_ns.part.11+0x26/0x30 net/core/net_namespace.c:385
net_drop_ns net/core/net_namespace.c:384 [inline]
cleanup_net+0x895/0xb60 net/core/net_namespace.c:502
process_one_work+0xbfd/0x1bc0 kernel/workqueue.c:2113
worker_thread+0x223/0x1990 kernel/workqueue.c:2247
kthread+0x37a/0x440 kernel/kthread.c:238
ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:524

The buggy address belongs to the object at ffff8801bde28080
which belongs to the cache net_namespace of size 6272
The buggy address is located 1236 bytes inside of
6272-byte region [ffff8801bde28080, ffff8801bde29900)
The buggy address belongs to the page:
page:00000000df6a4dc0 count:1 mapcount:0 mapping:00000000553659f1 index:0x0
compound_mapcount: 0
flags: 0x2fffc0000008100(slab|head)
raw: 02fffc0000008100 ffff8801bde28080 0000000000000000 0000000100000001
raw: ffffea0006f75da0 ffffea0006f60220 ffff8801d989fe00 0000000000000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
ffff8801bde28400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8801bde28480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ffff8801bde28500: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff8801bde28580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8801bde28600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================


---
This bug is generated by a dumb bot. It may contain errors.
See https://goo.gl/tpsmEJ for details.
Direct all questions to syzk...@googlegroups.com.
Please credit me with: Reported-by: syzbot <syzk...@googlegroups.com>

syzbot will keep track of this bug report.
Once a fix for this bug is merged into any tree, reply to this email with:
#syz fix: exact-commit-title
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug
report.
Note: all commands must start from beginning of the line in the email body.

[Eric Biggers]

[+RDS list and maintainer]

This is still happening regularly, though syzbot hasn't been able to generate a
reproducer yet. All the reports seem to involve rds_connect_worker()
encountering a freed network namespace (struct net) when calling
sock_create_kern() from rds_tcp_conn_path_connect(). Probably something in RDS
needs to be taking a reference to the network namespace and isn't, or the RDS
workqueue isn't being shut down correctly. You can see all reports of this on
the syzbot dashboard at
https://syzkaller.appspot.com/bug?id=1f45ae538a0453220337ccb84962249fdd67107f.
Last one was April 5 on Linus' tree (commit 3e968c9f1401088).

- Eric

[Sowmini Varadhan]

#syz dup: KASAN: use-after-free Read in rds_cong_queue_updates

There are a number of manifestations of this bug, basically
all suggest that the connect/reconnect etc workqs are somehow
being scheduled after the netns is deleted, despite the
code refactoring in Commit 3db6e0d172c (and looks like
the WARN_ONs in that commit are not even being triggered).
We've not been able to reproduce this issues, and without
a crash dump (or some hint of other threads that were running
at the time of the problem) are working on figuring out
the root-cause by code-inspection.

--Sowmini

[Sowmini Varadhan]

A request for the syszbot team: is it possible to get a list of
all threads when the panic happens (either via sysrq or through a crash
dump or some other means) since this may give a hint about what's
triggering the race

An additional comment to Eric's note in https://lkml.org/lkml/2018/4/8/240:
RDS cannot take net refcounts because this leads to a circular
dependancy: cleanup_net() is only called when the net refcount goes
to zero, and rds depends on callbacks from cleanup_net to clean up
its state (which led to Commit 681648e67d4, and the subsequent changes
around rds_destroy_pending).

--Sowmini

[Dmitry Vyukov]

On Tue, Apr 10, 2018 at 4:35 PM, Sowmini Varadhan
<sowmini....@oracle.com> wrote:
> On (04/08/18 21:04), Sowmini Varadhan wrote:
>>
>> #syz dup: KASAN: use-after-free Read in rds_cong_queue_updates
>>
>> There are a number of manifestations of this bug, basically
>> all suggest that the connect/reconnect etc workqs are somehow
>> being scheduled after the netns is deleted, despite the
>> code refactoring in Commit 3db6e0d172c (and looks like
>> the WARN_ONs in that commit are not even being triggered).
>> We've not been able to reproduce this issues, and without
>> a crash dump (or some hint of other threads that were running
>> at the time of the problem) are working on figuring out
>> the root-cause by code-inspection.
>
> A request for the syszbot team: is it possible to get a list of
> all threads when the panic happens (either via sysrq or through a crash
> dump or some other means) since this may give a hint about what's
> triggering the race

Hi Sowmini,

sysrq has several problems.
We currently do panic_on_warn, and in that state kernel will probably
not handle sysrq's.
In some cases, kernel will do panic unconditionally, so we don't have
an option. But even if it doesn't (besides the fact that we will
potentially get wall with thousands of reports), execution of this and
other CPUs has already left the point of crash.
It's also not completely clear when to inject it. If we inject it
right when we see "BUG:" on console, then the chances that we will get
intermixed crash report and sysrq output, in which case we won't even
be able to understand what the crash is and will have to throw it all
into trash bin.
Finally, I don't know how to send sysrq over GCE console connection.

It would be nice if kernel would print more useful info right at the
point of panic (maybe under an optional config or something), but
Peter was against this:
https://lkml.org/lkml/2018/4/7/84

kdump's also popped up several times in this context, but this is a
non-trivial amount of work, see
https://github.com/google/syzkaller/issues/491 for context.

We will do kdump collection and post-processing at some point. But I
don't have anything good to propose as an immidiate solution.

[Sowmini Varadhan]

On (04/11/18 15:09), Dmitry Vyukov wrote:
> sysrq has several problems.
:

> kdump's also popped up several times in this context, but this is a
> non-trivial amount of work, see

:

> We will do kdump collection and post-processing at some point. But I
> don't have anything good to propose as an immidiate solution.

I see, this is certainly not trivial to automate, esp at the scale
at which syzbot operates.

Having the backtrace from kdump would certainly help, hope we get
to this point soon. Having the dump itself is an additional
nice-to-have (if we can get to that)

Meanwhile, let me continue to stare at the code to see what
race-window we missed.

--Sowmini

[Tetsuo Handa]

Dmitry Vyukov wrote:
> > A request for the syszbot team: is it possible to get a list of
> > all threads when the panic happens (either via sysrq or through a crash
> > dump or some other means) since this may give a hint about what's
> > triggering the race
>

> It would be nice if kernel would print more useful info right at the
> point of panic (maybe under an optional config or something), but
> Peter was against this:
> https://lkml.org/lkml/2018/4/7/84
>
> kdump's also popped up several times in this context, but this is a
> non-trivial amount of work, see
> https://github.com/google/syzkaller/issues/491 for context.
>
> We will do kdump collection and post-processing at some point. But I
> don't have anything good to propose as an immidiate solution.
>

I browsed demo_setup.sh and thought that we could use a simple wrapper approach
provided that questions below are solved.

Question 1: When does the syzbot kill the qemu process?

If the kernel panic does not occur, when does the syzbot kill the qemu process?
For example, no panic within 10 minutes from the start of the qemu process?

If the kernel panic occurred, when does the syzbot kill the qemu process?
The kernel shows "Rebooting in 86400 seconds..", but the syzbot does not wait
for 86400 seconds, does it? How does the syzbot find that "since the kernel
panic occurred, I can kill the qemu process now" ?

Question 2: How and where should the result of vmcore analysis be saved?

The syzbot is providing "log" which contains both kernel messages and console
messages. But how is the "log" file generated? Is it possible to append the
result of vmcore analysis into the same "log" file?

A PoC implementation of the simple wrapper approach is attached. You are
worrying about disk space, but what I have in mind is to delete vmcore file
after the analysis. Getting SysRq-t from vmcore will be a nice starting point.
Tested with HEAD of https://github.com/crash-utility/crash#master and
http://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux#master using
command line shown below.

$ wrapper vmlinux /usr/libexec/qemu-kvm -kernel arch/x86/boot/bzImage --append console=ttyS0 -nographic

[Dmitry Vyukov]

On Thu, Apr 12, 2018 at 1:44 PM, Tetsuo Handa
<penguin...@i-love.sakura.ne.jp> wrote:
> Dmitry Vyukov wrote:
>> > A request for the syszbot team: is it possible to get a list of
>> > all threads when the panic happens (either via sysrq or through a crash
>> > dump or some other means) since this may give a hint about what's
>> > triggering the race
>>
>> It would be nice if kernel would print more useful info right at the
>> point of panic (maybe under an optional config or something), but
>> Peter was against this:
>> https://lkml.org/lkml/2018/4/7/84
>>
>> kdump's also popped up several times in this context, but this is a
>> non-trivial amount of work, see
>> https://github.com/google/syzkaller/issues/491 for context.
>>
>> We will do kdump collection and post-processing at some point. But I
>> don't have anything good to propose as an immidiate solution.
>>
>
> I browsed demo_setup.sh and thought that we could use a simple wrapper approach
> provided that questions below are solved.
>
> Question 1: When does the syzbot kill the qemu process?

First thing is that syzbot uses GCE VMs as test machines rather than
qemu (it can also test on android phones, some arm boards, etc, but
we, of course, don't need to support all of them).

GCE VMs don't have dump-guest-memory thing.

> If the kernel panic does not occur, when does the syzbot kill the qemu process?
> For example, no panic within 10 minutes from the start of the qemu process?
>
> If the kernel panic occurred, when does the syzbot kill the qemu process?
> The kernel shows "Rebooting in 86400 seconds..", but the syzbot does not wait
> for 86400 seconds, does it? How does the syzbot find that "since the kernel
> panic occurred, I can kill the qemu process now" ?

Yes, it looks for several things:
- something like "BUG:" appears in console output
- ssh to the machine exits
- both console and ssh output do not show any signs of life for 3 minutes
- qemu process, or GCE console connection, adb (depending on test
machine type) exits
Then waits for 10 seconds to gather all output (say, body of kernel
crash) and then kills qemu process, or deletes GCE VM.

> Question 2: How and where should the result of vmcore analysis be saved?
>
> The syzbot is providing "log" which contains both kernel messages and console
> messages. But how is the "log" file generated? Is it possible to append the
> result of vmcore analysis into the same "log" file?

This is somewhat involved. There are multiple machines that run
syzkaller, and they are generally dispensable, any won't be able to
hold all dumps (there are lots of thousands of bugs), and the web
server/mailed do not have access to disks of these machines. So
eventually dump should be uploaded to cloud storage, but for the
purposes of this discussion we can consider this as just running scp.

I think it will be better to not mix dump analysis and log. We can
decide in what exactly form this should be presented to end user, but
they should be stored separately. This will give much more freedom
regarding changes to actual presentation, garbage collection of old
dumps, etc.

[Tetsuo Handa]

Dmitry Vyukov wrote:
> > I browsed demo_setup.sh and thought that we could use a simple wrapper approach
> > provided that questions below are solved.
> >
> > Question 1: When does the syzbot kill the qemu process?
>
> First thing is that syzbot uses GCE VMs as test machines rather than
> qemu (it can also test on android phones, some arm boards, etc, but
> we, of course, don't need to support all of them).
>
> GCE VMs don't have dump-guest-memory thing.

Ouch! I thought syzbot runs tests inside qemu. ;-)

Then, exporting only tests which found bugs under GCE VMs in order to
try to capture vmcore by reproducing under qemu might be useful...

[Dmitry Vyukov]

Yes, but then it's a whole new subsystem, some queueing mechanism,
deployment and maintenance. There is also a question of how images
will be shared, or otherwise the qemu subsystem will need to
constantly rebuild images on specified git tree/commit/config/compiler
for each test. And also dumps will probably be most useful exactly for
bugs that can't be reliably reproduced (otherwise one just reproduce
it locally and then obtain any required info).
I think it's better long term to invest into kdump-based dump
collection from GCE VMs. Since it does not depend on VMM features it
can also be transparently extended to qemu, android phones, etc.

[Tetsuo Handa]

But kdump on GCE VMs is not available right now. Until kdump becomes available,
I want to use SysRq-t just before the system halts. There are many hung up or
stall reports but it is difficult to understand what is happening.

I updated wrapper.c to use notification from pvpanic module (CONFIG_PVPANIC=y).
Then, I noticed that we might be able to utilize panic notifier as a trigger
for obtaining SysRq-t. The code is shown below.

----------------------------------------
#include <linux/sched/debug.h>
#include <linux/notifier.h>
#include <linux/init.h>

static int sysrq_on_panic_notify(struct notifier_block *nb, unsigned long code,
void *unused)
{
show_state();
show_workqueue_state();
return NOTIFY_DONE;
}

static struct notifier_block sysrq_on_panic_nb = {
.notifier_call = sysrq_on_panic_notify,
};

static int __init sysrq_on_panic_init(void)
{
atomic_notifier_chain_register(&panic_notifier_list,
&sysrq_on_panic_nb);
return 0;
}
late_initcall(sysrq_on_panic_init);
----------------------------------------

Maybe this code (with enable/disable switch added) is suitable for
drivers/tty/sysrq.c for environments where kdump is not available.

[Dmitry Vyukov]

On Sat, Apr 14, 2018 at 6:40 PM, Tetsuo Handa

Interesting.
If we have something like this, this may be the simplest way to obtain
additional info. syzkaller already captures console output and we have
panic_on_warn=1 and we definitely can enable CONFIG_PVPANIC, or any
other config.
We probably also want cpu backtraces. This probably should be
configurable to some degree as to what types of info are dumped.

[Tetsuo Handa]

Something like this?

>From 82f156636ab2ed4d1042d765a27aa23c109d5197 Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <penguin...@I-love.SAKURA.ne.jp>
Date: Tue, 17 Apr 2018 22:50:30 +0900
Subject: [PATCH] sysrq: Allow obtaining SysRq upon kernel panic event.

While syzbot is finding many hungup bugs and stall bugs, currently we
can't capture vmcore on the platform which syzbot is running tests.
This situation makes us difficult to understand what is happening.

For now, allowing syzbot to obtain SysRq-t and SysRq-l just before
the kernel halts would be helpful. And this will remain true even after
it became possible to capture vmcore on the platform which syzbot uses.

Therefore, this patch utilizes panic notifier callback in order to allow
administrators to obtain SysRq under environments where it is difficult
to configure kdump.

Signed-off-by: Tetsuo Handa <penguin...@I-love.SAKURA.ne.jp>
---
drivers/tty/sysrq.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 49 insertions(+)

diff --git a/drivers/tty/sysrq.c b/drivers/tty/sysrq.c
index 6364890..bc099f2 100644
--- a/drivers/tty/sysrq.c
+++ b/drivers/tty/sysrq.c
@@ -1128,6 +1128,54 @@ static inline void sysrq_init_procfs(void)

#endif /* CONFIG_PROC_FS */

+/*
+ * Allow administrators to obtain SysRq upon panic() for environments where
+ * kdump is not available. Types to obtain is configured as bitmask of the
+ * following values. Can be set at sysrq.dump_on_panic= kernel command line
+ * and get/set via /sys/module/sysrq/parameters/dump_on_panic .
+ *
+ * 1 == "Show State" (SysRq-t).
+ * 2 == "Show Blocked State" (SysRq-w). Implied by "Show State".
+ * 4 == "Show Locks Held" (SysRq-d). Implied by "Show State".
+ * 8 == "Show workqueue state". Implied by "Show State".
+ * 16 == "Show backtrace of all active CPUs" (SysRq-l).
+ */
+static unsigned int sysrq_on_panic;
+
+static int sysrq_on_panic_notify(struct notifier_block *nb, unsigned long code,
+ void *unused)
+{
+ if (sysrq_on_panic & 1) {
+ sysrq_handle_showstate(0);
+ } else {
+ if (sysrq_on_panic & 2)
+ sysrq_handle_showstate_blocked(0);
+ if (IS_ENABLED(CONFIG_LOCKDEP) && (sysrq_on_panic & 4))
+ debug_show_all_locks();
+ if (sysrq_on_panic & 8)
+ show_workqueue_state();
+ }
+ /* No fall back, for we can't wait when we are already in panic(). */
+ if (IS_ENABLED(CONFIG_SMP) && (sysrq_on_panic & 16))
+ trigger_all_cpu_backtrace();
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block sysrq_panic_nb = {
+ .notifier_call = sysrq_on_panic_notify,
+ /*
+ * Call me after panic notifiers for watchdogs are told that there is
+ * no need to warn again because we are already in panic() state.
+ */
+ .priority = -1,
+};
+
+static inline void sysrq_register_panic_handler(void)
+{
+ atomic_notifier_chain_register(&panic_notifier_list, &sysrq_panic_nb);
+}
+module_param_named(dump_on_panic, sysrq_on_panic, uint, 0644);
+
static int __init sysrq_init(void)
{
sysrq_init_procfs();
@@ -1135,6 +1183,7 @@ static int __init sysrq_init(void)
if (sysrq_on())
sysrq_register_handler();

+ sysrq_register_panic_handler();
return 0;
}
device_initcall(sysrq_init);
--
1.8.3.1