[PATCH v2 0/6] kasan: add workqueue and timer stack for generic KASAN

[Walter Wu]

Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
In some of these access/allocation happened in process_one_work(),
we see the free stack is useless in KASAN report, it doesn't help
programmers to solve UAF on workqueue. The same may stand for times.

This patchset improves KASAN reports by making them to have workqueue
queueing stack and timer queueing stack information. It is useful for
programmers to solve use-after-free or double-free memory issue.

Generic KASAN will record the last two workqueue and timer stacks,
print them in KASAN report. It is only suitable for generic KASAN.

[1]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22+process_one_work
[2]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22%20expire_timers
[3]https://bugzilla.kernel.org/show_bug.cgi?id=198437

Walter Wu (6):
timer: kasan: record timer stack
workqueue: kasan: record workqueue stack
kasan: print timer and workqueue stack
lib/test_kasan.c: add timer test case
lib/test_kasan.c: add workqueue test case
kasan: update documentation for generic kasan

---

Changes since v1:
- Thanks for Marco and Thomas suggestion.
- Remove unnecessary code and fix commit log
- reuse kasan_record_aux_stack() and aux_stack
to record timer and workqueue stack.
- change the aux stack title for common name.

---

Documentation/dev-tools/kasan.rst | 4 ++--
kernel/time/timer.c | 3 +++
kernel/workqueue.c | 3 +++
lib/test_kasan.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
mm/kasan/report.c | 4 ++--
5 files changed, 64 insertions(+), 4 deletions(-)

[Marco Elver]

Much cleaner.

In general,

Acked-by: Marco Elver <el...@google.com>

but I left some more comments. I'm a bit worried about the tests,
because of KASAN-test KUnit rework, but probably not much we can do
until these are added to -mm tree.

Thanks,
-- Marco

[Walter Wu]

Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
In some of these access/allocation happened in process_one_work(),
we see the free stack is useless in KASAN report, it doesn't help
programmers to solve UAF on workqueue. The same may stand for times.

This patchset improves KASAN reports by making them to have workqueue

queueing stack and timer stack information. It is useful for programmers

to solve use-after-free or double-free memory issue.

Generic KASAN also records the last two workqueue and timer stacks and
prints them in KASAN report. It is only suitable for generic KASAN.

[1]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22+process_one_work
[2]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22%20expire_timers
[3]https://bugzilla.kernel.org/show_bug.cgi?id=198437

Walter Wu (6):
timer: kasan: record timer stack
workqueue: kasan: record workqueue stack
kasan: print timer and workqueue stack
lib/test_kasan.c: add timer test case
lib/test_kasan.c: add workqueue test case
kasan: update documentation for generic kasan

---

Changes since v2:
- modify kasan document to be more readable.
Thanks for Marco suggestion.

Changes since v1:
- Thanks for Marco and Thomas suggestion.
- Remove unnecessary code and fix commit log
- reuse kasan_record_aux_stack() and aux_stack
to record timer and workqueue stack.
- change the aux stack title for common name.

[Walter Wu]

Hi Marco,

Thanks for your review and friendly reminder.

If needed, I will rebase and resend the test case.


Walter

> Thanks,
> -- Marco

[Marco Elver]

On Tue, 25 Aug 2020 at 03:57, Walter Wu <walter...@mediatek.com> wrote:
>
> Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
> In some of these access/allocation happened in process_one_work(),
> we see the free stack is useless in KASAN report, it doesn't help
> programmers to solve UAF on workqueue. The same may stand for times.
>
> This patchset improves KASAN reports by making them to have workqueue
> queueing stack and timer stack information. It is useful for programmers
> to solve use-after-free or double-free memory issue.
>
> Generic KASAN also records the last two workqueue and timer stacks and
> prints them in KASAN report. It is only suitable for generic KASAN.
>
> [1]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22+process_one_work
> [2]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22%20expire_timers
> [3]https://bugzilla.kernel.org/show_bug.cgi?id=198437
>
> Walter Wu (6):
> timer: kasan: record timer stack
> workqueue: kasan: record workqueue stack
> kasan: print timer and workqueue stack
> lib/test_kasan.c: add timer test case
> lib/test_kasan.c: add workqueue test case
> kasan: update documentation for generic kasan

Acked-by: Marco Elver <el...@google.com>

> ---
>
> Changes since v2:
> - modify kasan document to be more readable.
> Thanks for Marco suggestion.
>
> Changes since v1:
> - Thanks for Marco and Thomas suggestion.
> - Remove unnecessary code and fix commit log
> - reuse kasan_record_aux_stack() and aux_stack
> to record timer and workqueue stack.
> - change the aux stack title for common name.
>
> ---
>
> Documentation/dev-tools/kasan.rst | 4 ++--
> kernel/time/timer.c | 3 +++
> kernel/workqueue.c | 3 +++
> lib/test_kasan.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> mm/kasan/report.c | 4 ++--
> 5 files changed, 64 insertions(+), 4 deletions(-)
>

> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20200825015654.27781-1-walter-zh.wu%40mediatek.com.

[Andrey Konovalov]

On Tue, Aug 25, 2020 at 10:26 AM 'Marco Elver' via kasan-dev
<kasa...@googlegroups.com> wrote:
>
> On Tue, 25 Aug 2020 at 03:57, Walter Wu <walter...@mediatek.com> wrote:
> >
> > Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
> > In some of these access/allocation happened in process_one_work(),
> > we see the free stack is useless in KASAN report, it doesn't help
> > programmers to solve UAF on workqueue. The same may stand for times.
> >
> > This patchset improves KASAN reports by making them to have workqueue
> > queueing stack and timer stack information. It is useful for programmers
> > to solve use-after-free or double-free memory issue.
> >
> > Generic KASAN also records the last two workqueue and timer stacks and
> > prints them in KASAN report. It is only suitable for generic KASAN.
> >
> > [1]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22+process_one_work
> > [2]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22%20expire_timers
> > [3]https://bugzilla.kernel.org/show_bug.cgi?id=198437
> >
> > Walter Wu (6):
> > timer: kasan: record timer stack
> > workqueue: kasan: record workqueue stack
> > kasan: print timer and workqueue stack
> > lib/test_kasan.c: add timer test case
> > lib/test_kasan.c: add workqueue test case
> > kasan: update documentation for generic kasan
>
> Acked-by: Marco Elver <el...@google.com>

Reviewed-by: Andrey Konovalov <andre...@google.com>

>
>
>
> > ---
> >
> > Changes since v2:
> > - modify kasan document to be more readable.
> > Thanks for Marco suggestion.
> >
> > Changes since v1:
> > - Thanks for Marco and Thomas suggestion.
> > - Remove unnecessary code and fix commit log
> > - reuse kasan_record_aux_stack() and aux_stack
> > to record timer and workqueue stack.
> > - change the aux stack title for common name.
> >
> > ---
> >
> > Documentation/dev-tools/kasan.rst | 4 ++--
> > kernel/time/timer.c | 3 +++
> > kernel/workqueue.c | 3 +++
> > lib/test_kasan.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > mm/kasan/report.c | 4 ++--
> > 5 files changed, 64 insertions(+), 4 deletions(-)
> >
> > --
> > You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/20200825015654.27781-1-walter-zh.wu%40mediatek.com.
>
> --
> You received this message because you are subscribed to the Google Groups "kasan-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/CANpmjNOvj%2B%3Dv7VDVDXpsUNZ9o0%2BKoJVJs0MjLhwr0XpYcYQZ5g%40mail.gmail.com.

[Dmitry Vyukov]

On Wed, Aug 26, 2020 at 2:30 PM 'Andrey Konovalov' via kasan-dev

Reviewed-by: Dmitry Vyukov <dvy...@google.com>

Thanks! The UAF reports with RCU stacks that I see now are just 🔥🔥🔥

> To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/CAAeHK%2ByVShDPCxVKDsO_5SwoM2ZG7x7byUJ74PtB7ekY61L2YQ%40mail.gmail.com.

[Andrey Konovalov]

Hi Walter,

This patchset needs to be rebased onto the KASAN-KUNIT patches, which just recently went into the mm tree.

Thanks!

[Walter Wu]

Hi Dmitry, Andrey,

Got it.

Thanks for your review and reminder.

Walter

[Walter Wu]

Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
In some of these access/allocation happened in process_one_work(),
we see the free stack is useless in KASAN report, it doesn't help
programmers to solve UAF on workqueue. The same may stand for times.

This patchset improves KASAN reports by making them to have workqueue
queueing stack and timer stack information. It is useful for programmers
to solve use-after-free or double-free memory issue.

Generic KASAN also records the last two workqueue and timer stacks and
prints them in KASAN report. It is only suitable for generic KASAN.

[1]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22+process_one_work
[2]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22%20expire_timers
[3]https://bugzilla.kernel.org/show_bug.cgi?id=198437

Walter Wu (6):
timer: kasan: record timer stack
workqueue: kasan: record workqueue stack
kasan: print timer and workqueue stack
lib/test_kasan.c: add timer test case
lib/test_kasan.c: add workqueue test case
kasan: update documentation for generic kasan

---
Changes since v3:
- testcases have merge conflict, so that need to
be rebased onto the KASAN-KUNIT.

Changes since v2:
- modify kasan document to be readable,

Thanks for Marco suggestion.

Changes since v1:
- Thanks for Marco and Thomas suggestion.
- Remove unnecessary code and fix commit log
- reuse kasan_record_aux_stack() and aux_stack
to record timer and workqueue stack.
- change the aux stack title for common name.

---

Documentation/dev-tools/kasan.rst | 5 +++--

kernel/time/timer.c | 3 +++
kernel/workqueue.c | 3 +++

lib/test_kasan_module.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
mm/kasan/report.c | 4 ++--
5 files changed, 66 insertions(+), 4 deletions(-)

[Alexander Potapenko]

> ---
> Documentation/dev-tools/kasan.rst | 5 +++--
> kernel/time/timer.c | 3 +++
> kernel/workqueue.c | 3 +++
> lib/test_kasan_module.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
> mm/kasan/report.c | 4 ++--
> 5 files changed, 66 insertions(+), 4 deletions(-)

While at it, can you remove a mention of call_rcu() from the
kasan_record_aux_stack() implementation, as it is no more
RCU-specific?

[Walter Wu]

On Thu, 2020-09-24 at 13:51 +0200, 'Alexander Potapenko' via kasan-dev
wrote:

Thank you for your reminder. v5 will do it.

Walter

[Thomas Gleixner]

On Thu, Sep 24 2020 at 12:01, Walter Wu wrote:
> Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
> In some of these access/allocation happened in process_one_work(),
> we see the free stack is useless in KASAN report, it doesn't help
> programmers to solve UAF on workqueue. The same may stand for times.
>
> This patchset improves KASAN reports by making them to have workqueue
> queueing stack and timer stack information. It is useful for programmers
> to solve use-after-free or double-free memory issue.
>
> Generic KASAN also records the last two workqueue and timer stacks and
> prints them in KASAN report. It is only suitable for generic KASAN.
>
> [1]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22+process_one_work
> [2]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22%20expire_timers

How are these links useful for people who do not have a gurgle account?

[Dmitry Vyukov]

On Wed, Sep 30, 2020 at 5:29 PM Thomas Gleixner <tg...@linutronix.de> wrote:
>
> On Thu, Sep 24 2020 at 12:01, Walter Wu wrote:
> > Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
> > In some of these access/allocation happened in process_one_work(),
> > we see the free stack is useless in KASAN report, it doesn't help
> > programmers to solve UAF on workqueue. The same may stand for times.
> >
> > This patchset improves KASAN reports by making them to have workqueue
> > queueing stack and timer stack information. It is useful for programmers
> > to solve use-after-free or double-free memory issue.
> >
> > Generic KASAN also records the last two workqueue and timer stacks and
> > prints them in KASAN report. It is only suitable for generic KASAN.

Walter, did you mail v5?
Checking statuses of KASAN issues and this seems to be not in linux-next.

> > [1]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22+process_one_work
> > [2]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22%20expire_timers
>
> How are these links useful for people who do not have a gurgle account?

This is a public mailing list archive, so effectively the same way as
lore links ;)

[Walter Wu]

Hi Dmitry,

On Tue, 2020-12-01 at 08:59 +0100, 'Dmitry Vyukov' via kasan-dev wrote:
> On Wed, Sep 30, 2020 at 5:29 PM Thomas Gleixner <tg...@linutronix.de> wrote:
> >
> > On Thu, Sep 24 2020 at 12:01, Walter Wu wrote:
> > > Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
> > > In some of these access/allocation happened in process_one_work(),
> > > we see the free stack is useless in KASAN report, it doesn't help
> > > programmers to solve UAF on workqueue. The same may stand for times.
> > >
> > > This patchset improves KASAN reports by making them to have workqueue
> > > queueing stack and timer stack information. It is useful for programmers
> > > to solve use-after-free or double-free memory issue.
> > >
> > > Generic KASAN also records the last two workqueue and timer stacks and
> > > prints them in KASAN report. It is only suitable for generic KASAN.
>
> Walter, did you mail v5?
> Checking statuses of KASAN issues and this seems to be not in linux-next.
>

Sorry for the delay in responding to this patch. I'm busy these few
months, so that suspend processing it.
Yes, I will send it next week. But v4 need to confirm the timer stack is
useful. I haven't found an example. Do you have some suggestion about
timer?

Thanks.
Walter

[Dmitry Vyukov]

On Tue, Dec 1, 2020 at 12:17 PM Walter Wu <walter...@mediatek.com> wrote:
>
> Hi Dmitry,
>
> On Tue, 2020-12-01 at 08:59 +0100, 'Dmitry Vyukov' via kasan-dev wrote:
> > On Wed, Sep 30, 2020 at 5:29 PM Thomas Gleixner <tg...@linutronix.de> wrote:
> > >
> > > On Thu, Sep 24 2020 at 12:01, Walter Wu wrote:
> > > > Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
> > > > In some of these access/allocation happened in process_one_work(),
> > > > we see the free stack is useless in KASAN report, it doesn't help
> > > > programmers to solve UAF on workqueue. The same may stand for times.
> > > >
> > > > This patchset improves KASAN reports by making them to have workqueue
> > > > queueing stack and timer stack information. It is useful for programmers
> > > > to solve use-after-free or double-free memory issue.
> > > >
> > > > Generic KASAN also records the last two workqueue and timer stacks and
> > > > prints them in KASAN report. It is only suitable for generic KASAN.
> >
> > Walter, did you mail v5?
> > Checking statuses of KASAN issues and this seems to be not in linux-next.
> >
>
> Sorry for the delay in responding to this patch. I'm busy these few
> months, so that suspend processing it.
> Yes, I will send it next week. But v4 need to confirm the timer stack is
> useful. I haven't found an example. Do you have some suggestion about
> timer?

Good question.

We had some use-after-free's what mention call_timer_fn:
https://groups.google.com/g/syzkaller-bugs/search?q=%22kasan%22%20%22use-after-free%22%20%22expire_timers%22%20%22call_timer_fn%22%20
In the reports I checked call_timer_fn appears in the "access" stack
rather in the "free" stack.

Looking at these reports I cannot conclude that do_init_timer stack
would be useful.
I am mildly leaning towards not memorizing do_init_timer stack for now
(until we have clear use cases) as the number of aux stacks is very
limited (2).

[Thomas Gleixner]

Just that it asked me to log in last time. That's why I wrote the
above. Today it does not, odd.

Thanks,

tglx

[Dmitry Vyukov]

On Tue, Dec 1, 2020 at 3:13 PM Thomas Gleixner <tg...@linutronix.de> wrote:
> >> > Syzbot reports many UAF issues for workqueue or timer, see [1] and [2].
> >> > In some of these access/allocation happened in process_one_work(),
> >> > we see the free stack is useless in KASAN report, it doesn't help
> >> > programmers to solve UAF on workqueue. The same may stand for times.
> >> >
> >> > This patchset improves KASAN reports by making them to have workqueue
> >> > queueing stack and timer stack information. It is useful for programmers
> >> > to solve use-after-free or double-free memory issue.
> >> >
> >> > Generic KASAN also records the last two workqueue and timer stacks and
> >> > prints them in KASAN report. It is only suitable for generic KASAN.
> >
> > Walter, did you mail v5?
> > Checking statuses of KASAN issues and this seems to be not in linux-next.
> >
> >> > [1]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22+process_one_work
> >> > [2]https://groups.google.com/g/syzkaller-bugs/search?q=%22use-after-free%22%20expire_timers
> >>
> >> How are these links useful for people who do not have a gurgle account?
> >
> > This is a public mailing list archive, so effectively the same way as
> > lore links ;)
>
> Just that it asked me to log in last time. That's why I wrote the
> above. Today it does not, odd.

Some random permissions settings changes were observed before, so I
can believe that.

[Walter Wu]

Yes, call stack already is useful for it in KASAN report.

> Looking at these reports I cannot conclude that do_init_timer stack
> would be useful.
> I am mildly leaning towards not memorizing do_init_timer stack for now
> (until we have clear use cases) as the number of aux stacks is very
> limited (2).
>

Got it. I will remove timer patch and send v5.
Thanks for your suggestion.

Walter