[syzbot] kernel BUG in dnotify_free_mark

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit: 247f34f7b803 Linux 6.1-rc2
git tree: upstream
console+strace: https://syzkaller.appspot.com/x/log.txt?x=157f594a880000
kernel config: https://syzkaller.appspot.com/x/.config?x=1d3548a4365ba17d
dashboard link: https://syzkaller.appspot.com/bug?extid=06cc05ddc896f12b7ec5
compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15585936880000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14ec85ba880000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/a5f39164dea4/disk-247f34f7.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/8d1b92f5a01f/vmlinux-247f34f7.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/1a4d2943796c/mount_0.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+06cc05...@syzkaller.appspotmail.com

------------[ cut here ]------------
kernel BUG at fs/notify/dnotify/dnotify.c:136!
invalid opcode: 0000 [#1] PREEMPT SMP KASAN
CPU: 0 PID: 56 Comm: kworker/u4:4 Not tainted 6.1.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/11/2022
Workqueue: events_unbound fsnotify_mark_destroy_workfn
RIP: 0010:dnotify_free_mark+0x53/0x60 fs/notify/dnotify/dnotify.c:136
Code: 48 89 df e8 ff b3 dd ff 48 83 3b 00 75 17 e8 e4 bc 89 ff 48 8b 3d 4d ce 0f 0c 4c 89 f6 5b 41 5e e9 a2 de dc ff e8 cd bc 89 ff <0f> 0b cc cc cc cc cc cc cc cc cc cc cc 55 41 57 41 56 41 55 41 54
RSP: 0018:ffffc90001577b68 EFLAGS: 00010293
RAX: ffffffff81fe1253 RBX: ffff888075d2b080 RCX: ffff888018d40000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff888075d2b000
RBP: ffffc90001577c30 R08: dffffc0000000000 R09: fffffbfff2325fe4
R10: fffffbfff2325fe4 R11: 1ffffffff2325fe3 R12: ffff888145e77800
R13: ffffc90001577bc0 R14: ffff888075d2b000 R15: ffff888075d2b000
FS: 0000000000000000(0000) GS:ffff8880b9a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f127cdcaa38 CR3: 000000001dd46000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
fsnotify_final_mark_destroy fs/notify/mark.c:278 [inline]
fsnotify_mark_destroy_workfn+0x2cc/0x340 fs/notify/mark.c:902
process_one_work+0x877/0xdb0 kernel/workqueue.c:2289
worker_thread+0xb14/0x1330 kernel/workqueue.c:2436
kthread+0x266/0x300 kernel/kthread.c:376
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
RIP: 0010:dnotify_free_mark+0x53/0x60 fs/notify/dnotify/dnotify.c:136
Code: 48 89 df e8 ff b3 dd ff 48 83 3b 00 75 17 e8 e4 bc 89 ff 48 8b 3d 4d ce 0f 0c 4c 89 f6 5b 41 5e e9 a2 de dc ff e8 cd bc 89 ff <0f> 0b cc cc cc cc cc cc cc cc cc cc cc 55 41 57 41 56 41 55 41 54
RSP: 0018:ffffc90001577b68 EFLAGS: 00010293
RAX: ffffffff81fe1253 RBX: ffff888075d2b080 RCX: ffff888018d40000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff888075d2b000
RBP: ffffc90001577c30 R08: dffffc0000000000 R09


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

[Hillf Danton]

On 28 Oct 2022 16:45:33 -0700

> syzbot found the following issue on:
>
> HEAD commit: 247f34f7b803 Linux 6.1-rc2
> git tree: upstream
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=157f594a880000
> kernel config: https://syzkaller.appspot.com/x/.config?x=1d3548a4365ba17d
> dashboard link: https://syzkaller.appspot.com/bug?extid=06cc05ddc896f12b7ec5

> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15585936880000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14ec85ba880000

Add debug info.

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 247f34f7b803

diff -pur a/fs/notify/dnotify/dnotify.c b/fs/notify/dnotify/dnotify.c
--- a/fs/notify/dnotify/dnotify.c 2022-10-29 11:33:20.738540400 +0800
+++ b/fs/notify/dnotify/dnotify.c 2022-10-29 13:44:44.968617800 +0800
@@ -138,9 +138,21 @@ static void dnotify_free_mark(struct fsn
kmem_cache_free(dnotify_mark_cache, dn_mark);
}

+static int dnotify_check_mark(struct fsnotify_mark *fsn_mark)
+{
+ struct dnotify_mark *dn_mark = container_of(fsn_mark,
+ struct dnotify_mark,
+ fsn_mark);
+ if (dn_mark->dn)
+ return 1;
+ else
+ return 0;
+}
+
static const struct fsnotify_ops dnotify_fsnotify_ops = {
.handle_inode_event = dnotify_handle_event,
.free_mark = dnotify_free_mark,
+ .check_mark = dnotify_check_mark,
};

/*
diff -pur a/fs/notify/mark.c b/fs/notify/mark.c
--- a/fs/notify/mark.c 2022-10-29 11:32:48.394489200 +0800
+++ b/fs/notify/mark.c 2022-10-29 13:39:52.986357800 +0800
@@ -331,6 +331,9 @@ void fsnotify_put_mark(struct fsnotify_m
spin_unlock(&destroy_lock);
queue_work(system_unbound_wq, &connector_reaper_work);
}
+
+ if (mark->group->ops->check_mark)
+ WARN_ON_ONCE(mark->group->ops->check_mark(mark));
/*
* Note that we didn't update flags telling whether inode cares about
* what's happening with children. We update these flags from
diff -pur a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
--- a/include/linux/fsnotify_backend.h 2022-10-29 13:35:58.637169500 +0800
+++ b/include/linux/fsnotify_backend.h 2022-10-29 13:37:12.518647400 +0800
@@ -165,6 +165,7 @@ struct fsnotify_ops {
void (*free_event)(struct fsnotify_group *group, struct fsnotify_event *event);
/* called on final put+free to free memory */
void (*free_mark)(struct fsnotify_mark *mark);
+ int (*check_mark)(struct fsnotify_mark *mark);
};

/*
--

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
WARNING in fsnotify_put_mark

------------[ cut here ]------------
WARNING: CPU: 0 PID: 4068 at fs/notify/mark.c:336 fsnotify_put_mark+0x8b6/0x9c0
Modules linked in:
CPU: 0 PID: 4068 Comm: syz-executor.0 Not tainted 6.1.0-rc2-syzkaller-dirty #0

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/11/2022

RIP: 0010:fsnotify_put_mark+0x8b6/0x9c0 fs/notify/mark.c:336
Code: 0c b9 01 00 00 00 bf 08 00 00 00 48 c7 c2 e0 9b c4 8c 48 83 c4 40 5b 41 5c 41 5d 41 5e 41 5f 5d e9 2f c7 58 ff e8 fa 34 8a ff <0f> 0b e9 02 ff ff ff e8 ee 34 8a ff 0f 0b e9 24 f9 ff ff e8 e2 34
RSP: 0018:ffffc90004d1fb48 EFLAGS: 00010293
RAX: ffffffff81fd9a26 RBX: 0000000000000001 RCX: ffff888020cd9d40
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffff88806eda6000 R08: ffffffff81fd991b R09: fffffbfff1c1b5e6
R10: fffffbfff1c1b5e6 R11: 1ffffffff1c1b5e5 R12: dffffc0000000000
R13: 0000000000000000 R14: ffff88807efa7900 R15: 1ffff1100ddb4c0e
FS: 0000555556a00400(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 000056399dba2950 CR3: 000000006d41b000 CR4: 00000000003506e0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>

fsnotify_destroy_marks+0x57d/0x6f0 fs/notify/mark.c:868
fsnotify_clear_marks_by_inode fs/notify/fsnotify.h:60 [inline]
__fsnotify_inode_delete fs/notify/fsnotify.c:22 [inline]
fsnotify_inode_delete include/linux/fsnotify.h:176 [inline]
fsnotify_unmount_inodes fs/notify/fsnotify.c:78 [inline]
fsnotify_sb_delete+0x287/0x4e0 fs/notify/fsnotify.c:92
generic_shutdown_super+0x9c/0x310 fs/super.c:481
kill_block_super+0x79/0xd0 fs/super.c:1427
deactivate_locked_super+0xa7/0xf0 fs/super.c:331
cleanup_mnt+0x494/0x520 fs/namespace.c:1186
task_work_run+0x243/0x300 kernel/task_work.c:179
resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
exit_to_user_mode_loop+0x124/0x150 kernel/entry/common.c:171
exit_to_user_mode_prepare+0xb2/0x140 kernel/entry/common.c:203
__syscall_exit_to_user_mode_work kernel/entry/common.c:285 [inline]
syscall_exit_to_user_mode+0x26/0x60 kernel/entry/common.c:296
do_syscall_64+0x49/0xb0 arch/x86/entry/common.c:86
entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f7892a8ca67
Code: ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffe3a5680c8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f7892a8ca67
RDX: 00007ffe3a568199 RSI: 000000000000000a RDI: 00007ffe3a568190
RBP: 00007ffe3a568190 R08: 00000000ffffffff R09: 00007ffe3a567f60
R10: 0000555556a018b3 R11: 0000000000000246 R12: 00007f7892ae5826
R13: 00007ffe3a569250 R14: 0000555556a01810 R15: 00007ffe3a569290
</TASK>


Tested on:

commit: 247f34f7 Linux 6.1-rc2
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=1479cab6880000

kernel config: https://syzkaller.appspot.com/x/.config?x=1d3548a4365ba17d
dashboard link: https://syzkaller.appspot.com/bug?extid=06cc05ddc896f12b7ec5

compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2

patch: https://syzkaller.appspot.com/x/patch.diff?x=11cb6f5e880000

[Hillf Danton]

On 28 Oct 2022 16:45:33 -0700

> syzbot found the following issue on:
>
> HEAD commit: 247f34f7b803 Linux 6.1-rc2
> git tree: upstream
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=157f594a880000
> kernel config: https://syzkaller.appspot.com/x/.config?x=1d3548a4365ba17d
> dashboard link: https://syzkaller.appspot.com/bug?extid=06cc05ddc896f12b7ec5

> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15585936880000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14ec85ba880000

Add debug info.

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 247f34f7b803

--- a/fs/notify/dnotify/dnotify.c
+++ b/fs/notify/dnotify/dnotify.c

@@ -138,9 +138,21 @@ static void dnotify_free_mark(struct fsn
kmem_cache_free(dnotify_mark_cache, dn_mark);
}

+static void dnotify_freeing_mark(struct fsnotify_mark *fsn_mark, struct fsnotify_group *group)

+{
+ struct dnotify_mark *dn_mark = container_of(fsn_mark,
+ struct dnotify_mark,
+ fsn_mark);
+ if (dn_mark->dn) {

+ kmem_cache_free(dnotify_struct_cache, dn_mark->dn);
+ dn_mark->dn = NULL;
+ }

+}
+
static const struct fsnotify_ops dnotify_fsnotify_ops = {
.handle_inode_event = dnotify_handle_event,
.free_mark = dnotify_free_mark,

+ .freeing_mark = dnotify_freeing_mark,
};

/*
--

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+06cc05...@syzkaller.appspotmail.com

Tested on:

commit: 247f34f7 Linux 6.1-rc2
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

console output: https://syzkaller.appspot.com/x/log.txt?x=1228bb86880000

kernel config: https://syzkaller.appspot.com/x/.config?x=1d3548a4365ba17d
dashboard link: https://syzkaller.appspot.com/bug?extid=06cc05ddc896f12b7ec5

compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2

patch: https://syzkaller.appspot.com/x/patch.diff?x=137b6716880000

Note: testing is done by a robot and is best-effort only.

[Jan Kara]

Hello!

[added some CCs to gather more ideas]

On Fri 28-10-22 16:45:33, syzbot wrote:
> syzbot found the following issue on:
>
> HEAD commit: 247f34f7b803 Linux 6.1-rc2
> git tree: upstream
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=157f594a880000
> kernel config: https://syzkaller.appspot.com/x/.config?x=1d3548a4365ba17d
> dashboard link: https://syzkaller.appspot.com/bug?extid=06cc05ddc896f12b7ec5
> compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15585936880000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14ec85ba880000
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/a5f39164dea4/disk-247f34f7.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/8d1b92f5a01f/vmlinux-247f34f7.xz
> mounted in repro: https://storage.googleapis.com/syzbot-assets/1a4d2943796c/mount_0.gz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+06cc05...@syzkaller.appspotmail.com
>
> ------------[ cut here ]------------
> kernel BUG at fs/notify/dnotify/dnotify.c:136!

OK, I've tracked this down to the problem in ntfs3 driver or maybe more
exactly in bad inode handling. What the reproducer does is that it mounts
ntfs3 image, places dnotify mark on filesystem's /, then accesses something
which finds that / is corrupted. This calls ntfs_bad_inode() which calls
make_bad_inode() which sets inode->i_mode to S_IFREG. So when the file
descriptor is closed, dnotify doesn't get properly shutdown because it
works only on directories. Now calling make_bad_inode() on live inode is
problematic because it can change inode type (e.g. from directory to
regular file) and that tends to confuse things - dnotify in this case.

Now it is easy to blame filesystem driver for calling make_bad_inode() on
live inode but given it seems to be relatively widespread maybe
make_bad_inode() should be more careful not to screw VFS? What do other
people think?

Honza
--
Jan Kara <ja...@suse.com>
SUSE Labs, CR

[Amir Goldstein]

Do you know why make_bad_inode() sets inode->i_mode to S_IFREG?
If it did not do that, would it solve the dnotify issue?

Thanks,
Amir.

[Jan Kara]

I suppose because i_mode can be set to some bogus value (e.g. when
make_bad_inode() is called while reading the inode from the disk). One idea
I had was that we'd do this setting only if i_mode was indeed invalid. But
note that make_bad_inode() also sets inode->i_op and inode->i_fop and that
can also cause some surprises for a live inode (e.g. if some concurrent
process is in the middle of some operation on the inode).

> If it did not do that, would it solve the dnotify issue?

Yes, if i_mode was kept untouched, dnotify problem would be fixed.