[syzbot] [cgroups?] [mm?] WARNING in __mod_memcg_lruvec_state

34 views
Skip to first unread message

syzbot

unread,
Apr 2, 2024, 4:03:28 AMApr 2
to ak...@linux-foundation.org, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, shakee...@linux.dev, syzkall...@googlegroups.com
Hello,

syzbot found the following issue on:

HEAD commit: 317c7bc0ef03 Merge tag 'mmc-v6.9-rc1' of git://git.kernel...
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15fd40c5180000
kernel config: https://syzkaller.appspot.com/x/.config?x=f64ec427e98bccd7
dashboard link: https://syzkaller.appspot.com/bug?extid=9319a4268a640e26b72b
compiler: gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7bc7510fe41f/non_bootable_disk-317c7bc0.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/efab473d72c0/vmlinux-317c7bc0.xz
kernel image: https://storage.googleapis.com/syzbot-assets/5ba3f56d362d/bzImage-317c7bc0.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+9319a4...@syzkaller.appspotmail.com

------------[ cut here ]------------
WARNING: CPU: 0 PID: 30105 at mm/memcontrol.c:865 __mod_memcg_lruvec_state+0x3fa/0x550 mm/memcontrol.c:865
Modules linked in:
CPU: 0 PID: 30105 Comm: syz-executor.2 Not tainted 6.9.0-rc1-syzkaller-00178-g317c7bc0ef03 #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
RIP: 0010:__mod_memcg_lruvec_state+0x3fa/0x550 mm/memcontrol.c:865
Code: 45 85 e4 75 1d 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc b8 00 04 00 00 e9 80 fd ff ff 89 c6 e9 a0 fd ff ff 90 <0f> 0b 90 e9 a7 fc ff ff 48 c7 c7 18 43 e1 8f e8 32 51 f8 ff e9 5e
RSP: 0018:ffffc900034beef8 EFLAGS: 00010202
RAX: 0000000000000292 RBX: 0000000000000001 RCX: 1ffffffff1fc2863
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff888024b92bc8
RBP: ffff888024b92000 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: ffff88801c326000 R14: 0000000000000001 R15: ffff888024b92000
FS: 00007f0811bf96c0(0000) GS:ffff88806b000000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000cfff1dd CR3: 000000003e4e2000 CR4: 0000000000350ef0
DR0: 0000000000000031 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
__update_lru_size include/linux/mm_inline.h:47 [inline]
lru_gen_update_size include/linux/mm_inline.h:199 [inline]
lru_gen_add_folio+0x62d/0xe80 include/linux/mm_inline.h:262
lruvec_add_folio include/linux/mm_inline.h:323 [inline]
lru_add_fn+0x3fc/0xd80 mm/swap.c:215
folio_batch_move_lru+0x243/0x400 mm/swap.c:233
lru_add_drain_cpu+0x534/0x860 mm/swap.c:667
lru_add_drain+0x109/0x440 mm/swap.c:767
__folio_batch_release+0x68/0xb0 mm/swap.c:1091
folio_batch_release include/linux/pagevec.h:101 [inline]
mpage_prepare_extent_to_map+0x112b/0x14b0 fs/ext4/inode.c:2505
ext4_do_writepages+0xa9e/0x3250 fs/ext4/inode.c:2632
ext4_writepages+0x303/0x730 fs/ext4/inode.c:2768
do_writepages+0x1a3/0x7f0 mm/page-writeback.c:2612
filemap_fdatawrite_wbc mm/filemap.c:397 [inline]
filemap_fdatawrite_wbc+0x148/0x1c0 mm/filemap.c:387
__filemap_fdatawrite_range+0xba/0x100 mm/filemap.c:430
file_write_and_wait_range+0xd0/0x140 mm/filemap.c:788
ext4_sync_file+0x296/0xf30 fs/ext4/fsync.c:158
vfs_fsync_range+0x141/0x230 fs/sync.c:188
generic_write_sync include/linux/fs.h:2793 [inline]
ext4_buffered_write_iter+0x2e6/0x3d0 fs/ext4/file.c:305
ext4_file_write_iter+0x874/0x1a40 fs/ext4/file.c:698
call_write_iter include/linux/fs.h:2108 [inline]
new_sync_write fs/read_write.c:497 [inline]
vfs_write+0x6db/0x1100 fs/read_write.c:590
ksys_write+0x12f/0x260 fs/read_write.c:643
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xd2/0x260 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x6d/0x75
RIP: 0033:0x7f0810e7dda9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 e1 20 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f0811bf90c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 00007f0810fabf80 RCX: 00007f0810e7dda9
RDX: 0000000000002000 RSI: 0000000020000180 RDI: 0000000000000007
RBP: 00007f0810eca47a R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 000000000000000b R14: 00007f0810fabf80 R15: 00007ffd57bcc8d8
</TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzk...@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Andrew Morton

unread,
Apr 3, 2024, 9:07:57 PMApr 3
to syzbot, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, shakee...@linux.dev, syzkall...@googlegroups.com
Well it beats me. I assume we failed to update for a new case. I'll
toss this into -next to perhaps shed a bit of light.

--- a/mm/memcontrol.c~__mod_memcg_lruvec_state-enhance-diagnostics
+++ a/mm/memcontrol.c
@@ -860,10 +860,12 @@ void __mod_memcg_lruvec_state(struct lru
case NR_ANON_THPS:
case NR_SHMEM_PMDMAPPED:
case NR_FILE_PMDMAPPED:
- WARN_ON_ONCE(!in_task());
+ if (WARN_ON_ONCE(!in_task()))
+ pr_warn("stat item index: %d\n", idx);
break;
default:
- VM_WARN_ON_IRQS_ENABLED();
+ if (VM_WARN_ON_IRQS_ENABLED())
+ pr_warn("stat item index: %d\n", idx);
}
}

_

Yosry Ahmed

unread,
Apr 4, 2024, 7:36:44 PMApr 4
to Andrew Morton, syzbot, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, shakee...@linux.dev, syzkall...@googlegroups.com, Yu Zhao
Line 865 from this commit should be this warning (i.e. warning because
IRQs are enabled). This also makes sense because __update_lru_size()
should not be updating any of the above stats.

folio_batch_move_lru() in the above call stack should be acquiring the
lock with IRQs disabled though, so I am not sure what's going on from
a quick look.

Adding Yu Zhao here.

Yu Zhao

unread,
Apr 4, 2024, 9:44:55 PMApr 4
to Yosry Ahmed, Andrew Morton, syzbot, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, shakee...@linux.dev, syzkall...@googlegroups.com
Probably an RT build where _irqsave doesn't disable IRQ?

Yosry Ahmed

unread,
Apr 5, 2024, 12:48:15 AMApr 5
to Yu Zhao, Andrew Morton, syzbot, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, shakee...@linux.dev, syzkall...@googlegroups.com
Yeah, that's possible. I don't have enough familiarity with RT to know
for sure. Perhaps lruvec->lru_lock needs to be changed to a
raw_spinlock_t? I am not sure if that's the correct course of action.

Shakeel Butt

unread,
Apr 14, 2024, 6:57:38 AMApr 14
to Yosry Ahmed, Yu Zhao, Andrew Morton, syzbot, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, syzkall...@googlegroups.com
On Thu, Apr 04, 2024 at 09:47:34PM -0700, Yosry Ahmed wrote:
[...]
> >
> > Probably an RT build where _irqsave doesn't disable IRQ?
>
> Yeah, that's possible. I don't have enough familiarity with RT to know
> for sure. Perhaps lruvec->lru_lock needs to be changed to a
> raw_spinlock_t? I am not sure if that's the correct course of action.

The kernel config does not have CONFIG_PREEMPT_RT enabled. However I
have not idea how irqs are getting enabled to raise this warning.

syzbot

unread,
May 6, 2024, 9:03:31 AM (11 days ago) May 6
to ak...@linux-foundation.org, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, shakee...@linux.dev, syzkall...@googlegroups.com, yosry...@google.com, yuz...@google.com
syzbot has found a reproducer for the following issue on:

HEAD commit: 2b84edefcad1 Add linux-next specific files for 20240506
git tree: linux-next
console+strace: https://syzkaller.appspot.com/x/log.txt?x=1164931f180000
kernel config: https://syzkaller.appspot.com/x/.config?x=b499929e4aaba1af
dashboard link: https://syzkaller.appspot.com/bug?extid=9319a4268a640e26b72b
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=123d5d1f180000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16527450980000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/6a22cf95ee14/disk-2b84edef.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/f5c45b515282/vmlinux-2b84edef.xz
kernel image: https://storage.googleapis.com/syzbot-assets/9bf98258a662/bzImage-2b84edef.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+9319a4...@syzkaller.appspotmail.com

------------[ cut here ]------------
__mod_memcg_lruvec_state: missing stat item 25
WARNING: CPU: 0 PID: 5091 at mm/memcontrol.c:999 __mod_memcg_lruvec_state+0x18c/0x430 mm/memcontrol.c:999
Modules linked in:
CPU: 0 PID: 5091 Comm: syz-executor374 Not tainted 6.9.0-rc7-next-20240506-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/27/2024
RIP: 0010:__mod_memcg_lruvec_state+0x18c/0x430 mm/memcontrol.c:999
Code: ad 35 94 0d 00 0f 85 a4 01 00 00 c6 05 a0 35 94 0d 01 90 48 c7 c7 80 53 d7 8b 48 c7 c6 54 27 b8 8d 44 89 fa e8 55 f1 54 ff 90 <0f> 0b 90 90 e9 7c 01 00 00 4c 8d 7c 24 60 4c 89 fb 48 c1 eb 03 42
RSP: 0018:ffffc9000338f1e0 EFLAGS: 00010246
RAX: 38a0635190e19e00 RBX: ffffffff8fad3cb9 RCX: ffff8880221f0000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffc9000338f2c0 R08: ffffffff815846c2 R09: fffffbfff1c3995c
R10: dffffc0000000000 R11: fffffbfff1c3995c R12: 0000000000000200
R13: dffffc0000000000 R14: 1ffff92000671e40 R15: 0000000000000019
FS: 00005555900be380(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000061cca0 CR3: 0000000075f1c000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
__mod_lruvec_state mm/memcontrol.c:1055 [inline]
__lruvec_stat_mod_folio+0x1a4/0x300 mm/memcontrol.c:1075
__folio_add_file_rmap mm/rmap.c:1444 [inline]
folio_add_file_rmap_pmd+0x29c/0x700 mm/rmap.c:1485
do_set_pmd+0x767/0xc40 mm/memory.c:4650
set_huge_pmd+0x28a/0x710 mm/khugepaged.c:1452
collapse_pte_mapped_thp+0x5c7/0x10b0 mm/khugepaged.c:1655
madvise_collapse+0x661/0xcc0 mm/khugepaged.c:2762
madvise_vma_behavior mm/madvise.c:1094 [inline]
madvise_walk_vmas mm/madvise.c:1268 [inline]
do_madvise+0xc5f/0x44d0 mm/madvise.c:1464
__do_sys_madvise mm/madvise.c:1481 [inline]
__se_sys_madvise mm/madvise.c:1479 [inline]
__x64_sys_madvise+0xa6/0xc0 mm/madvise.c:1479
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f1421be72a9
Code: 48 83 c4 28 c3 e8 37 17 00 00 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fffb7f38b78 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
RAX: ffffffffffffffda RBX: 00007fffb7f38d58 RCX: 00007f1421be72a9
RDX: 0000000000000019 RSI: 0000000000800018 RDI: 0000000020000000
RBP: 00007f1421c5a610 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
R13: 00007fffb7f38d48 R14: 0000000000000001 R15: 0000000000000001
</TASK>


---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

Yosry Ahmed

unread,
May 6, 2024, 10:12:04 AM (11 days ago) May 6
to syzbot, ak...@linux-foundation.org, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, shakee...@linux.dev, syzkall...@googlegroups.com, yuz...@google.com
On Mon, May 06, 2024 at 06:03:29AM -0700, syzbot wrote:
> syzbot has found a reproducer for the following issue on:
>
> HEAD commit: 2b84edefcad1 Add linux-next specific files for 20240506
> git tree: linux-next
> console+strace: https://syzkaller.appspot.com/x/log.txt?x=1164931f180000
> kernel config: https://syzkaller.appspot.com/x/.config?x=b499929e4aaba1af
> dashboard link: https://syzkaller.appspot.com/bug?extid=9319a4268a640e26b72b
> compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=123d5d1f180000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=16527450980000
>
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/6a22cf95ee14/disk-2b84edef.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/f5c45b515282/vmlinux-2b84edef.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/9bf98258a662/bzImage-2b84edef.xz
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+9319a4...@syzkaller.appspotmail.com
>
> ------------[ cut here ]------------
> __mod_memcg_lruvec_state: missing stat item 25
> WARNING: CPU: 0 PID: 5091 at mm/memcontrol.c:999 __mod_memcg_lruvec_state+0x18c/0x430 mm/memcontrol.c:999

This doesn't seem to be the same issue as the original one syzbot
reported. It's the same function but a different warning. I am not sure
how to tell syzbot that.

Anyway, this warning is the one newly introduced by Shakeel. It is
firing because NR_SHMEM_PMDMAPPED and/or NR_FILE_PMDMAPPED are being
updated using __lruvec_stat_mod_folio(), which also updates the memcg
stats. However, these stats are not exported per-memcg.

I think the following should fix it.

#syz test

diff --git a/mm/rmap.c b/mm/rmap.c
index 12be4241474ab..d3a26ea4dbae2 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1441,7 +1441,7 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio,

nr = __folio_add_rmap(folio, page, nr_pages, level, &nr_pmdmapped);
if (nr_pmdmapped)
- __lruvec_stat_mod_folio(folio, folio_test_swapbacked(folio) ?
+ __mod_node_page_state(folio, folio_test_swapbacked(folio) ?
NR_SHMEM_PMDMAPPED : NR_FILE_PMDMAPPED, nr_pmdmapped);
if (nr)
__lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr);

Shakeel Butt

unread,
May 6, 2024, 11:59:34 AM (11 days ago) May 6
to Yosry Ahmed, syzbot, ak...@linux-foundation.org, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, syzkall...@googlegroups.com, yuz...@google.com
Hey Yosry, can you please post your patch with the signoff? The patch
looks good and you can put my ack on it.

syzbot

unread,
May 6, 2024, 12:13:10 PM (11 days ago) May 6
to ak...@linux-foundation.org, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, shakee...@linux.dev, syzkall...@googlegroups.com, yosry...@google.com, yuz...@google.com
Hello,

syzbot tried to test the proposed patch but the build/boot failed:

mm/rmap.c:1444:25: error: incompatible pointer types passing 'struct folio *' to parameter of type 'struct pglist_data *' [-Werror,-Wincompatible-pointer-types]


Tested on:

commit: 2b84edef Add linux-next specific files for 20240506
git tree: linux-next
kernel config: https://syzkaller.appspot.com/x/.config?x=b499929e4aaba1af
dashboard link: https://syzkaller.appspot.com/bug?extid=9319a4268a640e26b72b
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
patch: https://syzkaller.appspot.com/x/patch.diff?x=1655a590980000

Yosry Ahmed

unread,
May 6, 2024, 12:13:32 PM (11 days ago) May 6
to Shakeel Butt, syzbot, ak...@linux-foundation.org, cgr...@vger.kernel.org, han...@cmpxchg.org, linux-...@vger.kernel.org, linu...@kvack.org, mho...@kernel.org, muchu...@linux.dev, roman.g...@linux.dev, syzkall...@googlegroups.com, yuz...@google.com
Will do. I noticed we also update these stats from
__folio_remove_rmap(). I will send a patch fixing these as well, so
I'll not include your Ack for that as it will be different.
Reply all
Reply to author
Forward
0 new messages