[Linux Kernel Bug] memory leak in ubi_attach

[Chenyuan Yang]

Dear Linux Kernel Developers for UBI,

We encountered "memory leak in ubi_attach" when testing UBI with
Syzkaller and our generated specifications.

syz repro: https://drive.google.com/file/d/17FoGw6akfufz05U-oRBP2wXmOiFF1VUq/view?usp=drive_link
C reproducer: https://drive.google.com/file/d/1ayd3lmHPvqNoI01pQEdU832EktpTUnZ_/view?usp=drive_link
report: https://drive.google.com/file/d/1hC2arY3FbQt-6L5rbDfY-DQ2oH82IIGq/view?usp=drive_link
stats: https://drive.google.com/file/d/1REig9fV0H1fYPWaiicc-JVLlCpo7TTw4/view?usp=drive_link

This memory leak is triggered by `ioctl$UBI_IOCATT`, where
`ubi_attach_info` invokes `kmem_cache_create`
(https://elixir.bootlin.com/linux/v6.7/source/drivers/mtd/ubi/attach.c#L1464).
It seems that the memory leak occurs when the slab cache is
successfully created. I apologize for not being able to conduct a
deeper analysis of the root cause, as my expertise in UBI drivers is
limited.

If you have any questions or require more information, please feel
free to contact us.

Reported-by: Chenyuan Yang <cheny...@gmail.com>

Best,
Chenyuan

[Zhihao Cheng]

在 2024/1/23 11:53, Chenyuan Yang 写道:
> Dear Linux Kernel Developers for UBI,
>
> We encountered "memory leak in ubi_attach" when testing UBI with
> Syzkaller and our generated specifications.
>
> syz repro: https://drive.google.com/file/d/17FoGw6akfufz05U-oRBP2wXmOiFF1VUq/view?usp=drive_link
> C reproducer: https://drive.google.com/file/d/1ayd3lmHPvqNoI01pQEdU832EktpTUnZ_/view?usp=drive_link
> report: https://drive.google.com/file/d/1hC2arY3FbQt-6L5rbDfY-DQ2oH82IIGq/view?usp=drive_link
> stats: https://drive.google.com/file/d/1REig9fV0H1fYPWaiicc-JVLlCpo7TTw4/view?usp=drive_link

I can't open above links in company, may you post these files in attachment?

>
> This memory leak is triggered by `ioctl$UBI_IOCATT`, where
> `ubi_attach_info` invokes `kmem_cache_create`
> (https://elixir.bootlin.com/linux/v6.7/source/drivers/mtd/ubi/attach.c#L1464).
> It seems that the memory leak occurs when the slab cache is
> successfully created. I apologize for not being able to conduct a
> deeper analysis of the root cause, as my expertise in UBI drivers is
> limited.
>
> If you have any questions or require more information, please feel
> free to contact us.
>
> Reported-by: Chenyuan Yang <cheny...@gmail.com>
>
> Best,
> Chenyuan
>
>

> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
>

[Zhihao Cheng]

在 2024/1/23 12:57, Chenyuan Yang 写道:
> Hi Zhihao,
>
> Thanks for your prompt reply! Here are the files related to this memory leak.
>
> Best,
> Chenyuan
>

Can you provide a kernel config and the cmdline? I can't reproduce the
problem by both C program and syz program. But after analyzing the log:

[<ffffffff8157274e>] __kmalloc_node_track_caller+0x4e/0x120
mm/slab_common.c:1027
[<ffffffff81561e1f>] kstrdup+0x3f/0x70 mm/util.c:62
[<ffffffff81561ea7>] kstrdup_const+0x57/0x80 mm/util.c:85
[<ffffffff824d4596>] kvasprintf_const+0xc6/0x110 lib/kasprintf.c:48
[<ffffffff84aaeadf>] kobject_set_name_vargs+0x3f/0xe0 lib/kobject.c:272
[<ffffffff84aaef41>] kobject_add_varg lib/kobject.c:366 [inline]
[<ffffffff84aaef41>] kobject_init_and_add+0x71/0xe0 lib/kobject.c:455
[<ffffffff8162085a>] sysfs_slab_add+0x10a/0x210 mm/slub.c:6168
[<ffffffff81628474>] __kmem_cache_create+0x1b4/0x5a0 mm/slub.c:5120
[<ffffffff81571dca>] create_cache mm/slab_common.c:235 [inline]
[<ffffffff81571dca>] kmem_cache_create_usercopy+0x16a/0x2e0
mm/slab_common.c:340
[<ffffffff81571f51>] kmem_cache_create+0x11/0x20 mm/slab_common.c:395
[<ffffffff82e40fc5>] alloc_ai drivers/mtd/ubi/attach.c:1464 [inline]
[<ffffffff82e40fc5>] ubi_attach+0xb5/0x1e00
drivers/mtd/ubi/attach.c:1560
[<ffffffff82e302f8>] ubi_attach_mtd_dev+0x878/0x1130
drivers/mtd/ubi/build.c:1000
[<ffffffff82e3176d>] ctrl_cdev_ioctl+0x1dd/0x250
drivers/mtd/ubi/cdev.c:1043
[<ffffffff816b0e23>] vfs_ioctl fs/ioctl.c:51 [inline]
[<ffffffff816b0e23>] __do_sys_ioctl fs/ioctl.c:871 [inline]

The leaked memory is located in 'kobj->name', the kobj comes from
kmem_cache and it is used to create entry under
sysfs('/sys/kernel/slab'). If kmem_cache_destroy() is missed to be
called in someone exiting path in UBI, there will be more memleak
messages reported (eg. ai->aeb_slab_cache). So I guess the potentional
problem has nothing to do with UBI.
After looking through the implementation of create_cache, the life time
of 'kobj->name' is along with kobj, it can be always released even in
error handling paths. To figure out what happens I need to reproduce it
on my local machine.

[Chenyuan Yang]

Hi Zhihao,

Thanks very much for your reply and further exploration of this crash!

I have attached the config to reproduce this memory leak. For the
command line, you can use qemu to boot the kernel (You need to build
the bullseye-image first).
```
mkdir -p logs
qemu-system-x86_64 \
-m 2G \
-smp 2 \
-kernel linux/arch/x86/boot/bzImage \
-append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
-drive file=image/bullseye-qemu.img,format=raw \
-net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
-net nic,model=e1000 \
-enable-kvm \
-nographic \
-pidfile vm.pid \
```

In qemu, you can run
```
gcc -pthread repro.c -o exe
./exe
```
to reproduce the memory leak.

Feel free to contact me if you have any questions.

Best,
Chenyuan

[Zhihao Cheng]

在 2024/1/24 22:41, Chenyuan Yang 写道:
> Hi Zhihao,
>

> Thanks very much for your reply and further exploration of this crash!
>
> I have attached the config to reproduce this memory leak. For the
> command line, you can use qemu to boot the kernel (You need to build
> the bullseye-image first).
> ```
> mkdir -p logs
> qemu-system-x86_64 \
> -m 2G \
> -smp 2 \
> -kernel linux/arch/x86/boot/bzImage \
> -append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
> -drive file=image/bullseye-qemu.img,format=raw \
> -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
> -net nic,model=e1000 \
> -enable-kvm \
> -nographic \
> -pidfile vm.pid \
> ```
>
> In qemu, you can run
> ```
> gcc -pthread repro.c -o exe
> ./exe
> ```
> to reproduce the memory leak.
>

The bad news is that I can't reproduce the problem with your kernel
config. The good news is that I seem to find the cause of the problem by
code reviewing:
alloc_ai -> kmem_cache_create -> kmem_cache_create_usercopy:
s = __kmem_cache_alias
s = find_mergeable // cannot find mergeable kmem_cache, s = NULL
create_cache
s = kmem_cache_zalloc
__kmem_cache_create(s)
sysfs_slab_add
kobject_init_and_add(&s->kobj)
kobject_init(kobj) // kobj->kref = 1
kobject_add_varg
kobject_set_name_vargs
kvasprintf_const
kstrdup_const
kstrdup
kmalloc_track_caller // allocated memory
kobj->name = s // assign allocated memory to kobj->name
kobject_add_internal
create_dir // failed caused by ENOMEM
// kobj->name won't be released even 's' is released

You can reproduce it by executing ubiattach -m0 with reproduce.diff applied:
[root@localhost ~]# cat /sys/kernel/debug/kmemleak
unreferenced object 0xffff88813b488080 (size 16):
comm "ubiattach", pid 9535, jiffies 4294943095
hex dump (first 16 bytes):
3a 30 30 30 30 30 35 36 00 00 00 00 00 00 00 00 :0000056........
backtrace (crc 8a117ab9):
[<ffffffff815f91c2>] __kmalloc_node_track_caller+0x322/0x420
[<ffffffff81575efc>] kstrdup+0x3c/0x70
[<ffffffff81575f8f>] kstrdup_const+0x5f/0x70
[<ffffffff82535426>] kvasprintf_const+0xc6/0x110
[<ffffffff84b88cb0>] kobject_set_name_vargs+0x40/0xd0
[<ffffffff84b8911b>] kobject_init_and_add+0x8b/0xf0
[<ffffffff815faba9>] sysfs_slab_add+0x119/0x1e0
[<ffffffff815fc2dc>] __kmem_cache_create+0x41c/0x590
[<ffffffff81586019>] kmem_cache_create_usercopy+0x169/0x300
[<ffffffff815861c1>] kmem_cache_create+0x11/0x20
[<ffffffff82ec8896>] ubi_attach+0xc6/0x1de0
[<ffffffff82eb77a3>] ubi_attach_mtd_dev+0x8a3/0x1120
[<ffffffff82eb8bfc>] ctrl_cdev_ioctl+0x1fc/0x270
[<ffffffff816ca312>] __x64_sys_ioctl+0xf2/0x140
[<ffffffff84bc3970>] do_syscall_64+0x50/0x140
[<ffffffff84c0008b>] entry_SYSCALL_64_after_hwframe+0x63/0x6b

There are two ways to fix it:
1. Release kobj->name directly in the error baranch of
kobject_add_internal() after calling create_dir().
2. Put kobj's refcnt in the error baranch of kobject_add_internal()
after calling create_dir().
I'm not sure which one is better or whether there are other fix methods,
maybe you can send an email in kobj related mailiing lists to confirm
the problem.

[Chenyuan Yang]

Dear Linux Kernel Developers for UBI and Kernel Object,

I am writing to inquire if there have been any updates regarding this
memory leak issue. This issue remains reproducible on the latest
Linux version (6.12-rc2, commit
8cf0b93919e13d1e8d4466eb4080a4c4d9d66d7b).

Thank you for your attention to this matter.

Best,
Chenyuan

> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/0171b6cc-95ee-3538-913b-65a391a446b3%40huawei.com.

[Greg KH]

On Thu, Oct 10, 2024 at 08:23:22PM -0500, Chenyuan Yang wrote:
> Dear Linux Kernel Developers for UBI and Kernel Object,
>
> I am writing to inquire if there have been any updates regarding this
> memory leak issue. This issue remains reproducible on the latest
> Linux version (6.12-rc2, commit
> 8cf0b93919e13d1e8d4466eb4080a4c4d9d66d7b).
>
> Thank you for your attention to this matter.

Can you create a patch for this as you seem to have a good reproducer
for the problem.

thanks,

greg k-h

[Ryder Wang]

By walking through all the related code, it looks to be a bug in slub.c rather than kobject or ubifs.

sysfs_slab_add() calls kobject_init_and_add():
- If kobject_init_and_add fails, sysfs_slab_add() will go to *out*. But unluckily, *out* code block will never release s->kobj, but it is expected to do so.

Below is the function comment of kobject_init_and_add():
* If this function returns an error, kobject_put() must be called to properly clean up the memory associated with the object. *
==> It means sysfs_slab_add() shall release the related kobject.

________________________________________
From: linux-mtd <linux-mt...@lists.infradead.org> on behalf of Chenyuan Yang <cheny...@gmail.com>
Sent: Friday, October 11, 2024 09:23
To: linu...@lists.infradead.org
Cc: ric...@nod.at; miquel...@bootlin.com; Zhihao Cheng; vign...@ti.com; linux-...@vger.kernel.org; syzk...@googlegroups.com; Zijie Zhao; gre...@linuxfoundation.org; raf...@kernel.org; ak...@linux-foundation.org
Subject: Re: [Linux Kernel Bug] memory leak in ubi_attach

Best,
Chenyuan

______________________________________________________

[gre...@linuxfoundation.org]

On Tue, Oct 15, 2024 at 03:41:24AM +0000, Ryder Wang wrote:
> By walking through all the related code, it looks to be a bug in slub.c rather than kobject or ubifs.
>
> sysfs_slab_add() calls kobject_init_and_add():
> - If kobject_init_and_add fails, sysfs_slab_add() will go to *out*. But unluckily, *out* code block will never release s->kobj, but it is expected to do so.
>
> Below is the function comment of kobject_init_and_add():
> * If this function returns an error, kobject_put() must be called to properly clean up the memory associated with the object. *
> ==> It means sysfs_slab_add() shall release the related kobject.

Yup, that's a bug, please send a patch to fix this!

thanks,

greg k-h