Re: [syzbot] [rdma?] WARNING in gid_table_release_one (3)

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-by: syzbot+b0da83...@syzkaller.appspotmail.com
Tested-by: syzbot+b0da83...@syzkaller.appspotmail.com

Tested on:

commit: dd6adb14 RDMA/core: Fix WARNING in gid_table_release_one
git tree: https://github.com/zhuyj/linux.git v6.17_fix_gid_table_release_one
console output: https://syzkaller.appspot.com/x/log.txt?x=16d81342580000
kernel config: https://syzkaller.appspot.com/x/.config?x=2c614fa9e6f5bdc1
dashboard link: https://syzkaller.appspot.com/bug?extid=b0da83a6c0e2e2bddbd4
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8

Note: no patches were applied.
Note: testing is done by a robot and is best-effort only.

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-by: syzbot+b0da83...@syzkaller.appspotmail.com
Tested-by: syzbot+b0da83...@syzkaller.appspotmail.com

Tested on:

commit: dd6adb14 RDMA/core: Fix WARNING in gid_table_release_one
git tree: https://github.com/zhuyj/linux.git v6.17_fix_gid_table_release_one

console output: https://syzkaller.appspot.com/x/log.txt?x=11041342580000

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
WARNING in gid_table_release_one

------------[ cut here ]------------
GID entry ref leak for dev syz1 index 2 ref=363, state: 3
WARNING: CPU: 1 PID: 50 at drivers/infiniband/core/cache.c:827 release_gid_table drivers/infiniband/core/cache.c:824 [inline]
WARNING: CPU: 1 PID: 50 at drivers/infiniband/core/cache.c:827 gid_table_release_one+0x5ae/0x6c0 drivers/infiniband/core/cache.c:904
Modules linked in:
CPU: 1 UID: 0 PID: 50 Comm: kworker/u8:3 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
Workqueue: ib-unreg-wq ib_unregister_work
RIP: 0010:release_gid_table drivers/infiniband/core/cache.c:824 [inline]
RIP: 0010:gid_table_release_one+0x5ae/0x6c0 drivers/infiniband/core/cache.c:904
Code: e8 03 0f b6 04 28 84 c0 0f 85 cc 00 00 00 44 8b 03 48 c7 c7 60 7c 2b 8c 48 8b 74 24 28 44 89 fa 8b 4c 24 50 e8 73 e7 35 f9 90 <0f> 0b 90 90 44 8b 74 24 04 4c 8b 7c 24 20 4c 8b 64 24 48 e9 15 fe
RSP: 0018:ffffc90000bb78f8 EFLAGS: 00010246
RAX: 124fa0acf3bf2700 RBX: ffff8880268c1990 RCX: ffff888020289e40
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000002
RBP: dffffc0000000000 R08: 0000000000000003 R09: 0000000000000004
R10: dffffc0000000000 R11: fffffbfff1b7a678 R12: ffff88802ed4e2d8
R13: 00000000000001a8 R14: ffff88806a158010 R15: 0000000000000002
FS: 0000000000000000(0000) GS:ffff88812646a000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005555712ce808 CR3: 000000006b6c8000 CR4: 00000000003526f0
Call Trace:
<TASK>
ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509
device_release+0x9c/0x1c0 drivers/base/core.c:-1
kobject_cleanup lib/kobject.c:689 [inline]
kobject_release lib/kobject.c:720 [inline]
kref_put include/linux/kref.h:65 [inline]
kobject_put+0x22b/0x480 lib/kobject.c:737
process_one_work kernel/workqueue.c:3263 [inline]
process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346
worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427
kthread+0x711/0x8a0 kernel/kthread.c:463
ret_from_fork+0x4bc/0x870 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>


Tested on:

commit: ad2cc78b RDMA/core: Fix WARNING in gid_table_release_one

git tree: https://github.com/zhuyj/linux.git v6.17_fix_gid_table_release_one

console output: https://syzkaller.appspot.com/x/log.txt?x=11dfa17c580000

[Jason Gunthorpe]

I think this disproves the theory that the the gid is sitting in a
work queue waiting to be cleaned up..

So we still need to find out what is holding on to the reference...

Jason

[Yanjun.Zhu]

Got it.

Thanks a lot.

Yanjun.Zhu

>
> Jason

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-by: syzbot+b0da83...@syzkaller.appspotmail.com
Tested-by: syzbot+b0da83...@syzkaller.appspotmail.com

Tested on:

commit: 1c3df6de RDMA/core: Fix WARNING in gid_table_release_one

git tree: https://github.com/zhuyj/linux.git v6.17_fix_gid_table_release_one

console output: https://syzkaller.appspot.com/x/log.txt?x=10908532580000

kernel config: https://syzkaller.appspot.com/x/.config?x=2c614fa9e6f5bdc1
dashboard link: https://syzkaller.appspot.com/bug?extid=b0da83a6c0e2e2bddbd4
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8

Note: no patches were applied.

[Leon Romanovsky]

Yes, this is makes more sense to me when multiple ib_wq flush.

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
WARNING in gid_table_release_one

------------[ cut here ]------------

GID entry ref leak for dev syz1 index 2 ref=328, state: 3
WARNING: CPU: 1 PID: 49 at drivers/infiniband/core/cache.c:829 release_gid_table drivers/infiniband/core/cache.c:826 [inline]
WARNING: CPU: 1 PID: 49 at drivers/infiniband/core/cache.c:829 gid_table_release_one+0x64c/0x780 drivers/infiniband/core/cache.c:908
Modules linked in:
CPU: 1 UID: 0 PID: 49 Comm: kworker/u8:3 Not tainted syzkaller #0 PREEMPT(full)

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
Workqueue: ib-unreg-wq ib_unregister_work

RIP: 0010:release_gid_table drivers/infiniband/core/cache.c:826 [inline]
RIP: 0010:gid_table_release_one+0x64c/0x780 drivers/infiniband/core/cache.c:908
Code: e8 03 0f b6 04 28 84 c0 0f 85 f1 00 00 00 44 8b 03 48 c7 c7 00 7d 2b 8c 48 8b 74 24 28 44 89 fa 8b 4c 24 50 e8 35 e5 35 f9 90 <0f> 0b 90 90 44 8b 74 24 0c 4c 8b 64 24 48 48 8b 1c 24 e9 1a fe ff
RSP: 0018:ffffc90000b978f0 EFLAGS: 00010246
RAX: 79fe70c1b2777a00 RBX: ffff88802780ac90 RCX: ffff888021eb0000

RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000002
RBP: dffffc0000000000 R08: 0000000000000003 R09: 0000000000000004

R10: dffffc0000000000 R11: fffffbfff1b7a680 R12: ffff888075294cd8
R13: 00000000000001a8 R14: ffff88802e928010 R15: 0000000000000002

FS: 0000000000000000(0000) GS:ffff88812646a000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 0000200000000240 CR3: 000000007e658000 CR4: 00000000003526f0

Call Trace:
<TASK>
ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509
device_release+0x9c/0x1c0 drivers/base/core.c:-1
kobject_cleanup lib/kobject.c:689 [inline]
kobject_release lib/kobject.c:720 [inline]
kref_put include/linux/kref.h:65 [inline]
kobject_put+0x22b/0x480 lib/kobject.c:737
process_one_work kernel/workqueue.c:3263 [inline]
process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346
worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427
kthread+0x711/0x8a0 kernel/kthread.c:463
ret_from_fork+0x4bc/0x870 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>


Tested on:

commit: 25f4e06f RDMA/core: Fix WARNING in gid_table_release_one

git tree: https://github.com/zhuyj/linux.git v6.17_fix_gid_table_release_one

console output: https://syzkaller.appspot.com/x/log.txt?x=109f532f980000

kernel config: https://syzkaller.appspot.com/x/.config?x=2c614fa9e6f5bdc1
dashboard link: https://syzkaller.appspot.com/bug?extid=b0da83a6c0e2e2bddbd4
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8

[Yanjun.Zhu]

It’s still unclear what is holding the reference. From my tests, if we
wait here for a short time, all the references are eventually released.
It’s quite strange.

Yanjun.Zhu

>>
>> Jason

[Yanjun.Zhu]

On 11/5/25 10:50 AM, Leon Romanovsky wrote:
>

ib_unreg_wq also has to handle gid. Thus the 2 workqueues (ib_wq and
ib_unreg_wq) handle gid.

Thus it is not enough to just flush ib_wq.

Yanjun.Zhu

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
WARNING in gid_table_release_one

------------[ cut here ]------------

GID entry ref leak for dev syz1 index 2 ref=381, state: 3
WARNING: CPU: 0 PID: 3018 at drivers/infiniband/core/cache.c:829 release_gid_table drivers/infiniband/core/cache.c:826 [inline]
WARNING: CPU: 0 PID: 3018 at drivers/infiniband/core/cache.c:829 gid_table_release_one+0x64c/0x780 drivers/infiniband/core/cache.c:908
Modules linked in:
CPU: 0 UID: 0 PID: 3018 Comm: kworker/u8:9 Not tainted syzkaller #0 PREEMPT(full)

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
Workqueue: ib-unreg-wq ib_unregister_work

RIP: 0010:release_gid_table drivers/infiniband/core/cache.c:826 [inline]
RIP: 0010:gid_table_release_one+0x64c/0x780 drivers/infiniband/core/cache.c:908

Code: e8 03 0f b6 04 28 84 c0 0f 85 f1 00 00 00 44 8b 03 48 c7 c7 00 7d 2b 8c 48 8b 74 24 28 44 89 fa 8b 4c 24 50 e8 05 e5 35 f9 90 <0f> 0b 90 90 44 8b 74 24 0c 4c 8b 64 24 48 48 8b 1c 24 e9 1a fe ff
RSP: 0018:ffffc9000bf478f0 EFLAGS: 00010246
RAX: eb4f68ae1be28b00 RBX: ffff888028b86a90 RCX: ffff88802e945ac0

RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000002
RBP: dffffc0000000000 R08: 0000000000000003 R09: 0000000000000004

R10: dffffc0000000000 R11: fffffbfff1b7a680 R12: ffff8880746374d8
R13: 00000000000001a8 R14: ffff888054220010 R15: 0000000000000002
FS: 0000000000000000(0000) GS:ffff88812636a000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007f1cff997f98 CR3: 000000007478c000 CR4: 00000000003526f0

Call Trace:
<TASK>
ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509
device_release+0x9c/0x1c0 drivers/base/core.c:-1
kobject_cleanup lib/kobject.c:689 [inline]
kobject_release lib/kobject.c:720 [inline]
kref_put include/linux/kref.h:65 [inline]
kobject_put+0x22b/0x480 lib/kobject.c:737
process_one_work kernel/workqueue.c:3263 [inline]
process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346
worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427
kthread+0x711/0x8a0 kernel/kthread.c:463
ret_from_fork+0x4bc/0x870 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>


Tested on:

commit: af436594 RDMA/core: Fix WARNING in gid_table_release_one

git tree: https://github.com/zhuyj/linux.git v6.17_fix_gid_table_release_one

console output: https://syzkaller.appspot.com/x/log.txt?x=15f6ca58580000

kernel config: https://syzkaller.appspot.com/x/.config?x=2c614fa9e6f5bdc1
dashboard link: https://syzkaller.appspot.com/bug?extid=b0da83a6c0e2e2bddbd4
compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
WARNING in gid_table_release_one

------------[ cut here ]------------

GID entry ref leak for dev syz1 index 2 ref=353, state: 3
WARNING: CPU: 1 PID: 36 at drivers/infiniband/core/cache.c:817 release_gid_table drivers/infiniband/core/cache.c:814 [inline]
WARNING: CPU: 1 PID: 36 at drivers/infiniband/core/cache.c:817 gid_table_release_one+0x39e/0x570 drivers/infiniband/core/cache.c:894
Modules linked in:
CPU: 1 UID: 0 PID: 36 Comm: kworker/u8:2 Not tainted syzkaller #0 PREEMPT(full)

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
Workqueue: ib-unreg-wq ib_unregister_work

RIP: 0010:release_gid_table drivers/infiniband/core/cache.c:814 [inline]
RIP: 0010:gid_table_release_one+0x39e/0x570 drivers/infiniband/core/cache.c:894
Code: 04 28 84 c0 0f 85 9d 00 00 00 45 8b 06 48 c7 c7 60 7c 2b 8c 48 8b 74 24 28 4c 8b 74 24 08 44 89 f2 44 89 e1 e8 83 e9 35 f9 90 <0f> 0b 90 90 e9 c5 fe ff ff e8 64 7e 72 f9 4c 8b 64 24 40 4c 89 e0
RSP: 0018:ffffc90000ac7900 EFLAGS: 00010246
RAX: d89947cea50d0900 RBX: ffff888076c960d8 RCX: ffff8881436c9e40

RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000002

RBP: ffff888076c96000 R08: 0000000000000003 R09: 0000000000000004
R10: dffffc0000000000 R11: fffffbfff1b7a678 R12: 0000000000000161
R13: dffffc0000000000 R14: 0000000000000002 R15: 1ffff1100ed92c1b
FS: 0000000000000000(0000) GS:ffff88812646a000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007f9b2bf056c0 CR3: 000000000db38000 CR4: 00000000003526f0

Call Trace:
<TASK>
ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509
device_release+0x9c/0x1c0 drivers/base/core.c:-1
kobject_cleanup lib/kobject.c:689 [inline]
kobject_release lib/kobject.c:720 [inline]
kref_put include/linux/kref.h:65 [inline]
kobject_put+0x22b/0x480 lib/kobject.c:737
process_one_work kernel/workqueue.c:3263 [inline]
process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346
worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427
kthread+0x711/0x8a0 kernel/kthread.c:463
ret_from_fork+0x4bc/0x870 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>


Tested on:

commit: e365ec00 RDMA/core: Fix WARNING in gid_table_release_one

git tree: https://github.com/zhuyj/linux.git v6.17_fix_gid_table_release_one

console output: https://syzkaller.appspot.com/x/log.txt?x=17a24a92580000

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
WARNING in gid_table_release_one

------------[ cut here ]------------

Possibly kref leak, ref:403, state: 3
WARNING: CPU: 0 PID: 12 at drivers/infiniband/core/cache.c:820 release_gid_table drivers/infiniband/core/cache.c:819 [inline]
WARNING: CPU: 0 PID: 12 at drivers/infiniband/core/cache.c:820 gid_table_release_one+0x4fb/0x800 drivers/infiniband/core/cache.c:897
Modules linked in:
CPU: 0 UID: 0 PID: 12 Comm: kworker/u8:0 Not tainted syzkaller #0 PREEMPT(full)

Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
Workqueue: ib-unreg-wq ib_unregister_work

RIP: 0010:release_gid_table drivers/infiniband/core/cache.c:819 [inline]
RIP: 0010:gid_table_release_one+0x4fb/0x800 drivers/infiniband/core/cache.c:897
Code: 49 01 c6 4c 89 f0 48 c1 e8 03 42 0f b6 04 28 84 c0 0f 85 be 01 00 00 41 8b 16 48 c7 c7 60 7d 2b 8c 44 89 e6 e8 16 e2 35 f9 90 <0f> 0b 90 90 e9 c4 fe ff ff e8 f7 76 72 f9 4c 8b 7c 24 48 4c 89 f8
RSP: 0018:ffffc90000117900 EFLAGS: 00010246
RAX: a2ec906fdf0a5800 RBX: ffff888030cab6d8 RCX: ffff88801c6e5ac0

RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000002

RBP: ffff888079d48010 R08: 0000000000000003 R09: 0000000000000004
R10: dffffc0000000000 R11: fffffbfff1b7a680 R12: 0000000000000193
R13: dffffc0000000000 R14: ffff888028a68690 R15: ffff888030cab600
FS: 0000000000000000(0000) GS:ffff88812636a000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

CR2: 00007f49bf3056c0 CR3: 00000000772e8000 CR4: 00000000003526f0

Call Trace:
<TASK>
ib_device_release+0xd2/0x1c0 drivers/infiniband/core/device.c:509
device_release+0x9c/0x1c0 drivers/base/core.c:-1
kobject_cleanup lib/kobject.c:689 [inline]
kobject_release lib/kobject.c:720 [inline]
kref_put include/linux/kref.h:65 [inline]
kobject_put+0x22b/0x480 lib/kobject.c:737
process_one_work kernel/workqueue.c:3263 [inline]
process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3346
worker_thread+0x8a0/0xda0 kernel/workqueue.c:3427
kthread+0x711/0x8a0 kernel/kthread.c:463
ret_from_fork+0x4bc/0x870 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>


Tested on:

commit: 690e9fda RDMA/core: Fix WARNING in gid_table_release_one

git tree: https://github.com/zhuyj/linux.git v6.17_fix_gid_table_release_one

console output: https://syzkaller.appspot.com/x/log.txt?x=16bbb012580000

[Yanjun.Zhu]

On 11/6/25 1:02 PM, syzbot wrote:
> Hello,
>
> syzbot has tested the proposed patch but the reproducer is still triggering an issue:
> WARNING in gid_table_release_one
>
> ------------[ cut here ]------------
> Possibly kref leak, ref:403, state: 3
> WARNING: CPU: 0 PID: 12 at drivers/infiniband/core/cache.c:820 release_gid_table drivers/infiniband/core/cache.c:819 [inline]
> WARNING: CPU: 0 PID: 12 at drivers/infiniband/core/cache.c:820 gid_table_release_one+0x4fb/0x800 drivers/infiniband/core/cache.c:897
> Modules linked in:
> CPU: 0 UID: 0 PID: 12 Comm: kworker/u8:0 Not tainted syzkaller #0 PREEMPT(full)
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/02/2025
> Workqueue: ib-unreg-wq ib_unregister_work

The problem occurred in ib_unreg_wq. It is a workqueue in ib device.

I will work on this problem.

Yanjun.Zhu