Re: [PATCH] keys: use long workqueue for garbage collection

[Jarkko Sakkinen]

On Sat, Oct 24, 2026 at 06:29:44AM +0200, Eduard Zateev wrote:
> The key garbage collector can wait for RCU grace periods. This can occupy a
> normal events worker for a long time and contribute to workqueue lockups.
>
> Queue key GC on system_long_wq because its work may wait for long periods.
>
> Reported-by: syzbot+c15de4...@syzkaller.appspotmail.com
> Link: https://syzkaller.appspot.com/bug?extid=c15de43f1e7fffbb7b9b
> Signed-off-by: Eduard Zateev <hacker...@gmail.com>

This commit message is not huge selling point for any possible argument.

> ---
> security/keys/gc.c | 9 +++++----
> 1 file changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/security/keys/gc.c b/security/keys/gc.c
> index eda445f815d..b174b8c511b 100644
> --- a/security/keys/gc.c
> +++ b/security/keys/gc.c
> @@ -19,6 +19,7 @@ unsigned key_gc_delay = 5 * 60;
> * Reaper for unused keys.
> */
> static void key_garbage_collector(struct work_struct *work);
> +/* Key GC may wait for RCU grace periods, so queue it on system_long_wq. */
> DECLARE_WORK(key_gc_work, key_garbage_collector);
>
> /*
> @@ -57,7 +58,7 @@ void key_schedule_gc(time64_t gc_at)
>
> if (gc_at <= now || test_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags)) {
> kdebug("IMMEDIATE");
> - schedule_work(&key_gc_work);
> + queue_work(system_long_wq, &key_gc_work);
> } else if (gc_at < key_gc_next_run) {
> kdebug("DEFERRED");
> key_gc_next_run = gc_at;
> @@ -85,7 +86,7 @@ void key_set_expiry(struct key *key, time64_t expiry)
> void key_schedule_gc_links(void)
> {
> set_bit(KEY_GC_KEY_EXPIRED, &key_gc_flags);
> - schedule_work(&key_gc_work);
> + queue_work(system_long_wq, &key_gc_work);
> }
>
> /*
> @@ -119,7 +120,7 @@ void key_gc_keytype(struct key_type *ktype)
> set_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags);
>
> kdebug("schedule");
> - schedule_work(&key_gc_work);
> + queue_work(system_long_wq, &key_gc_work);
>
> kdebug("sleep");
> wait_on_bit(&key_gc_flags, KEY_GC_REAPING_KEYTYPE,
> @@ -322,7 +323,7 @@ static void key_garbage_collector(struct work_struct *work)
> }
>
> if (gc_state & KEY_GC_REAP_AGAIN)
> - schedule_work(&key_gc_work);
> + queue_work(system_long_wq, &key_gc_work);
> kleave(" [end %x]", gc_state);
> return;
>
> --
> 2.55.0
>

Br, Jarkko

[Jarkko Sakkinen]

On Sat, Oct 24, 2026 at 08:20:55AM +0300, Jarkko Sakkinen wrote:
> On Sat, Oct 24, 2026 at 06:29:44AM +0200, Eduard Zateev wrote:
> > The key garbage collector can wait for RCU grace periods. This can occupy a
> > normal events worker for a long time and contribute to workqueue lockups.
> >
> > Queue key GC on system_long_wq because its work may wait for long periods.
> >
> > Reported-by: syzbot+c15de4...@syzkaller.appspotmail.com
> > Link: https://syzkaller.appspot.com/bug?extid=c15de43f1e7fffbb7b9b
> > Signed-off-by: Eduard Zateev <hacker...@gmail.com>
>
> This commit message is not huge selling point for any possible argument.

You ignored key_gc_timed_out(), which leaves code to a broken state and
switching workqueue is not solving any possible root cause of anything.
The long queue exists mostly for the sake of flush_workqueue().

Please stop sending patches like this in the future.

Br, Jarkko

[Jarkko Sakkinen]

Makes me wonder was this even tried a single time TBH given the obvious
glitch. I would bet for no.

Br, Jarkko

[sashi...@kernel.org]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] keys: concurrent key_garbage_collector() via schedule_work() in key_put()
--

commit 49fec25be24d41f55255824b44e92e9f57128440
Author: Eduard Zateev <hacker...@gmail.com>

keys: use long workqueue for garbage collection

This commit switches the key garbage collector work item (key_gc_work) in
security/keys/gc.c from system_percpu_wq to system_long_wq because key
garbage collection can block waiting for RCU grace periods.

Link: https://syzkaller.appspot.com/bug?extid=c15de43f1e7fffbb7b9b

> diff --git a/security/keys/gc.c b/security/keys/gc.c
> index eda445f815d47..b174b8c511b4b 100644

> --- a/security/keys/gc.c
> +++ b/security/keys/gc.c
> @@ -19,6 +19,7 @@ unsigned key_gc_delay = 5 * 60;
> * Reaper for unused keys.
> */
> static void key_garbage_collector(struct work_struct *work);
> +/* Key GC may wait for RCU grace periods, so queue it on system_long_wq. */
> DECLARE_WORK(key_gc_work, key_garbage_collector);
>
> /*
> @@ -57,7 +58,7 @@ void key_schedule_gc(time64_t gc_at)
>
> if (gc_at <= now || test_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags)) {
> kdebug("IMMEDIATE");
> - schedule_work(&key_gc_work);
> + queue_work(system_long_wq, &key_gc_work);

[Severity: Critical]
Does this allow key_garbage_collector() to run concurrently with itself across
two CPUs because key_put() still queues key_gc_work on system_percpu_wq?

While this patch updates the four schedule_work(&key_gc_work) call sites in
security/keys/gc.c to use system_long_wq, key_put() in security/keys/key.c
still calls schedule_work(&key_gc_work):

security/keys/key.c:key_put() {
if (key) {
key_check(key);

if (refcount_dec_and_test(&key->usage)) {
/* Mark key as safe for GC after key->user done. */
clear_bit_unlock(KEY_FLAG_USER_ALIVE, &key->flags);
schedule_work(&key_gc_work);
}
}
}

First, since key_put() is the path that clears KEY_FLAG_USER_ALIVE to trigger
reaping of unreferenced keys and the resulting synchronize_rcu() call, normal
system_percpu_wq workers will still execute key_garbage_collector() and block
on RCU grace periods.

Second, in __queue_work(), cross-CPU non-reentrancy for a work_struct that is
already executing relies on matching the target workqueue against the running
worker's workqueue:

kernel/workqueue.c:__queue_work() {
...
last_pool = get_work_pool(work);
if (last_pool && last_pool != pool) {
worker = find_worker_executing_work(last_pool, work);

if (worker && worker->current_pwq->wq == wq) {
pwq = worker->current_pwq;\
pool = pwq->pool;
WARN_ON_ONCE(pool != last_pool);
} else {
/* meh... not running there, queue here */
raw_spin_unlock(&last_pool->lock);
raw_spin_lock(&pool->lock);
}
}
...
}

Once process_one_work() clears WORK_STRUCT_PENDING_BIT and begins executing
key_garbage_collector() on one CPU under system_long_wq, a concurrent
key_put() -> schedule_work(&key_gc_work) on another CPU passes wq ==
system_percpu_wq (or vice versa). Because worker->current_pwq->wq == wq is
false, __queue_work() queues key_gc_work on the second CPU's pool and runs a
second instance of key_garbage_collector() in parallel while the first is
sleeping in synchronize_rcu(), cond_resched(), or down_write(&key->sem).

key_garbage_collector() assumes only a single instance can run at a time:

security/keys/gc.c:key_garbage_collector() {
static LIST_HEAD(graveyard);
static u8 gc_state; /* Internal persistent state */
...
/* As only this function is permitted to remove things from the key
* serial tree, if cursor is non-NULL then it will always point to a
* valid node in the tree - even if lock got dropped.
*/
spin_lock(&key_serial_lock);
cursor = rb_first(&key_serial_tree);
...
}

Can concurrent execution of key_garbage_collector() corrupt the static
graveyard list and gc_state, free keys in key_gc_unused_keys() before the
other worker's synchronize_rcu() completes, or rb_erase() and free the key
pointed to by the other worker's cursor while key_serial_lock is dropped?

Should key_put() in security/keys/key.c also be updated to call
queue_work(system_long_wq, &key_gc_work)?

--
Sashiko AI review · https://sashiko.dev/#/patchset/20261024042944.52...@gmail.com?part=1

[Eduard Zateev]

The key garbage collector can wait for RCU grace periods. This can occupy a
normal events worker for a long time and contribute to workqueue lockups.

Queue key GC on system_long_wq because its work may wait for long periods.

Reported-by: syzbot+c15de4...@syzkaller.appspotmail.com
Link: https://syzkaller.appspot.com/bug?extid=c15de43f1e7fffbb7b9b
Signed-off-by: Eduard Zateev <hacker...@gmail.com>

---
security/keys/gc.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/security/keys/gc.c b/security/keys/gc.c
index eda445f815d..b174b8c511b 100644

--- a/security/keys/gc.c
+++ b/security/keys/gc.c
@@ -19,6 +19,7 @@ unsigned key_gc_delay = 5 * 60;
* Reaper for unused keys.
*/
static void key_garbage_collector(struct work_struct *work);
+/* Key GC may wait for RCU grace periods, so queue it on system_long_wq. */
DECLARE_WORK(key_gc_work, key_garbage_collector);

/*
@@ -57,7 +58,7 @@ void key_schedule_gc(time64_t gc_at)

if (gc_at <= now || test_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags)) {
kdebug("IMMEDIATE");
- schedule_work(&key_gc_work);
+ queue_work(system_long_wq, &key_gc_work);

} else if (gc_at < key_gc_next_run) {
kdebug("DEFERRED");
key_gc_next_run = gc_at;
@@ -85,7 +86,7 @@ void key_set_expiry(struct key *key, time64_t expiry)
void key_schedule_gc_links(void)
{
set_bit(KEY_GC_KEY_EXPIRED, &key_gc_flags);

- schedule_work(&key_gc_work);
+ queue_work(system_long_wq, &key_gc_work);
}

/*
@@ -119,7 +120,7 @@ void key_gc_keytype(struct key_type *ktype)
set_bit(KEY_GC_REAP_KEYTYPE, &key_gc_flags);

kdebug("schedule");

- schedule_work(&key_gc_work);
+ queue_work(system_long_wq, &key_gc_work);

kdebug("sleep");
wait_on_bit(&key_gc_flags, KEY_GC_REAPING_KEYTYPE,

@@ -322,7 +323,7 @@ static void key_garbage_collector(struct work_struct *work)
}

if (gc_state & KEY_GC_REAP_AGAIN)

- schedule_work(&key_gc_work);
+ queue_work(system_long_wq, &key_gc_work);