scheduler leaves 'zombie' processes behind

[Tom Clerckx]

My current py4web version:

version 1.20250518.2 (git commit 075eac5f77fafee8fb53f94deecf16ee219d8484)

When checking out the scheduler, I see that for every process started by the scheduler,  a zombie process is left behind.

Sample:

import time

from pydal.tools.scheduler import Scheduler

from py4web import DAL

def my_task(**inputs):

print(f"task running with {inputs}")

time.sleep(20)

return {}

schedulerdb = DAL(

"sqlite://testscheduler.db",

folder="./",

pool_size=1,

migrate=True,

fake_migrate=False,

)

scheduler = Scheduler(

schedulerdb,

#logger=logger,

max_concurrent_runs=5

)

scheduler.register_task("my_task", my_task)

scheduler.enqueue_run("my_task", inputs={})

scheduler.enqueue_run("my_task", inputs={})

scheduler.enqueue_run("my_task", inputs={})

scheduler.enqueue_run("my_task", inputs={})

scheduler.start()

ps auxf output shows zombie processes:

tclerckx 1873587 4.8 0.1 123280 38040 pts/3 Sl+ 01:14 0:00 | \_ /home/tclerckx/.pyenv/versions/py4web/bin/python schedulertest.py

tclerckx 1873641 0.0 0.0 0 0 pts/3 Z+ 01:14 0:00 | \_ [python] <defunct>

tclerckx 1873643 0.0 0.0 0 0 pts/3 Z+ 01:14 0:00 | \_ [python] <defunct>

tclerckx 1873645 0.1 0.0 0 0 pts/3 Z+ 01:14 0:00 | \_ [python] <defunct>

tclerckx 1873647 0.0 0.0 0 0 pts/3 Z+ 01:14 0:00 | \_ [python] <defunct>

The number of zombie processes accumulates for every new scheduler run.

[Tom Clerckx]

I saw that this was fixed in pydal 20250226.3

I tested it and confirm the fix.

Grazie!

[Tom Clerckx]

For some reason my post doesn't get through - apology if this messages gets posted multiple times.

The change in pydal can lead to the following error:

  File "/odindrive/temp/2025-04-23-py4web-migration/pydal.git/pydal/tools/scheduler.py", line 115, in __init__
    signal.signal(signal.SIGCHLD, signal.SIG_IGN)
  File "/home/tclerckx/.pyenv/versions/3.10.14/lib/python3.10/signal.py", line 56, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread of the main interpreter

Potential fix:

diff --git a/pydal/tools/scheduler.py b/pydal/tools/scheduler.py

index 01f0e57c..dabdaf28 100644

--- a/pydal/tools/scheduler.py

+++ b/pydal/tools/scheduler.py

@@ -112,7 +112,8 @@ class Scheduler: # pylint: disable=too-many-instance-attributes

sleep_time=10,

logger=None,

):

- signal.signal(signal.SIGCHLD, signal.SIG_IGN)

+ if threading.current_thread().name == "MainThread":

+ signal.signal(signal.SIGCHLD, signal.SIG_IGN)

self.db = db

self.max_concurrent_runs = max_concurrent_runs

self.folder = folder

[Massimo]

I would like to understand how to reproduce this. I do not think the scheduler works fine if the class Scheduler is instantiated in a thread other than main. It will not collect zombie processes.

[Tom Clerckx]

Indeed.

This happens during development when the code is reloaded. I think the reload happens on a separate thread, causing this issue.

Op za 7 jun 2025 om 07:02 schreef Massimo <massimo....@gmail.com>

--
You received this message because you are subscribed to a topic in the Google Groups "py4web" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/py4web/j_kvw7QiWT4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to py4web+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/py4web/8d19249d-70de-465c-a78a-6505d6a37e8fn%40googlegroups.com.