Py4web scheduler - tracking task progress

[Paul Gerrard]

Hi all,

I'm building an app which uses the OpenAI API to process prompts using gpt-5-mini. Some of the prompts take over a minute to process and I was getting timeout problems. So... after a bit of research I thought I'd try the Scheduler to run these prompts through the AI aysynchronously.

I have a couple of trial tasks that work ok, but I have a question...

When a task is queued, does the scheduler.enqueue_run return any info to the calling code?

Ideally, I'd like to get the ID of the task_run record so I could monitor progress (maybe using some javascript in a web page) and act on the response captured in the output field in db.task_run record. I suppose I could generate a unique reference for the task name and find the record in the task_run table that way, but is there a simpler way to do this?

Thanks for any ideas in advance, as always :O)

Paul.

[Massimo DiPierro]

Good question. This is an oversight. I just added a pydal commit ee77a21176b20c893693dc53f9f03dee5127877f that makes enqueue_run return the run_id. I will release a new py4web today using this.

[Paul Gerrard]

Thanks so much Massimo. I have a related quesiton, I'm afraid.

I have a task that I can test and works OK when run from the shell. When I try to use the scheduler to run it, it seems to enqueue OK, and the task_run table shows it queued, but the task_run shows status=unknown, and the PID, STARTED_ON, COMPLETED_ON, LOG, AND OUTPUT fields are all None. 

So I guess the scheduler didn't accept the task or it crashed in some way. Maybe my code failed in some way, but there's no ticket in the Py4web log and no evidence to follow up/debug the problem. Does the scheduler have any other status/logging info it could capture to help? I can look at the scheduler code to maybe capture it, but I'm scratching my head at the moment.

Sorry to bug you.

[Massimo DiPierro]

This must be a bug of some kind. I will have to audit the schedule and see what may be causing it. I will do it asap.

[Paul Gerrard]

Hi again Massimo,

I'm seeing some strange behaviour in an ajax call I've been using to list the task_run table. (I'll explain later).

After reboots and resets etc. I am now seeing error messages in the task_run log for example:

Traceback (most recent call last): File "/home/servicemanager/.cache/pypoetry/virtualenvs/apps-XdOjxuR5-py3.12/lib/python3.12/site-packages/pydal/tools/scheduler.py", line 300, in exec_child output = self.tasks[run.name](**run.inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/www-data/py4web/apps/_default/controllers/dictionary.py", line 418, in generateDefinitionTask response, msg = dictAI.getTermDefinition(profile, term) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/www-data/py4web/apps/_default/modules/dictAI.py", line 14, in getTermDefinition application = db.applications[profile.appid] ^^^^^^^^^^^^^ AttributeError: 'dict' object has no attribute 'appid'

OK - I can work on that. 

* But this message isn't appearing in the tickets dashboard for some reason (or is this expected behaviour?) *

(The ajax call simply lists a table of the records in the task_run table. I *know* I have deleted all the task_run records but the ajax returns what looks like a cached response. I think that's MY problem. That confused the hell out of me, I have to say.)

So I suspect there isn't a bug at your end. I just have a query on whether scheduled tasks - on error - log to the ticket system? 

[Massimo DiPierro]

They do not. But they are supposed to log to the traceback in the db.task_run.log field and the task status is supposed to be set to "failed". Do you see a log at all?