Scheduler error on last stable web2py version 2.24.1

[Lisandro]

I've recently upgraded to web2py Version 2.24.1-stable+timestamp.2023.03.23.05.07.17

It's running on python 3.9.14, Rocky Linux RHEL9, using PostgreSQL 15.2 for database.

Since I did the upgrade, the scheduler fails from time to time with this traceback:

ERROR:web2py.scheduler.main#1531711:    error storing result
Traceback (most recent call last):
  File "/var/www/medios/gluon/scheduler.py", line 1077, in wrapped_report_task
    self.report_task(task, task_report)
  File "/var/www/medios/gluon/scheduler.py", line 1101, in report_task
    db(sr.id == task.run_id).update(
  File "/var/www/medios/gluon/packages/dal/pydal/objects.py", line 2789, in update
    ret = db._adapter.update(table, self.query, row.op_values())
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line 586, in update
    raise e
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line 581, in update
    self.execute(sql)
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/__init__.py", line 69, in wrap
    return f(*args, **kwargs)
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line 468, in execute
    rv = self.cursor.execute(command, *args[1:], **kwargs)
psycopg2.OperationalError: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
Traceback (most recent call last):
  File "/var/www/medios/gluon/scheduler.py", line 1077, in wrapped_report_task
    self.report_task(task, task_report)
  File "/var/www/medios/gluon/scheduler.py", line 1101, in report_task
    db(sr.id == task.run_id).update(
  File "/var/www/medios/gluon/packages/dal/pydal/objects.py", line 2789, in update
    ret = db._adapter.update(table, self.query, row.op_values())
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line 586, in update
    raise e
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line 581, in update
    self.execute(sql)
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/__init__.py", line 69, in wrap
    return f(*args, **kwargs)
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line 468, in execute
    rv = self.cursor.execute(command, *args[1:], **kwargs)
psycopg2.OperationalError: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/var/www/medios/gluon/shell.py", line 321, in run
    exec(python_code, _env)
  File "<string>", line 1, in <module>
  File "/var/www/medios/gluon/scheduler.py", line 949, in loop
    self.wrapped_report_task(task, self.execute(task))
  File "/var/www/medios/gluon/scheduler.py", line 1082, in wrapped_report_task
    db.rollback()
  File "/var/www/medios/gluon/packages/dal/pydal/base.py", line 825, in rollback
    self._adapter.rollback()
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/__init__.py", line 57, in wrap
    return f(*args, **kwargs)
  File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line 1012, in rollback
    return self.connection.rollback()
psycopg2.InterfaceError: connection already closed

I've checked PostgreSQL logs but there is no error or apparent problem at the time scheduler fails. The database instance has several databases and no error log is reported, everything runs smoothly. It's just the scheduler that reports that error (and after it, it doesn't run anymore). Where else should I look?

Any help will be much appreciated.

Warm regards,

Lisandro

[Massimiliano]

There was an issue but should be fixed now.

https://github.com/web2py/web2py/issues/1999

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/9ffdc6f1-8448-4784-a344-7f176545e9d9n%40googlegroups.com.

--

Massimiliano

[Lisandro]

(sorry, I've deleted my previous message because I have more detailed information).

Thank you for pointing that out, I had seen that issue but I missed out that I had to use Scheduler(..., with_spawn=True) so I did that change and, apparently, scheduler is stable (hasn't crashed so far). However, since that change, all the tasks that are from a different app that the one from where the scheduler was run, fail with this traceback:

Traceback (most recent call last): File "/home/limon/medios/gluon/scheduler.py", line 494, in executor functions = current._scheduler.tasks AttributeError: '_thread._local' object has no attribute '_scheduler'

Let me explain a bit more: I have one web2py instance with several applications, each one with its own database. One of these apps is the main one and the scheduler connects to its database.

...web2py/applications/main/...

...web2py/applications/app1/...

...web2py/applications/app2/...

I run three scheduler workers for the main app with this command:

/opt/virtualenvs/py39/bin/python /home/limon/web2py/web2py.py -K main,main,main

In applications/main/models/scheduler.py I instantiate the Scheduler like this:

scheduler = Scheduler(db, max_empty_runs=0, heartbeat=5, use_spawn=True)

And then, from several parts of my application I queue tasks. Some of these tasks are defined in the "main" application, but some others are defined in "app1" or "app2". 

Well, since the change to Scheduler(..., use_spawn=True) all the tasks within "main" application run ok, but all the other ones fail with the traceback I showed before. 

Notice I run the three scheduler workers for "main" application, I'm not sure if that has something to do with the issue. But I can confirm all this setup was working smoothly before use_spawn=True.

What could be happening?

Any help will be much appreciated.

I'll keep investigating and post here if I find something.

Thanks!

[Lisandro]

I've posted an issue and I've already proposed a small fix and made a pull request :)