Scheduler tasks fail intermitently with no apparent reason (sys.exit(1))

[Lisandro]

I've seeing this for a while now, but I can't figure out why it happens.

I have several tasks (around 150) that run once per day (at different hours, maybe some of them run simultaneously). 

They always run successfully with no problems. However, once every time, one of them fail with this traceback:

Traceback (most recent call last): File "/var/www/medios/gluon/scheduler.py", line 481, in executor extra_request={'is_scheduler': True}) File "/var/www/medios/gluon/shell.py", line 175, in env sys.exit(1) SystemExit: 1

Other times it fails with this traceback:

Traceback (most recent call last): File "/var/www/medios/gluon/scheduler.py", line 501, in executor result = dumps(_function(*args, **vars)) File "applications/thetimes/compiled/models.db.py", line 517, in newsletter File "applications/thetimes/modules/globales.py", line 1008, in enviar_newsletter sleep(1+throttle_count*3) File "/var/www/medios/gluon/scheduler.py", line 901, in signal.signal(signal.SIGTERM, lambda signum, stack_frame: sys.exit(1)) SystemExit: 1

It doesn't happen very often, but still I would like to know if there is something in my code that I should fix, or is something related to the scheduler itself.

Any observation or suggestion will be appreciated. Thanks!

[Niphlod]

Timeout?!

[Lisandro]

That was my first guess, but the task status in those cases was FAILED, not TIMEOUT.

I forgot to mention I'm using web2py version 2.16.1-stable+timestamp.2017.11.14.05.54.25

I'll keep monitoring and see if I can add any detail.

Thanks!

El martes, 15 de enero de 2019, 5:24:56 (UTC-3), Niphlod escribió:

Timeout?!

[Dave S]

.

Mine mostly look like

Traceback (most recent call last):

  File "C:\Users\Dave\web2py_win\web2py_win 2.14.6\web2py\gluon\scheduler.py", line 313, in executor
    result = dumps(_function(*args, **vars))
  File "applications\badlist\models\scheduler.py", line 6, in mp_fetch
    bin=fetch(url)
NameError: global name 'fetch' is not defined

where the scheduler clearly is just wrapping the error.   The traceback shows up in the scheduler_run table.

So I guess I'm not much help, except by negative example.

/dps

 

[Leonel Câmara]

Lisandro that problem is appearing when trying to create an environment to execute the task, where it imports the models from the application. Is it possible the folder of the application has been removed or moved before the scheduler task is run?

This would explain why it doesn't happen frequently.

Dave did you import fetch?

from gluon.tools import fetch

[Lisandro]

Thank you Leonel, that could be the reason, or at least could be related.

The "models" folder in those cases wasn't removed or anything like that, but still I should mention a couple of things about my scenario:

* I have several web2py apps installed, serving multiple websites; each website has two applications, and for every website, the "models" folder is present in the first application, and it's symlinked in the second one.

* Websites are created on demand: a scheduled task installs new applications, creating the necessary folders, db, etc.

The times those tasks failed, they didn't correspond to an application that was being installed at that time. 

I always run the scheduler with -K <main_app>, and main_app is of course always installed.

I'm not sure how the scheduler works, maybe it needs to read the models folder in every installed application? 

[Dave S]

Yes, I eventually got that straightened out.  Most of the subsequent errors have been similar, or typos in local var names.  Oh, another set is when I split a string to get fields x, y, and z ... but z isn't there (input error; I need to beef up my validation).

My post was just to illustrate not having the exit(1) Lisandro ran into.

/dps
 

[Dave S]

On Tuesday, January 15, 2019 at 11:31:51 AM UTC-8, Lisandro wrote:

Thank you Leonel, that could be the reason, or at least could be related.

The "models" folder in those cases wasn't removed or anything like that, but still I should mention a couple of things about my scenario:

* I have several web2py apps installed, serving multiple websites; each website has two applications, and for every website, the "models" folder is present in the first application, and it's symlinked in the second one.

* Websites are created on demand: a scheduled task installs new applications, creating the necessary folders, db, etc.

The times those tasks failed, they didn't correspond to an application that was being installed at that time. 

I always run the scheduler with -K <main_app>, and main_app is of course always installed.

I'm not sure how the scheduler works, maybe it needs to read the models folder in every installed application? 

Try stopping the scheduler for other apps during an install.  The task doing the install should have all it's environment set up [by the scheduler in it's app] before the the new app folder is created, so that should be alright.  You might want to make sure the installer app doesn't have anything else scheduled too close in time to the installation.

How often are you installing new apps?

/dps

 

El martes, 15 de enero de 2019, 15:34:53 (UTC-3), Leonel Câmara escribió:

Lisandro that problem is appearing when trying to create an environment to execute the task, where it imports the mod tels from the application. Is it possible the folder of the application has been removed or moved before the scheduler task is run?

[Lisandro]

Thank you all you guys for the help.

I've found an inconsistence in my own code. I have a scheduled task that checks all the other tasks of the scheduler and sends me an email if something failed or timed out. The email includes the error traceback, but an error is making the function to include an incorrect traceback, so that's where my confusion came. Sorry for the bother, I think it's working as it is supposed to.

To answer Dave's question: new apps are installed around 5 times per day. Also, once per week a bunch of apps are removed. This process works smoothly (I use uwsgi chain reload to avoid any issues). 

Thanks again!