scheduler: Cannot duplicate a Singleton

180 views
Skip to first unread message

Adi

unread,
Oct 4, 2012, 3:42:59 PM10/4/12
to web...@googlegroups.com
We promoted latest night build to production, and scheduler started failing. At the same time there were some code changes, so I'm having a hard time tracing the source of the problem.

Any insights while I'm trying to figure it out?

Thanks,
Adi

db: mysql
redhat linux
w2p:
Version 2.0.9 (2012-10-02 03:55:58) dev

scheduler_run table: traceback field:
Traceback (most recent call last):
  File "/opt/web-apps/web2py/gluon/scheduler.py", line 203, in executor
    _env = env(a=a,c=c,import_models=True)
  File "/opt/web-apps/web2py/gluon/shell.py", line 134, in env
    sys.exit(1)
SystemExit: 1


scheduler log:
/opt/web-apps/web2py/gluon/dal.py:6597: DeprecationWarning: object.__new__() takes no parameters
  db = super(DAL, cls).__new__(cls, uri, *args, **kwargs)
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2012
Version 2.0.9 (2012-10-02 03:55:58) dev
Database drivers available: SQLite(sqlite3), MySQL(pymysql), PostgreSQL(pg8000), IMAP(imaplib)
starting single-scheduler for "crm"...
Traceback (most recent call last):
  File "/opt/web-apps/web2py/gluon/restricted.py", line 209, in restricted
    exec ccode in environment
  File "applications/crm/models/db.py", line 47, in <module>
    db = DAL('mysql://crm:password@localhost:3307/CRM',migrate_enabled=False, fake_migrate_all=True)
  File "/opt/web-apps/web2py/gluon/dal.py", line 6595, in __new__
    raise RuntimeError, 'Cannot duplicate a Singleton'
RuntimeError: Cannot duplicate a Singleton

Massimo Di Pierro

unread,
Oct 4, 2012, 4:14:40 PM10/4/12
to web...@googlegroups.com
can you try again the latest?

Niphlod

unread,
Oct 4, 2012, 4:15:04 PM10/4/12
to web...@googlegroups.com
update to trunk, the problem is fixed. PS: Nightly build generally are stable, but a few times (like the ones with potentially hige impacts - like last changes made in DAL initialization) are, nonetheless, for testing purposes, not production ready!

Adnan Smajlovic

unread,
Oct 4, 2012, 9:03:18 PM10/4/12
to web...@googlegroups.com
Still the same problem:

Traceback (most recent call last):
  File "/opt/web-apps/web2py/gluon/scheduler.py", line 203, in executor
    _env = env(a=a,c=c,import_models=True)
  File "/opt/web-apps/web2py/gluon/shell.py", line 134, in env
    sys.exit(1)
SystemExit: 1

Is there any line of code that I can use to verify if the changes were applied?

I took a trunk source from github. Should I have done it from somewhere else?

Thanks

--
 
 
 



Massimo Di Pierro

unread,
Oct 4, 2012, 9:42:43 PM10/4/12
to web...@googlegroups.com
This error can arise in two cases:
- you are using an a recent trunk version (not the latest) which has a bug
- you are using the latest trunk (no bug) but you have a duplicated connection in your code:

db = DAL(....)
db = DAL(....)

perhaps in different model files. Previous web2py did not check and allowed you to have two connections to the same database. We found this is often source or errors. The new web2py checks and does not allow you to do it. Your would get the error you see.


On Thursday, 4 October 2012 14:42:59 UTC-5, Adi wrote:

Adnan Smajlovic

unread,
Oct 4, 2012, 10:10:53 PM10/4/12
to web...@googlegroups.com
I'm using 6 connections at the same time, 3 go to different databases on the same server, and 3 to totally different servers. Each connection has a different name

db = DAL('mysql://crm:password@localhost:3307/CRM',migrate_enabled=True)
db_us = DAL('mysql://web_US:password@localhost:3307/DataUS',migrate_enabled=False, fake_migrate_all=True)
db_ca = DAL('mysql://web_CA:password@localhost:3307/DataCA', migrate_enabled=False, fake_migrate_all=True)
...

Could something like this be a problem:
In a scheduler function, based on application that inserted a task, I assign relevant connection to another object to use?
 
if application='ca':
    db_front_end = db_ca
elif application='us':
    db_front_end = db_us

use db_front_end to retrieve data and perform scheduled task


--
 
 
 


Massimo Di Pierro

unread,
Oct 4, 2012, 10:32:25 PM10/4/12
to web...@googlegroups.com
No. this should not be a problem. This is only be a problem if any two have the same connection string.

Adnan Smajlovic

unread,
Oct 5, 2012, 12:00:52 AM10/5/12
to web...@googlegroups.com
Will run more tests in the morning on a brand new application, since I can't see anything wrong (connection string duplication) in code right now. Unfortunately, can't roll back this functionality now, so will have to figure it out quickly. Thanks for help Massimo and Niphlod.


--
 
 
 



Adi

unread,
Oct 5, 2012, 9:27:22 AM10/5/12
to web...@googlegroups.com

Created a new application from admin panel, accessing single mysql db, inserted one task that sends an email and prints a test message.

Version 2.0.9 (2012-10-04 20:17:03) dev



def test():
    db
.scheduler_task.insert(
                             status
='QUEUED',
                             application_name
='test2',
                             task_name
='send reminder email',
                             function_name
='send_msg',
                             enabled
=True,
                             start_time
= request.now,
                             next_run_time
= request.now+datetime.timedelta(minutes=2),
                             stop_time
= request.now+datetime.timedelta(days=1),
                             repeats
= 1, # run 1 times
                             period
= 120,
                             timeout
= 60, # should take less than 60 seconds
                             
)



started scheduler:
asm21:web2py209 adnan$ python web2py.py -a pass -K test2 -X

web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2012
Version 2.0.9 (2012-10-04 20:17:03) dev

Database drivers available: SQLite(sqlite3), MySQL(pymysql), PostgreSQL(pg8000), IMAP(imaplib)
please visit:
 starting scheduler for "test2"...    http://127.0.0.1:8000

use "kill -SIGTERM 831" to shutdown the web2py server
Currently running 1 scheduler processes
Processes started
ERROR:Rocket.Errors.Port8000:Socket 127.0.0.1:8000 in use by other process and it won't share.
WARNING:Rocket.Errors.Port8000:Listener started when not ready.

Traceback (most recent call last):
  File "/Users/adnan/web2py209/gluon/restricted.py", line 209, in restricted
    exec ccode in environment
  File "applications/test2/models/db.py", line 15, in <module>

    db = DAL('mysql://crm:password@localhost:3307/CRM',migrate_enabled=False, fake_migrate_all=True)
  File "/Users/adnan/web2py209/gluon/dal.py", line 6600, in __new__

    raise RuntimeError, 'Cannot duplicate a Singleton'
RuntimeError: Cannot duplicate a Singleton


traceback error in the scheduler_run table after task was run (first time):

Traceback (most recent call last):
  File "/opt/web-apps/web2py/gluon/scheduler.py", line 203, in executor
    _env = env(a=a,c=c,import_models=True)
  File "/opt/web-apps/web2py/gluon/shell.py", line 127, in env
    environment = build_environment(request, response, session)
  File "/opt/web-apps/web2py/gluon/compileapp.py", line 388, in build_environment
    t = environment['T'] = translator(request)
  File "/opt/web-apps/web2py/gluon/languages.py", line 437, in __init__
    self.set_current_languages()
  File "/opt/web-apps/web2py/gluon/languages.py", line 492, in set_current_languages
    pl_info = self.get_possible_languages_info('default')
  File "/opt/web-apps/web2py/gluon/languages.py", line 471, in get_possible_languages_info
    info = read_possible_languages(self.folder)
  File "/opt/web-apps/web2py/gluon/languages.py", line 250, in read_possible_languages
    lambda: read_possible_languages_aux(langdir))
  File "/opt/web-apps/web2py/gluon/cfs.py", line 40, in getcfs
    return filter() if callable(filter) else ''
  File "/opt/web-apps/web2py/gluon/languages.py", line 250, in <lambda>
    lambda: read_possible_languages_aux(langdir))
  File "/opt/web-apps/web2py/gluon/languages.py", line 215, in read_possible_languages_aux
    flist = oslistdir(langdir)
OSError: [Errno 2] No such file or directory: 'applications/test2/languages'

traceback error in the scheduler_run table after task was run (second time):

Traceback (most recent call last):
  File "/Users/adnan/web2py209/gluon/scheduler.py", line 203, in executor
    _env = env(a=a,c=c,import_models=True)
  File "/Users/adnan/web2py209/gluon/shell.py", line 134, in env
    sys.exit(1)
SystemExit: 1

Please let me know if there is anything else I can test to help figure out where exactly is the problem? It's a bit urgent for me to figure this out, or I have to roll back production before weekend starts :(

Adi

unread,
Oct 5, 2012, 11:21:46 AM10/5/12
to web...@googlegroups.com

for what it's worth, just want to let you know that after rolling back to "Version 2.0.9 (2012-09-13 23:51:30) stable" tasks are getting executed, marked as COMPLETED, but schedule_run is being marked as FAILED w bellow specified sys.exit(1).

anyway... this is perfect for now, and customers won't suffer :)

scheduler and workflow are two crucial things in complex applications from my prospective, so big thanks for scheduler, and can't wait to start testing workflow once it's in.

Massimo Di Pierro

unread,
Oct 5, 2012, 11:59:59 AM10/5/12
to web...@googlegroups.com
To summarize:
1) 9/13 version works
2) 10/4 version does not work as you expected

Did you try any 10/5 version?

Adnan Smajlovic

unread,
Oct 5, 2012, 5:24:06 PM10/5/12
to web...@googlegroups.com
yes, i just confirmed by running in parallel 9/13 and 10/5

results:
9/13 partially works (will execute task, mark it as "COMPLETED" in scheduler_tasks, but will mark it as "FAILED" in scheduler_run and won't update details in there...
10/5 doesn't work (RuntimeError: Cannot duplicate a Singleton)


--
 
 
 



--

Thanks,
Adnan

video: http://vimeo.com/24653283


(MySQL 5.5.27-log) rackspace_CRM_scheduler_task.jpg

Massimo Di Pierro

unread,
Oct 5, 2012, 5:58:37 PM10/5/12
to web...@googlegroups.com
Can you share some code so we ret reproduce it and fix it? We do not need all the code. Just a minimalist example that causes the same problem.

Adi

unread,
Oct 5, 2012, 6:25:49 PM10/5/12
to web...@googlegroups.com
Of course. This is what I have in a test application:

Models
======
db.py

if not request.env.web2py_runtime_gae:
   
# used tunnel
   
db_s = DAL('mysql://crm:password@localhost:3307/CRM',migrate_enabled=False, fake_migrate_all=True)
else:
   
## connect to Google BigTable (optional 'google:datastore://namespace')
    db_s
= DAL('google:datastore')
   
## store sessions and tickets there
    session
.connect(request, response, db_s = db_s)



db_scheduler.py
# coding: utf8
from scheduler import Scheduler
import datetime
import urllib, urllib2
from urlparse import urlparse
import unicodedata

def send_msg():
    email_sent
=mail.send(to='te...@email.com',
        subject
='email_subject',
        message
='message')
   
print 'sent'
   
return
   
myscheduler
= Scheduler(db_s, dict(send_msg=send_msg))



Controllers
===========
test.py
def test():
    db_s
.scheduler_task.insert(

                             status
='QUEUED',
                             application_name
='test2',

                             task_name
='send test abandoned email',

                             function_name
='send_msg',
                             enabled
=True,
                             start_time
= request.now,
                             next_run_time
= request.now+datetime.timedelta(minutes=2),
                             stop_time
= request.now+datetime.timedelta(days=1),
                             repeats
= 1, # run 1 times

                             period
= 120, # every 2m

                             timeout
= 60, # should take less than 60 seconds
                             
)



Scheduler started:
$ python web2py.py -a pass -K test2 -X > log_scheduler.log

Niphlod

unread,
Oct 5, 2012, 6:40:09 PM10/5/12
to web...@googlegroups.com
while I test it, could you please verify that when you updated web2py the scheduler_* table have been migrated too ?
it seems in all your code posted migration on that db are always turned off.
please inspect the scheduler_task table and see if it has a column named "uuid".

BTW: from scheduler import Scheduler shouldn't work unless you're messing with sys.path . the correct import would be from gluon.scheduler import Scheduler

Adi

unread,
Oct 5, 2012, 6:44:27 PM10/5/12
to web...@googlegroups.com
CREATE TABLE `scheduler_task` (
 
`id` int(11) NOT NULL AUTO_INCREMENT,
 
`application_name` varchar(255) DEFAULT NULL,
 
`task_name` varchar(255) DEFAULT NULL,
 
`group_name` varchar(255) DEFAULT NULL,
 
`status` varchar(255) DEFAULT NULL,
 
`function_name` varchar(255) DEFAULT NULL,
 
`uuid` varchar(255) DEFAULT NULL,
 
`args` longtext,
 
`vars` longtext,
 
`enabled` char(1) DEFAULT NULL,
 
`start_time` datetime DEFAULT NULL,
 
`next_run_time` datetime DEFAULT NULL,
 
`stop_time` datetime DEFAULT NULL,
 
`repeats` int(11) DEFAULT NULL,
 
`retry_failed` int(11) DEFAULT NULL,
 
`period` int(11) DEFAULT NULL,
 
`timeout` int(11) DEFAULT NULL,
 
`sync_output` int(11) DEFAULT NULL,
 
`times_run` int(11) DEFAULT NULL,
 
`times_failed` int(11) DEFAULT NULL,
 
`last_run_time` datetime DEFAULT NULL,
 
`assigned_worker_name` varchar(255) DEFAULT NULL,
 
`created_on` datetime DEFAULT NULL,
 
`modified_by` int(11) DEFAULT NULL,
 
`created_by` int(11) DEFAULT NULL,
 
`is_active` char(1) DEFAULT NULL,
 
`modified_on` datetime DEFAULT NULL,
  PRIMARY KEY
(`id`),
  UNIQUE KEY
`uuid` (`uuid`),
  KEY
`modified_by__idx` (`modified_by`),
  KEY
`created_by__idx` (`created_by`),
  CONSTRAINT
`scheduler_task_ibfk_1` FOREIGN KEY (`modified_by`) REFERENCES `auth_user` (`id`) ON DELETE CASCADE,
  CONSTRAINT
`scheduler_task_ibfk_2` FOREIGN KEY (`created_by`) REFERENCES `auth_user` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=235 DEFAULT CHARSET=utf8;

CREATE TABLE
`scheduler_run` (
 
`id` int(11) NOT NULL AUTO_INCREMENT,
 
`scheduler_task` int(11) DEFAULT NULL,
 
`status` varchar(255) DEFAULT NULL,
 
`start_time` datetime DEFAULT NULL,
 
`stop_time` datetime DEFAULT NULL,
 
`output` longtext,
 
`result` longtext,
 
`traceback` longtext,
 
`worker_name` varchar(255) DEFAULT NULL,
 
`modified_by` int(11) DEFAULT NULL,
 
`is_active` char(1) DEFAULT NULL,
 
`created_by` int(11) DEFAULT NULL,
 
`created_on` datetime DEFAULT NULL,
 
`modified_on` datetime DEFAULT NULL,
  PRIMARY KEY
(`id`),
  KEY
`scheduler_task__idx` (`scheduler_task`),
  KEY
`modified_by__idx` (`modified_by`),
  KEY
`created_by__idx` (`created_by`),
  CONSTRAINT
`scheduler_run_ibfk_1` FOREIGN KEY (`scheduler_task`) REFERENCES `scheduler_task` (`id`) ON DELETE CASCADE,
  CONSTRAINT
`scheduler_run_ibfk_2` FOREIGN KEY (`modified_by`) REFERENCES `auth_user` (`id`) ON DELETE CASCADE,
  CONSTRAINT
`scheduler_run_ibfk_3` FOREIGN KEY (`created_by`) REFERENCES `auth_user` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=373 DEFAULT CHARSET=utf8;

Niphlod

unread,
Oct 5, 2012, 6:59:04 PM10/5/12
to web...@googlegroups.com
ok, the problems seems to be related to migrate, migrate_enabled and fake_migrate_all. Can you try removing those from the db connection just to try it out ?

Adi

unread,
Oct 5, 2012, 8:17:56 PM10/5/12
to web...@googlegroups.com
sorry clicked on post too soon...

i didn't change anything related to syspath (all untouched), and i think i took the sample code from previous book... so it may as well be that i'm calling it wrongly... gtg right now. will test changes as soon as i'm back

Adi

unread,
Oct 5, 2012, 8:39:50 PM10/5/12
to web...@googlegroups.com
Yes, new application was using the existing database, so fixing broken migration fixed the problem of not executing task. Now it executed and marked status as COMPLETED. The problem with scheduler_run table stayed the same. Status: FAILED, and traceback looks as bellow:

Traceback (most recent call last):
  File "/Users/adnan/web2py209/gluon/scheduler.py", line 203, in executor
    _env = env(a=a,c=c,import_models=True)
  File "/Users/adnan/web2py209/gluon/shell.py", line 134, in env
    sys.exit(1)
SystemExit: 1



Massimo Di Pierro

unread,
Oct 7, 2012, 2:12:44 AM10/7/12
to web...@googlegroups.com
Can you please check the latest trunk and let us know if the problem has gone?

Adnan Smajlovic

unread,
Oct 7, 2012, 1:57:47 PM10/7/12
to web...@googlegroups.com
The task executed properly, marked status "COMPLETED" in scheduler_task table, but didn't insert a record into scheduler_run table. I will  create a new mysql database and let you know if the problem is there...


--
 
 
 


Niphlod

unread,
Oct 7, 2012, 2:16:38 PM10/7/12
to web...@googlegroups.com
are you aware that if your task doesn't return nothing and doesn't go into exception the new scheduler will erase the scheduler_run record ?

Adnan Smajlovic

unread,
Oct 7, 2012, 2:43:59 PM10/7/12
to web...@googlegroups.com
Oh Niphlod :) No, I wasn't aware of that...

In that case it works as expected :) I created a new db, and introduced a return value, which caused a record into into scheduler_run with status "COMPLETED", and additional details in "result" field. This is perfect...

Thank you very much for scheduler. It's an instrumental part of our application.

When do you think this will go into a nightly build, so we can test it in production?

Thanks,
Adnan

--
 
 
 

Niphlod

unread,
Oct 7, 2012, 4:52:55 PM10/7/12
to web...@googlegroups.com
It's not the scheduler code that has changed/improved, it's a major change in DAL code that needed some tune-ups. That same change in DAL needs to be tested throughly, but I think nightly builds will come soon.
Reply all
Reply to author
Forward
0 new messages