Scheduler error: Exception in thread Thread-1 and connection is closed

96 views
Skip to first unread message

kenny c

unread,
Nov 21, 2015, 8:12:05 PM11/21/15
to web2py-users
HI, I am not sure what causes this issue..

I am running Web2py with Postgresql, I am getting this exception in thread while running a scheduler. and I get InterfaceError("Connection is closed") .

I am wondering if Postgresql has timeout setting or scheduler timeout? (I made sure that timeout is long enough to handle all the jobs)

Any help would be appreciated. Thank you and please see the log below.

ERROR:web2py.scheduler.example#6580:Error retrieving status
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/www-data/web2py/gluon/scheduler.py", line 442, in run
    self.send_heartbeat(counter)
  File "/home/www-data/web2py/gluon/scheduler.py", line 1041, in send_heartbeat
    db.rollback()
  File "/home/www-data/web2py/gluon/packages/dal/pydal/base.py", line 937, in rollback
    self._adapter.rollback()
  File "/home/www-data/web2py/gluon/packages/dal/pydal/adapters/base.py", line 1332, in rollback
    return self.connection.rollback()
  File "/home/www-data/web2py/gluon/contrib/pg8000/core.py", line 1429, in rollback
    self.execute(self._cursor, "rollback", None)
  File "/home/www-data/web2py/gluon/contrib/pg8000/core.py", line 1622, in execute
    raise InterfaceError("connection is closed")
InterfaceError: connection is closed

Traceback (most recent call last):
  File "/home/www-data/web2py/gluon/shell.py", line 271, in run
    exec(python_code, _env)
  File "<string>", line 1, in <module>
  File "/home/www-data/web2py/gluon/scheduler.py", line 730, in loop
    self.wrapped_report_task(task, self.async(task))
  File "/home/www-data/web2py/gluon/scheduler.py", line 879, in wrapped_report_task
    db.rollback()
  File "/home/www-data/web2py/gluon/packages/dal/pydal/base.py", line 937, in rollback
    self._adapter.rollback()
  File "/home/www-data/web2py/gluon/packages/dal/pydal/adapters/base.py", line 1332, in rollback
    return self.connection.rollback()
  File "/home/www-data/web2py/gluon/contrib/pg8000/core.py", line 1429, in rollback
    self.execute(self._cursor, "rollback", None)
  File "/home/www-data/web2py/gluon/contrib/pg8000/core.py", line 1622, in execute
    raise InterfaceError("connection is closed")
InterfaceError: connection is closed

kenny c

unread,
Nov 21, 2015, 11:00:31 PM11/21/15
to web2py-users
I think there is timeout issue on the scheduler and I am not sure which one it is..

Seems to me that the Scheduler stops if it takes longer than 5 minutes in regardless of the value of timeout saved in the task row.

Niphlod

unread,
Nov 23, 2015, 6:11:04 PM11/23/15
to web2py-users
not to point fingers around, but can you try installing psycopg2 ? it'll override pg8000 that showed here and there some obscure bugs: pscopg2 is certainly more solid. If the same happens with psycopg2, can we see the actual function queued (or a function that replicates this behaviour) ?


BTW: send_heartbeat() shouldn't timeout that easily: it's called every 3 seconds so AT LEAST THAT connection should never be dropped.

kenny c

unread,
Nov 24, 2015, 7:15:15 AM11/24/15
to web...@googlegroups.com
Just tested with psycopg and same error.
=====
ERROR:web2py.scheduler.postit#20382:Error retrieving status
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/www-data/web2py/gluon/scheduler.py", line 442, in run
    self.send_heartbeat(counter)
  File "/home/www-data/web2py/gluon/scheduler.py", line 1041, in send_heartbeat
    db.rollback()
  File "/home/www-data/web2py/gluon/packages/dal/pydal/base.py", line 937, in rollback
    self._adapter.rollback()
  File "/home/www-data/web2py/gluon/packages/dal/pydal/adapters/base.py", line 1332, in rollback
    return self.connection.rollback()
InterfaceError: connection already closed

Traceback (most recent call last):
  File "/home/www-data/web2py/gluon/shell.py", line 271, in run
    exec(python_code, _env)
  File "<string>", line 1, in <module>
  File "/home/www-data/web2py/gluon/scheduler.py", line 730, in loop
    self.wrapped_report_task(task, self.async(task))
  File "/home/www-data/web2py/gluon/scheduler.py", line 879, in wrapped_report_task
    db.rollback()
  File "/home/www-data/web2py/gluon/packages/dal/pydal/base.py", line 937, in rollback
    self._adapter.rollback()
  File "/home/www-data/web2py/gluon/packages/dal/pydal/adapters/base.py", line 1332, in rollback
    return self.connection.rollback()
InterfaceError: connection already closed
=====

function crawls data from five RSS feed links that can have more than one page, due to this connection issue, I defined istart=0 and iend=1 to read only one page at a time

def feeddata(istart,iend):
    import feedparser
    import datetime
    import ciso8601
    import random
    for i in range(istart,iend):
        print i
        for a in feed_list:
            e = feedparser.parse(a+str(i))
            for row in e.entries:
                dtime = float(row.a)/1000.0
                d = datetime.datetime.fromtimestamp(dtime).strftime('%Y-%m-%d %H:%M:%S')
                dt_endtime = ciso8601.parse_datetime_unaware(d)
                img_link = BeautifulSoup(row.f).find('img')['src']
                if row.rx_bidcount > 0:
                    db.abpost.update_or_insert(db.abpost.post_id == row.id,
                    post_id = row.id,
                    item_id = str(row.id).split('/')[5].split('?')[0],
                    published = row.published,
                    a = row.a,
                    b = row.b,
                    c = row.c,
                    d = float(row.d)/100,
                    e = row.e,
                    f = f,
                    g = row.g,
                    h = row.h,
                    img_link = img_link,
                    upvote = random.randint(10, 1000),
                    downvote = random.randint(1, 5),
                    title = row.title,
                    title_detail = row.title_detail)
                    db.commit()
    return "Done"

Niphlod

unread,
Nov 24, 2015, 9:52:30 AM11/24/15
to web2py-users
please remove any print statement .... and .... why are you importing modules inside the function ?

kenny c

unread,
Dec 14, 2015, 1:45:25 PM12/14/15
to web...@googlegroups.com
Sorry for late reply and thank you so much, Niphlod! 

I am still playing with Scheduler, so I didn't know how I should use it. 

One more question,

I am testing the scheduler to write a csv file from reading JSON formatted data,

The scheduler stops at one point, I get "STOPPED" on the terminal

I am using csvkit library and here's the code.

def json2csv(datatype):
turl = ''
filetype=''
filetype=str('cdef-'+datatype+'.csv')
inputfile = urllib2.urlopen(turl)
outputfile = convert.json2csv(inputfile,key='value')
myFile = open('/home/www-data/web2py/applications/cdef/static/cdef/'+filetype,'w')
myFile.write(outputfile)
myFile.close()
db.data_dw.update_or_insert(db.data_dw.datatype==datatype,datatype=datatype,filename=filetype,fundtype='cdef',uploaddate=request.now)
db.commit()
return True

kenny c

unread,
Dec 14, 2015, 2:16:52 PM12/14/15
to web2py-users
I think it has to do with csvkit since loading big file creates this issue.

Thank you as always! Let's go! Web2PY!
Reply all
Reply to author
Forward
0 new messages