Long-running process using Django ORM maxes out psql connections

[Paco de Kumite]

Hello.  I have a standalone script that is long-running (i.e is supposed to run in a loop forever).  This script uses the Django ORM for work on a database.  For example, every two seconds it does a MyModel.objects.all().

Works awesome, but after a while I get

OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections

The db in use is postgresql with psycopg2 backend.  If I watch the number of connections in psql I see the number go up and up and up.  So it's like the above query creates a NEW connection every time through the loop.  I read that in django connections get closed at the end of  "request", but since this is a long running process with no request the signal that causes the connection to close doesn't ever get triggered and the connections never close.

Ideally I think I would want to create a single connection for the process and keep it open forever (or at least a long time) and somehow "pass" that connection to the ORM to use, rather than opening a new connection for every query.

There is also connection pooling http://www.craigkerstiens.com/2013/03/07/Fixing-django-db-connections/, but now that I got this error I remember that I had this same problem in the past with a long-running script that uses django orm.  In that case I tried using connection pooling, but it didn't work.  In the end I migrated all the db stuff to use pymongo and handled connections manually (which ended up working like a charm).  I'd like to avoid this route this time.

Any insight on how to handle this would be great.

[Paco de Kumite]

This may not be the cleanest, but it works:

First make a decorator, like this:

import django.db

from functools import wraps

def close_dbconn(func):

@wraps(func)

def close_dbconn_func(*args, **kwargs):

func(*args,**kwargs)

django.db.close_old_connections()

return close_dbconn_func

Then any function in the script that makes db queries needs to be decorated with @close_dbconn