cx_Oracle - Django ORM - Reference Count Increase

[Anurag Chourasia]

All,

I have a Long Running Python Process that uses Django ORM against Oracle database.

The size of the process keeps on increasing steadily. 

I was profiling this process using mem_top and i find that the reference count of one particular data type <list> increases continuously with iterations.

Datatype is <type 'list'> [{u'time': u'0.004', u'sql': u'QUERY = u\'SELECT "RANGE_STATUS"."I

References increased from 534 to 53295

This list seems to have almost all the queries that were executed using the ORM

Does this mean that there is a memory leak in the cx_Oracle module or somewhere in Django ORM?

Here is the memtop result between two iterations 10 minutes apart.

WARNING : 27/12/2014 05:45:46 PM : 

refs:
9500 <type 'dict'> {'TAPE_DRIVE_FORMAT': -1610612736, 'SLE_ERROR': 1, 'IMAGE_REL_I386_SEC
8410 <type 'list'> [('200', '343045', 1321), ('200', '343046', 1322), ('200', '343047', 1
1578 <type 'dict'> {'logging.atexit': None, 'django.core.files.errno': None, 'django.test
1567 <type 'list'> ['mem_top', 'mem_top', 'datetime', 'datetime', 'sys', 'string', 'os', 
688 <type 'dict'> {'FILE_SYSTEM_ATTR': 2, 'GetDriveTypeW': <built-in function GetDriveTy
688 <type 'dict'> {'FILE_SYSTEM_ATTR': 2, 'GetDiskFreeSpaceEx': <built-in function GetDi
534 <type 'list'> [{u'time': u'0.004', u'sql': u'QUERY = u\'SELECT "RANGE_STATUS"."I
510 <type 'dict'> {'GetDiskFreeSpaceEx': <built-in function GetDiskFreeSpaceEx>, 'SetThr
510 <type 'dict'> {'GetDiskFreeSpaceEx': <built-in function GetDiskFreeSpaceEx>, 'SetThr
370 <type 'list'> [<weakref at 01CF5030; to 'type' at 1E228030 (type)>, <weakref at 01CF
types:
8625 <type 'function'>
3778 <type 'tuple'>
3128 <type 'dict'>
1672 <type 'list'>
1661 <type 'cell'>
1440 <type 'weakref'>
1351 <type 'wrapper_descriptor'>
1103 <type 'builtin_function_or_method'>
888 <type 'type'>
734 <type 'getset_descriptor'>

WARNING : 27/12/2014 05:54:37 PM : 

refs:
53295 <type 'list'> [{u'time': u'0.004', u'sql': u'QUERY = u\'SELECT "RANGE_STATUS"."I
9500 <type 'dict'> {'TAPE_DRIVE_FORMAT': -1610612736, 'SLE_ERROR': 1, 'IMAGE_REL_I386_SEC
8410 <type 'list'> [('200', '343045', 1321), ('200', '343046', 1322), ('200', '343047', 1
1578 <type 'dict'> {'logging.atexit': None, 'django.core.files.errno': None, 'django.test
1567 <type 'list'> ['mem_top', 'mem_top', 'datetime', 'datetime', 'sys', 'string', 'os', 
749 <type 'list'> ['A. HISTORY OF THE SOFTWARE', '==========================', '', 'Pyth
688 <type 'dict'> {'FILE_SYSTEM_ATTR': 2, 'GetDriveTypeW': <built-in function GetDriveTy
688 <type 'dict'> {'FILE_SYSTEM_ATTR': 2, 'GetDiskFreeSpaceEx': <built-in function GetDi
510 <type 'dict'> {'GetDiskFreeSpaceEx': <built-in function GetDiskFreeSpaceEx>, 'SetThr
510 <type 'dict'> {'GetDiskFreeSpaceEx': <built-in function GetDiskFreeSpaceEx>, 'SetThr
types:
8625 <type 'function'>
3778 <type 'tuple'>
3130 <type 'dict'>
1675 <type 'list'>
1661 <type 'cell'>
1440 <type 'weakref'>
1351 <type 'wrapper_descriptor'>
1103 <type 'builtin_function_or_method'>
888 <type 'type'>
734 <type 'getset_descriptor'>

Please guide.

Regards,

Guddu

[Michiel Overtoom]

On Dec 27, 2014, at 22:23, Anurag Chourasia wrote:

> I have a Long Running Python Process that uses Django ORM against Oracle database.
> The size of the process keeps on increasing steadily.

Are you running in Debug mode?

http://stackoverflow.com/questions/1339293/python-memory-leak-debugging
http://www.dimagi.com/django-orm-memory-leaks-in-debug-mode-73877/

Greetings,

--
"You can't actually make computers run faster, you can only make them do less." - RiderOfGiraffes

[Guddu]

Thanks Michiel....That was the problem indeed. I had Debug=True. Changing it to False solved the issue for me.

Regards,

Guddu