cannot allocate memory for output buffer ERROR
kerby2000
That is me again.
I've got a task to store big binary chunks of data into Postgres
database.
I'm using your ocpgdb wrapper and now I'm getting out of memory error.
(see code below)
If I decrease number of connections to 2 ( connectionsNum = 2) my
test runs fine.
As soon as I increase it to 5 or 6. I'm getting out of memory error.
Is there any way to free memory for each connection at the end of the
loop?
Any help is appreciated.
Kind regards,
Sergey
import ocpgdb
import os
import datetime
import time
memCheck = "ps aux | grep '%s' | grep -v grep | awk '{ print $5 }'"%
(os.path.split(__file__)[1])
print "> memory at process start"
os.system(memCheck)
index = 0
connectionsNum = 6
connections = [ ocpgdb.connect( database = "datapool", user =
"datapool", password = "datapool", host = "192.168.0.10", port=5432)
for i in range(connectionsNum) ]
data = "".zfill(1024*1024*300)
data = ocpgdb.bytea(data)
while True:
print "\nbeginning iteration #%d"%(index+1)
print "> memory at iteration start"
os.system(memCheck)
db = connections[index%connectionsNum]
c=db.cursor()
sql = "INSERT INTO data"\
"( id, data ) "\
"VALUES ( '%s', %%s );" %
(datetime.datetime.utcnow(),datetime.datetime.utcnow())
c.execute(sql, data)
index += 1
c.close()
db.commit()
print "> memory at iteration end"
os.system(memCheck)
The output:
> memory at process start
11012
beginning iteration #1
> memory at iteration start
318360
> memory at iteration end
842652
beginning iteration #2
> memory at iteration start
842652
> memory at iteration end
1366944
beginning iteration #3
> memory at iteration start
1366944
> memory at iteration end
1891236
beginning iteration #4
> memory at iteration start
1891236
> memory at iteration end
2415528
beginning iteration #5
> memory at iteration start
2415528
Traceback (most recent call last):
File "/home/user/test.py", line 100, in ?
c.execute(sql, data)
File "/usr/lib/python2.4/site-packages/ocpgdb/connection.py", line
84, in execute
result = self._execute(cmd, args)
File "/usr/lib/python2.4/site-packages/ocpgdb/connection.py", line
65, in _execute
return self.connection._execute(cmd, args)
File "/usr/lib/python2.4/site-packages/ocpgdb/connection.py", line
225, in _execute
result = PgConnection.execute(self, cmd, args)
oclibpq.OperationalError: cannot allocate memory for output buffer
Andrew McNamara
>database.
Databases, in general, are pretty poor at storing big blobs of binary
data, although Postgres is no worse than the rest in my experience. As
a comparison, on my desktop machine, writing (and syncing) 300MB to a
(consumer-grade) local disk takes a few seconds, but getting Postgres
to write it takes several minutes.
>I'm using your ocpgdb wrapper and now I'm getting out of memory error.
>(see code below)
>
>If I decrease number of connections to 2 ( connectionsNum = 2) my
>test runs fine.
>As soon as I increase it to 5 or 6. I'm getting out of memory error.
>Is there any way to free memory for each connection at the end of the
>loop?
The increasing memory use does suggest a memory leak of some sort.
Normally a (python) object will be released once all references to it
are dropped unless there is a cycle. The cyclic gc runs every 700(?) VM
instructions, but you can run it on demand with gc.collect(), although
this will only help if you have cycles in your reference graphs - it
makes no difference here if I add a call to gc.collect() to your script.
It's also possible there is a memory leak in oclibpq, or that libpq itself
is deliberately keeping buffers attached to the connection object. My bet
is on libpq buffers... yep, just tried your script again, this keeping
multiple connections, but closing them and re-opening them after each
use. Memory consumption is now stable.
--
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/