Issue 205 in couchdb-python: Having a well-defined way to reset the connection pool would be useful

[couchdb...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 205 by wickedg...@gmail.com: Having a well-defined way to reset
the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

What steps will reproduce the problem?
1. connect to couchdb
2. fork() the python process
3. try to use the db in the child process

What is the expected output? What do you see instead?

The child gets exceptions like:

File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
1003, in rows
self._fetch()
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
990, in _fetch
data = self.view._exec(self.options)
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
880, in _exec
_, _, data = self.resource.get_json(**self._encode_options(options))
File "/usr/local/lib/python2.7/dist-packages/couchdb/http.py", line 394,
in get_json
if 'application/json' in headers.get('content-type'):
TypeError: argument of type 'NoneType' is not iterable

and

File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
1003, in rows
self._fetch()
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
990, in _fetch
data = self.view._exec(self.options)
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
878, in _exec
**self._encode_options(options))
File "/usr/local/lib/python2.7/dist-packages/couchdb/http.py", line 401,
in post_json
data = json.decode(data.read())
File "/usr/local/lib/python2.7/dist-packages/couchdb/http.py", line 94,
in read
bytes = self.resp.read(size)
File "/usr/lib/python2.7/httplib.py", line 541, in read
return self._read_chunked(amt)
File "/usr/lib/python2.7/httplib.py", line 586, in _read_chunked
raise IncompleteRead(''.join(value))
httplib.IncompleteRead: IncompleteRead(1258 bytes read)

What version of the product are you using? On what operating system?

v0.8 on Ubuntu, python 2.7

Please provide any additional information below.

I'm able to work around the problem by including code like the following
under 0.8:

with my_db.resource.session.lock:
my_db.resource.session.conns = {}

Trunk has a connection pool object, so it looks like that would now be:

with my_db.resource.session.connection_pool.lock:
my_db.resource.session.connection_pool.conns = {}

Having this encapsulated in a public API function would be useful for
us. :)

Thanks!

[couchdb...@googlecode.com]

Comment #1 on issue 205 by djc.ochtman: Having a well-defined way to reset

the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

Could we make the connection pool thread-local, or something like that?
I.e. something that does the right thing without needing the user's
cooperation.

[couchdb...@googlecode.com]

Comment #2 on issue 205 by wickedg...@gmail.com: Having a well-defined way

to reset the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

That would be great, if something like that would work. Threading.local
won't do the trick by itself, though:

Python 2.7.2 (v2.7.2:8527427914a2, Jun 11 2011, 15:22:34)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import threading
>>> import os
>>> def f():
... l = threading.local()
... l.foo = 'bar'
... pid = os.fork()
... if pid == 0:
... print dir(l)
...
>>> f()
>>> ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'foo']

Namespacing the connections by os.getpid() would probably work for the fork
case, though. Something along the lines of:

# Session
def __init__(self, ...):
...
self._connectionPool_dict = {}
...

@property
def connection_pool(self):
if os.getpid() not in self._connectionPool_dict:
self._connectionPool_dict.clear()
self._connectionPool_dict[os.getpid()] = ConnectionPool(...)
return self._connectionPool_dict[os.getpid()]

Not sure what the intended behavior is WRT threading, but that could be
added to the mix too if desired.

[couchdb...@googlecode.com]

Comment #3 on issue 205 by matt.goo...@gmail.com: Having a well-defined way

to reset the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

I have no objection to extending the connection pool - that's actually one
of the reasons I extracted the code to the ConnectionPool class. However,
isn't this just the common problem of forking a process with open file
handles? The solution is generally to fork first and open files later.

So, in this particular case I think it's best to fork the process and
create a new couchdb.Server or couchdb.Database instance in the child
process for it to use.

[couchdb...@googlecode.com]

Comment #4 on issue 205 by wickedg...@gmail.com: Having a well-defined way

to reset the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

That approach presumes a lot about the application design. For my case,
the parts doing the forking and the parts talking to couchdb are pretty
loosely coupled, and broadcasting "hey, we just forked; reset all of your
connections please" wouldn't be very clean (as attested to by the hack that
I put in place to work around this issue for the time being). I can't not
connect to couchdb in the parent, because the information about what/when
to fork is contained there.

I was surprised by the current couchdb-python behavior because I don't
typically lump open files and http requests into the same stateful bucket -
keep-alives seem like an implementation detail, in this case. Not having
customers have to be aware of and account for that would be nice.

[couchdb...@googlecode.com]

Updates:
Owner: wickedg...@gmail.com
Labels: Milestone-0.9

Comment #5 on issue 205 by wickedg...@gmail.com: Having a well-defined way

to reset the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

(No comment was entered for this change.)

[couchdb...@googlecode.com]

Comment #6 on issue 205 by djc.ochtman: Having a well-defined way to reset

the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

Any progress on this?

[couchdb...@googlecode.com]

Comment #7 on issue 205 by kxepal: Having a well-defined way to reset the

connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

It's easy to fix, but hard to test due to there is one big god method
Session.request. Is it allowed to apply additional refactoring during fix?

[couchdb...@googlecode.com]

Comment #8 on issue 205 by djc.ochtman: Having a well-defined way to reset

the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

Please do the refactoring in a separate patch first, then the fix after
that. :)

[couchdb...@googlecode.com]

Comment #9 on issue 205 by kxepal: Having a well-defined way to reset the

connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

Got it, thanks!

[couchdb...@googlecode.com]

Comment #10 on issue 205 by wickedg...@gmail.com: Having a well-defined way

to reset the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

I've been trying to reproduce a minimal test case for my original problem
(forking), but haven't had any success in doing so against trunk. I need
to spin up a virtual env and try against 0.8; it's possible that the
changes between 0.8 and now have fixed the problem already. That would be
nice, but I suspect that I'm just doing a poor job of simulating the
conditions of my original problem.

[couchdb...@googlecode.com]

Comment #11 on issue 205 by wickedg...@gmail.com: Having a well-defined way

to reset the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

We've worked around this issue in our application code. I would like to
remove the Milestone-0.9 label; any objections?

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[couchdb...@googlecode.com]

Comment #12 on issue 205 by wickedg...@gmail.com: Having a well-defined way

[couchdb...@googlecode.com]

Updates:
Labels: -Milestone-0.9

Comment #13 on issue 205 by djc.ochtman: Having a well-defined way to reset

the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

Consider it done. ;)

[couchdb...@googlecode.com]

Comment #14 on issue 205 by panny.w...@gmail.com: Having a well-defined way

to reset the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

Would you please shed some light on how to work around this issue as we are
encountering similar issue when accessing the designed view.

Two similar exceptions are as follows:

Traceback (most recent call last):

File "/opt/prophetstor/federator/ussfed/common/database/couchdb/couchdb_store.py",
line 216, in find_by_name
if res and len(res) > 0:
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
1160, in __len__
return len(self.rows)
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
1176, in rows

self._fetch()
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line

1163, in _fetch

data = self.view._exec(self.options)
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line

1029, in _exec
_, _, data = _call_viewlike(self.resource, options)
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
1088, in _call_viewlike
return resource.get_json(**_encode_view_options(options))
File "/usr/local/lib/python2.7/dist-packages/couchdb/http.py", line 523,
in get_json
return self._request_json('GET', path, headers=headers, **params)
File "/usr/local/lib/python2.7/dist-packages/couchdb/http.py", line 547,
in _request_json

if 'application/json' in headers.get('content-type'):
TypeError: argument of type 'NoneType' is not iterable


and

Traceback (most recent call last):

File "/opt/prophetstor/federator/ussfed/common/database/couchdb/couchdb_store.py",
line 216, in find_by_name
if res and len(res) > 0:
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
1160, in __len__
return len(self.rows)
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
1176, in rows

self._fetch()
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line

1163, in _fetch

data = self.view._exec(self.options)
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line

1029, in _exec
_, _, data = _call_viewlike(self.resource, options)
File "/usr/local/lib/python2.7/dist-packages/couchdb/client.py", line
1088, in _call_viewlike
return resource.get_json(**_encode_view_options(options))
File "/usr/local/lib/python2.7/dist-packages/couchdb/http.py", line 523,
in get_json
return self._request_json('GET', path, headers=headers, **params)
File "/usr/local/lib/python2.7/dist-packages/couchdb/http.py", line 548,
in _request_json
data = json.decode(data.read())
File "/usr/local/lib/python2.7/dist-packages/couchdb/http.py", line 175,

in read
bytes = self.resp.read(size)

File "/usr/lib/python2.7/httplib.py", line 543, in read
return self._read_chunked(amt)
File "/usr/lib/python2.7/httplib.py", line 597, in _read_chunked
raise IncompleteRead(''.join(value))
IncompleteRead: IncompleteRead(0 bytes read)

[couchdb...@googlecode.com]

Comment #15 on issue 205 by panny.w...@gmail.com: Having a well-defined way

to reset the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

Forget to mention that we are using version 0.8

[couchdb...@googlecode.com]

Comment #16 on issue 205 by djc.ochtman: Having a well-defined way to reset

the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

You should start by upgrading to 0.9.

[couchdb...@googlecode.com]

Comment #17 on issue 205 by wickedg...@gmail.com: Having a well-defined way

to reset the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

We use the following properties:

@property
def db(self):
if self._db_pid != os.getpid():
self.db = couchdb.Database(self.url)

return self._db

@db.setter
def db(self, value):
self._db_pid = os.getpid()
self._db = value

I don't recall if we put this code in place before or after we switched to
0.9. I suspect it was before, but it was long enough ago that I can't say
for certain.

[couchdb...@googlecode.com]

Comment #18 on issue 205 by djc.ochtman: Having a well-defined way to reset

the connection pool would be useful
http://code.google.com/p/couchdb-python/issues/detail?id=205

This issue has been migrated to GitHub. Please continue discussion here:

https://github.com/djc/couchdb-python/issues/205