Issue 146 in couchdb-python: Up-to-dated view server - CouchDB-Python

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Issue 146 in couchdb-python: Up-to-dated view server
Received-SPF: pass (google.com: domain of 3Yc2NTQ4OAAomy4mrnl-z83ryxqyyqvomyno....@codesite.bounces.google.com designates 209.85.218.75 as permitted sender) client-ip=209.85.218.75;
MIME-Version: 1.0
Reply-To: codesite-nore...@google.com
References: <16-12560507179679177234-16084185522831819867-couchdb-python=googlecode.com@googlecode.com>
 <0-12560507179679177234-16084185522831819867-couchdb-python=googlecode.com@googlecode.com>
In-Reply-To: <16-12560507179679177234-16084185522831819867-couchdb-python=googlecode.com@googlecode.com>
Message-ID: <17-12560507179679177234-16084185522831819867-couchdb-python=googlecode.com@googlecode.com>
Date: Sat, 26 Mar 2011 11:26:25 +0000
Subject: Re: Issue 146 in couchdb-python: Up-to-dated view server
From: couchdb-pyt...@googlecode.com
To: couchdb-python@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes


Comment #17 on issue 146 by extempor...@gmail.com: Up-to-dated view server
http://code.google.com/p/couchdb-python/issues/detail?id=146

Yeah, I'll have to look into that 'require' thing. On first glance, it  
looks like couchjs is doing a request to the design doc for the  
dependencies.

Right, as said in a previous post, in order for the above-the-function  
option to work, without the use of a closure, you (the view server function  
compiler) would have to check every key in the locals dictionary that exec  
generates to see if it has a __module__ attribute, and if that attribute  
has the value of None. The only backwards compatible requirement we need is  
that there is only one callable object (usually a function) that has  
__module__ set to None (since non-imported local functions/classes will  
have __module__ of None).

>>> code = """
... from datetime import datetime
...
... class OldStyleClass:
...     pass
...
... class NewStyleClass(object):
...     pass
...
... y = 17
...
... def test():
...     return 5
... """
>>> locals = {}
>>> exec code in {}, locals
>>> locals
{'y': 17, 'test': <function test at 0x7f7e25f10320>, 'NewStyleClass':  
<class 'NewStyleClass'>, 'OldStyleClass': <class __builtin__.OldStyleClass  
at 0x7f7e25f196b0>, 'datetime': <type 'datetime.datetime'>}
>>> for key in locals:
...   if callable(locals[key]):
...     if locals[key].__module__:
...       print key, "is *not* a candidate, since it's imported from",  
locals[key].__module__
...     else:
...       print key, "is a candidate (hopefully the only one, or we'll have  
to error out)"
...   else:
...       print key, "isn't even callable, so we don't care about it"
...
y isn't even callable, so we don't care about it
test is a candidate (hopefully the only one, or we'll have to error out)
NewStyleClass is *not* a candidate, since it's imported from __builtin__
OldStyleClass is *not* a candidate, since it's imported from __builtin__
datetime is *not* a candidate, since it's imported from datetime

Huh, so apparently class definitions inside of an exec will be associated  
with the __builtin__ module, so we'd have to check for that, as well. But  
in general, it's easy to do a backwards-compatible check for non-imported  
callables.

Oh, perhaps the answer to the module distribution problem is to put a  
custom import mechanism that checks for those modules as attachments to the  
view/list/show functions design doc before checking the normal on-disk  
module path. Couch's _changes API would need to be monitored for design doc  
changes by couchpy too, so that couchpy can know when it needs to reload  
modules. If this were achievable, you could bundle your modules in the  
design doc itself (regular zip files and eggs could be supported).

The best way to get a good system in place for this is not to work around  
Couch's API, but instead to work directly with the Apache Couch community  
to support everything we're talking about, since none of it violates the  
side-effect-free requirements of couch if dependency checking can be moved  
into couch. This wouldn't mean that couch would have to understand any  
programming language, but would be able to handle changes to certain  
special design doc keys. For example, couch could *hypothetically* do:

{
   "_id": "_design/app",
   "lib": {
     "calc": "def something_statistical(a,b,c,d): return (a,b,c,d)",
     "image": "#some-package v1.3.2",
     "chart": "#other-package v4.1.7",
   },
   "depends": {
     "views.test": ["lib/calc"],
     "shows.graph": ["lib/chart"],
     "lib.chart": ["lib/image", "_attachments/something_local.egg"]
   },
   "views": {
     "test": {
       "map": "def fun(doc): yield doc['_id'],  
calc.something_statistical(*[doc.get(k) for k in 'abcd'])"
     }
   }
}

Once again, this doesn't exist in couch, but if it were implemented, couch  
would only need to know how to interpret the "depends" key. If a string  
in "lib" changes (couch doesn't need to know or care what the contents of  
that string mean), then everything that depends on it needs to get updated,  
just like it reindexes views when the view function strings are changed. In  
the case of list or shows, this would mean setting a new Etag that  
invalidates client-cached versions of the previous show/list results. Couch  
would also need to send the dependency to the view server when it's needed,  
in the form of some kind of addlib command. couchpy itself could ignore the  
# version stubs, since those would just be there to provide an easy upgrade  
path for libraries. Or it could compare the version shown there to the  
version of the module it imports, and update the design doc if a new  
version is found on the module path. Dependencies starting  
with "_attachments" could be handled specially by couch.
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
