Processing input with CouchApps
Jason Davies
I've been thinking about how to process input with CouchApps. By this
I mean processing of request data that may/may not be used to update/
create documents in CouchDB.
We already have in-built methods for processing CouchDB output i.e.
_list and _show, essentially converting JSON documents and views into
various formats. I think the missing piece is some way to process
input requests from various formats to potentially be saved in
CouchDB. For example, I would like to be able to write various API
endpoints that receive name/value pairs as input from other Web
services, and transform these into a JSON document that is saved in
CouchDB. Real-world examples include processing callbacks from Google
Checkout or Paypal payment gateways (i.e. receiving postbacks of XML
or name/value pairs).
I realise I can already accomplish this with custom _external
processes, but I'd want to be able to do this for fully standalone
CouchApps that can be replicated to and fro.
I understand the now deprecated _action stuff would have supported
this, and I'm wondering if we can discuss adding something like
_action, perhaps a more limited version (as I understand it the reason
it was dropped was because it was more than we want to support right
now - but perhaps a carefully designed version would be kosher?)
Thanks,
Jason
Chris Anderson
>
> I understand the now deprecated _action stuff would have supported
> this, and I'm wondering if we can discuss adding something like
> _action, perhaps a more limited version (as I understand it the reason
> it was dropped was because it was more than we want to support right
> now - but perhaps a carefully designed version would be kosher?)
>
Let's push on this a bit and see where it can go.
The two main reasons we pulled back from action.js are:
1) the security implications of allowing CouchDB to make
behind-the-scenes HTTP requests are thorny (although not
insurmountable). I for one would like to see the possibility of a full
blown feed-reader CouchApp that doesn't depend on other software.
2) once you've got endpoints which can query Couch via http, there's
no way to keep them RESTful. That is, they wouldn't be cacheable
without intense developer effort. Also, it's possible to mix rendering
logic with db-update logic. The idea of an app making recursive HTTP
requests to itself is scary, especially when you consider that each of
those requests might have to go through a hashing proxy, in the case
of a cluster. (It's nothing a Rails app backed by Couch wouldn't do,
but do we really want a database to act like this?)
My opposition on these two fronts is not principled, just pragmatic.
Perhaps once we have the show and list APIs solid and documented (and
understood by developers) we can add a no-holds barred action function
(essentially _external married to design-docs in a Couch-sanctioned
way.)
> Real-world examples include processing callbacks from Google
> Checkout or Paypal payment gateways (i.e. receiving postbacks of XML
> or name/value pairs).
In the meantime, would an endpoint that accepts non-JSON POST
requests, and wraps them in a JSON envelope for later processing work?
Eg, you'd post some XML (or whatever) to:
POST /db/_raw_doc
and it would create a document like:
{
"_id" : "some uuid",
"_rev" : "2487612",
"raw" : "the xml or whatever that is the raw post body"
}
You'd pick this up for processing in the standard CouchDB way (through
views on docs as state-machines or via the update notifier) so there'd
be no processing within the update request, which is probably a good
thing.
Chris
--
Chris Anderson
http://jchris.mfdz.com
Jason Davies
>
> > I understand the now deprecated _action stuff would have supported
> > this, and I'm wondering if we can discuss adding something like
> > _action, perhaps a more limited version (as I understand it the reason
> > it was dropped was because it was more than we want to support right
> > now - but perhaps a carefully designed version would be kosher?)
>
> Let's push on this a bit and see where it can go.
>
> The two main reasons we pulled back from action.js are:
>
> 1) the security implications of allowing CouchDB to make
> behind-the-scenes HTTP requests are thorny (although not
> insurmountable). I for one would like to see the possibility of a full
> blown feed-reader CouchApp that doesn't depend on other software.
because someone could give me a malicious CouchApp, or is there some
other angle I'm missing?
I imagine a full-blown feed-reader could be done in CouchApp as long
as all the feed requests are done client-side, or are you thinking of
a multi-user feed-reader that caches feeds for users?
> 2) once you've got endpoints which can query Couch via http, there's
> no way to keep them RESTful. That is, they wouldn't be cacheable
> without intense developer effort. Also, it's possible to mix rendering
> logic with db-update logic. The idea of an app making recursive HTTP
> requests to itself is scary, especially when you consider that each of
> those requests might have to go through a hashing proxy, in the case
> of a cluster. (It's nothing a Rails app backed by Couch wouldn't do,
> but do we really want a database to act like this?)
kind of internal flag? One of the great things about CouchDB is, as
you say, its RESTful nature and I wouldn't want _action to be unwieldy
and dangerous.
> My opposition on these two fronts is not principled, just pragmatic.
> Perhaps once we have the show and list APIs solid and documented (and
> understood by developers) we can add a no-holds barred action function
> (essentially _external married to design-docs in a Couch-sanctioned
> way.)
most problems simply by writing an _external script, but then it feels
like a hack because it can't be replicated as part of my design doc.
But like you say in your reasoning above, you want to be very careful
about letting this happen, I wouldn't want to give people the ability
to hose my CouchDB instance just by even unintentionally writing a bad
_action script. So, I'd agree about putting this off for a bit until
more thought is put into making it safe. Perhaps forcing people to
accept some kind of disclaimer before the _action script is enabled
would work!
> > Real-world examples include processing callbacks from Google
> > Checkout or Paypal payment gateways (i.e. receiving postbacks of XML
> > or name/value pairs).
>
> In the meantime, would an endpoint that accepts non-JSON POST
> requests, and wraps them in a JSON envelope for later processing work?
> Eg, you'd post some XML (or whatever) to:
>
> POST /db/_raw_doc
>
> and it would create a document like:
>
> {
> "_id" : "some uuid",
> "_rev" : "2487612",
> "raw" : "the xml or whatever that is the raw post body"
>
> }
>
> You'd pick this up for processing in the standard CouchDB way (through
> views on docs as state-machines or via the update notifier) so there'd
> be no processing within the update request, which is probably a good
> thing.
particular response (the former expects XML in the response, not sure
about the latter), so POSTing data for later processing without a
proper response wouldn't really work here as the processing would need
to be done immediately.
Perhaps it could work, though, if there was a way to write a custom
_post or _update handler, similar to _show, just that it takes a
request, does some processing, then returns a response body along with
a JSON doc to be inserted/updated into the database? The JSON doc
could then be processed later by a db-update-notification script if
necessary, perhaps if it needed to be broken into multiple docs, etc.
Of course this last point can't be replicated, but I imagine it's not
that common a case. It would have to pass through the usual security
validation stuff, so security shouldn't be a concern.
Thanks,
Jason
Chris Anderson
> So, I'd agree about putting this off for a bit until
> more thought is put into making it safe. Perhaps forcing people to
> accept some kind of disclaimer before the _action script is enabled
> would work!
Part of my interest in keeping _action out for a time, is that, it's
the "easy" way to do things that might more properly be done using
other means. If _action is available, then it become the answer to
everything and prevents us from thinking more about better ways to
accomplish certain tasks.
> Perhaps it could work, though, if there was a way to write a custom
> _post or _update handler, similar to _show, just that it takes a
> request, does some processing, then returns a response body along with
> a JSON doc to be inserted/updated into the database?
Damien's talked about optional functions that can be run at insert
time (like validations, only with side-effects on the document
itself.). Obviously you'd want to restrict those functions to the
updates that need them (unlike validations which run on every update).
Which starts to sound a lot like what you describe.
I think the one hang we might have, is that the _update function would
not be able to know if the update fails due to rev conflict or
validation error. If the response to the external request can be
crafted based only on the well-formed-ness of the request, then it
will be fine. If it depends on local state as well, it gets to be more
complex.
We're really getting somewhere!
Jason Davies
> > Perhaps it could work, though, if there was a way to write a custom
> > _post or _update handler, similar to _show, just that it takes a
> > request, does some processing, then returns a response body along with
> > a JSON doc to be inserted/updated into the database?
>
> Damien's talked about optional functions that can be run at insert
> time (like validations, only with side-effects on the document
> itself.). Obviously you'd want to restrict those functions to the
> updates that need them (unlike validations which run on every update).
> Which starts to sound a lot like what you describe.
they will only accept JSON? It seems that the core CouchDB stuff is
all about JSON, and the _list/_show stuff is more about transforming
into other formats. With some kind of _update function we could then
support transforming *back* to JSON from those other formats.
> I think the one hang we might have, is that the _update function would
> not be able to know if the update fails due to rev conflict or
> validation error. If the response to the external request can be
> crafted based only on the well-formed-ness of the request, then it
> will be fine. If it depends on local state as well, it gets to be more
> complex.
map/reduce views. We could have process_request(req, doc) and
process_response(resp, doc), for handling the update request and the
response (which may be a failure status due to update conflict, etc.)
Would that work?
Jason
Chris Anderson
>
> On Feb 28, 7:21 pm, Chris Anderson <jch...@apache.org> wrote:
>> On Sat, Feb 28, 2009 at 11:05 AM, Jason Davies <jason.dav...@gmail.com> wrote:
>> > Perhaps it could work, though, if there was a way to write a custom
>> > _post or _update handler, similar to _show, just that it takes a
>> > request, does some processing, then returns a response body along with
>> > a JSON doc to be inserted/updated into the database?
>>
>> Damien's talked about optional functions that can be run at insert
>> time (like validations, only with side-effects on the document
>> itself.). Obviously you'd want to restrict those functions to the
>> updates that need them (unlike validations which run on every update).
>> Which starts to sound a lot like what you describe.
>
> Yeah, the only caveat to these optional functions is that I'm guessing
> they will only accept JSON? It seems that the core CouchDB stuff is
> all about JSON, and the _list/_show stuff is more about transforming
> into other formats. With some kind of _update function we could then
> support transforming *back* to JSON from those other formats.
I think the same API could process both JSON documents as well as
arbitrary updates. The function signature would be like
function(req, user_ctx) {
return doc;
}
The only thing harder in a generalized update function is the ability
to pass in the previous rev of the doc, if it exists. I think we could
work that in (and I'm not even sure it's appropriate.)
> I guess we could split _update into two functions, in a similar way to
> map/reduce views. We could have process_request(req, doc) and
> process_response(resp, doc), for handling the update request and the
> response (which may be a failure status due to update conflict, etc.)
> Would that work?
I think that could work - do we care about maintaining state between
the two functions (that's the only trouble I see, is if you want some
state to be shared between req and resp that is not part of the doc --
maybe we discourage that, too...). It might perform better if this
were one function with a complex signature (like reduce/rereduce), but
that's an implementation detail...