Building a REST-ful API to expose this data.

[bouvard]

Hey everyone,

As I have a fair amount of time when I can think about this project
but not actually write code, I have been doing a lot of pen-and-paper
planning for the API that will expose this data. Ideally, I want to
provide an interface that is functional enough to encourage widespread
adoption forward-thinking enough that changes in either CouchDB or the
availability of the data will not require us to alter the API.

If any of you have used CouchDB before then you know that CouchDB
does not support not support a lot of features of a traditional RDBMS--
in particular its ability to run ad hoc queries is very, very
limited. All queries need to be defined up front and utilize a single
parameter if they are going to run efficiently. Additionally,
CouchDB's structure has interesting implications for paging and the
accepted way of doing it is to pass back the "startkey" of the next
page to the client with each group of data and then pass that in to
the server with the request for the next page. (There is no way to
jump to an arbitrary page without buffering all the data on the server-
side.)

I don't think that these limitations are particularly onerous for a
data set such as this one, however, they do make it rather difficult
to design a traditional "REST-ful" URI scheme. After several
revision, the best plan I have come up with is something like this:

# Events is a common root, branch the view, legislative is the
parameter, and pagekeywould be the id of the first document starting
the page (id being a ISO 8601 datetime string referring to the date of
the event)
/events/branch/legislative/pagekey

# Same as above, but with an entity view
/events/entity/House of Representatives/pagekey

# For returning a range of documents, by date (paging is implicitly
supported)
/events/between/startdate;enddate

# For returning events "before" and "after" a given date, with paging.
/events/before/enddate/pagekey
/events/after/startdate/pagekey

# For returning all events, with paging
/events/all/pagekey

If you haven't worked with CouchDB before, this probably seems like
a somewhat eccentric way of structuring URLs. Does anyone have any
ideas how this might be improved upon? I know it isn't truly REST-
ful, but it seems to be about as close as I can get given the
structure of the underlying database.

I'm looking forward to hearing any feedback on the design (and also
any use cases anyone has in mind).

Cheers,
Chris

[Derek Williams]

Chris,

I'm using similar technologies (CouchDB, RESTful API) for physician data.  There are quite a few views that can be created ahead of time to make things simpler.  I've added 20 or so views, which makes the inserts a bit slow, but everything else is extremely fast.  I'll send some more details when I get back to the office.

Best,

Derek

--
Derek Williams
Cell: 970.214.8928
Home Office: 970.416.8996

[bouvard]

I'd be really happy to see what you've done Derek. Our update/insert
load for this data is going to be very low once the initial dataset is
loaded, but I expect to be doing a lot of reads. Any insights you
have gained from your project will surely be applicable to this one.

Thanks,
Chris