RESTful batch operations
Steve M
---------------------
I've been trying to buld a web application in a RESTful fashion, using
for guidance among other resources the Leonard Richardson and Sam Ruby
book _RESTful Web Services_. I've learned that REST is a style of
architecting applications, and that Pylons with the map.resource()
function of routes can be seen as one particular way of defining and
working within some RESTful constraints.
The way that map.resource() sets things up is basically as follows.
There are two types of URL, collections (/resources) and individuals (/
resources/6), and four HTTP verbs, GET, POST, PUT, and DELETE. These
are variously combined to allow for seven different REST actions:
index, show, new, edit, create, update, and delete.
If you group the REST actions by type of URL, you get the actions that
work on collections: index, new, create
and the actions that work on individuals: show, edit, update, delete
To make a RESTful web application under these conditions means to
express all functionality the application provides in terms of
resources and those seven actions acting on resources. This is an
interesting and challenging project, and to the extent that it can be
successful, the benefits of the RESTful approach are significant. Now,
on to my current problem!
Batch Operations
----------------
How do you implement batch operations on collections of resources?
Imagine you have a resource Messages. If you "GET /messages" it
returns an index of messages.
If you "DELETE /messages/4" it will delete message 4.
Perhaps the "GET /messages" index includes for each resource in the
collection a form with a button that allows the client to send an HTTP
DELETE to that message's URL. So if there are ten messages in the URL,
there will be ten items returned by the index action, and those will
include ten forms, one to send a DELETE to each of the ten messages.
Now, suppose you wish to have a way for the user to do a batch delete
on a subset of messages. The traditional (non-RESTful) way to achieve
this is to have the list of ten messages inside a single form, with a
checkbox for each list item. The Delete button would submit an
overloaded POST to some URL such as "POST /messages/delete" and the
server can delete all the checked off items. Or, alternatively, there
will be an even more generic URL, "/messages/batch" and the fact that
the selected messages are to be deleted or moved to a different folder
will be specified as an input element of the form. But of course
neither of these is available in the current RESTful framework. The
way to delete a set of five different messages is to send five HTTP
requests, each of the form "DELETE /messages/:id".
Is it possible to allow batch operations within the RESTful
constraints? If so, what is the best way to do this?
Three Ideas
-----------
1. Use AJAX techniques to submit the five requests for the user.
Basically have the form submit intercepted by a javascript which fires
off five asynchronous HTTP DELETE requests to the five URLS and
collects the responses for display. This requires no changes to the
server API. If the client does not have javascript, the interface can
degrade gracefully to requiring a separate submit button click for
each delete.
2. Of the seven REST actions, four of them (show, edit, update,
delete) act upon individual resources. Extend these to be able to act
on collections of resources, and therefore also generalize the URL
formats to allow for addressing subsets of collections. For example,
the URL "/messages/3,4,18,22-50" refers to a subset of the collection
of messages. An HTTP DELETE to that URL would delete each resource in
that subset.
In this case, of the remaining 3 REST actions that act only upon
collections, index, new, and create, one of them, index, would be
updated to act on proper subsets of the collections, and the other two
would behave the same whether the URL refers to the entire collection
or some subset.
One big challenge of this approach is creating a scheme for addressing
arbitrary subsets of a collection, and by addressing I mean having it
in the URL. The scheme would have to be such that the URLs could be
created using HTML forms. Finally, after developing the scheme it
would also require significant changes to routes to support it.
3. Use overloaded POST to create new REST actions in addition to the
basic seven that are meant to support batch operations. For example,
three new actions called batch_create, batch_update, batch_delete. In
this case it is not necessary to address subsets in the URL; the
collection URL for a resource can be used instead. However support in
routes map.resource might be required to route the incoming POST to
the proper RestController action. The POST request body for each of
those actions can contain the set of resources and representations to
act upon.
One difficulty here is figuring out the best way to modify the route
configuration to identify the three new actions. Currently, the only
trickery routes explicitly supports for RESTful apps is the _method
input element used to translate POST to PUT or DELETE, which makes up
for a deficiency in modern browsers and/or HTML 4 that makes it
impossible for most clients to construct an actual HTTP PUT request.
But the current problem doesn't need to translate one HTTP method to
another HTTP method, it needs to include additional information to
distinguish between multiple actions at the collection level. Perhaps
it can be done on the model of the new action, for which the url is '/
messages/new'. So in this case we can have '/messages/batch_create', '/
messages/batch_update', etc.
I'm currently leaning toward option 3. One downside is that a strength
of the RESTful approach is the simple server API. Option 3 does
increase the breadth of the server API, but on the other hand it isn't
obvious that it does this in a way that is inconsistent with REST
principles.
So, has anyone tried to tackle this problem before, of batch
operations in the RESTful framework? Did you try anything like the
approaches I've described above? Any stories of success, pitfalls to
beware, known theoretical limits or false assumptions I've made, or
general engineering advice?
Thanks
Mike Orr
>
> RESTfulness in Pylons
> ---------------------
>
> I've been trying to buld a web application in a RESTful fashion, using
> for guidance among other resources the Leonard Richardson and Sam Ruby
> book _RESTful Web Services_. I've learned that REST is a style of
> architecting applications, and that Pylons with the map.resource()
> function of routes can be seen as one particular way of defining and
> working within some RESTful constraints.
map.resource follows the Atom specificiation, which is a particular
implementation of REST. It's good for resources that can be
serialized or accessed via web services. It's possibly overkill for
web-only resources, especially those that don't implement some of the
modes (index, delete, add).
The main problem with map.resource is that there's no URL symmetry
between the form and its action, yet unrelated actions are doubled up
under the same URL. This is confusing to remember, though it makes
sense for serializable/web service resources.
The main feature of REST is that each URL component should contain the
one on its right, and IDs generally come before actions. So for
instance, one of my sites has a combined details page with tabbed
sections, but a separate edit form for each section. So the display
URL is /sites/N, but the forms are /sites/N/mytab/edit . (Adding is
/sites/add; deleting is not allowed.) This is not Atom-compliant but
it is RESTful.
A simple /resources/add, /resources/N/edit, and /resources/N/delete
scheme, where GET displays the form and POST does the action, would
also be RESTful. I want to add another resource type to Routes for
this, but it won't be anytime soon.
> Perhaps the "GET /messages" index includes for each resource in the
> collection a form with a button that allows the client to send an HTTP
> DELETE to that message's URL. So if there are ten messages in the URL,
> there will be ten items returned by the index action, and those will
> include ten forms, one to send a DELETE to each of the ten messages.
Yes, but the primary purpose of "GET /messages" is to show links for
viewing the resources. Delete buttons can go alongside these, but
should not replace them.
> How do you implement batch operations on collections of resources?
>
> Imagine you have a resource Messages. If you "GET /messages" it
> returns an index of messages.
> If you "DELETE /messages/4" it will delete message 4.
>
>
> Now, suppose you wish to have a way for the user to do a batch delete
> on a subset of messages. The traditional (non-RESTful) way to achieve
> this is to have the list of ten messages inside a single form, with a
> checkbox for each list item. The Delete button would submit an
> overloaded POST to some URL such as "POST /messages/delete" and the
> server can delete all the checked off items. Or, alternatively, there
> will be an even more generic URL, "/messages/batch" and the fact that
> the selected messages are to be deleted or moved to a different folder
> will be specified as an input element of the form. But of course
> neither of these is available in the current RESTful framework. The
> way to delete a set of five different messages is to send five HTTP
> requests, each of the form "DELETE /messages/:id".
There is no REST syntax for batch operations. So do whatever's
easiest. I would put all the data in POST variables. The purpose of
RESTful URLs is so the user can bookmark them and type them, and
non-human agents can manage the resource without having to know the
site's esoteric syntax. None of these apply to batch operations,
which are normally ad hoc (i.e., not repeatable). So
/messages/batch/delete?id=1&id=2
would be acceptable, or
/messages/batch?action=delete&id=1&id=2
(but using POST of course, since GET is not supposed to have
destructive consequences.)
You can define these routes within map.resource or as separate routes.
You can use request.params.getall("id") to get a list of all id
parameters, and url_for("batch", id=[1, 2]) to go the other way.
> 1. Use AJAX techniques to submit the five requests for the user.
> Basically have the form submit intercepted by a javascript which fires
> off five asynchronous HTTP DELETE requests to the five URLS and
> collects the responses for display. This requires no changes to the
> server API. If the client does not have javascript, the interface can
> degrade gracefully to requiring a separate submit button click for
> each delete.
AJAX is outside the realm of URL design. Use it if you like it or it
improves latency.
> 2. Of the seven REST actions, four of them (show, edit, update,
> delete) act upon individual resources. Extend these to be able to act
> on collections of resources, and therefore also generalize the URL
> formats to allow for addressing subsets of collections. For example,
> the URL "/messages/3,4,18,22-50" refers to a subset of the collection
> of messages. An HTTP DELETE to that URL would delete each resource in
> that subset.
You could write your own convention for this, but I don't know if
there's any standard. I don't think map.resource requires the ID to
be numeric, so it could have embedded commas or hyphens.
> In this case, of the remaining 3 REST actions that act only upon
> collections, index, new, and create, one of them, index, would be
> updated to act on proper subsets of the collections, and the other two
> would behave the same whether the URL refers to the entire collection
> or some subset.
map.resource does not use NEW or CREATE. It uses POST to create a
resource, and PUT to modify it. Note that since POST is defined as
"do an arbitrary action", it's ambiguous enough to accommodate batch
operations.
> One big challenge of this approach is creating a scheme for addressing
> arbitrary subsets of a collection, and by addressing I mean having it
> in the URL. The scheme would have to be such that the URLs could be
> created using HTML forms. Finally, after developing the scheme it
> would also require significant changes to routes to support it.
I don't think it would require changes to Routes. You just have to
put routes with a keyword above those that have an ID in the same
position. The id would come in as a single string, which the
controller action would have to parse. You could make a fork of
Routes that splits batch operations into single ones, but it won't be
widely adopted because it's not a standard. And the controller action
itself may pay prefer to get all the IDs together anyway so it can do
a bulk database query.
> 3. Use overloaded POST to create new REST actions in addition to the
> basic seven that are meant to support batch operations. For example,
> three new actions called batch_create, batch_update, batch_delete. In
> this case it is not necessary to address subsets in the URL; the
> collection URL for a resource can be used instead. However support in
> routes map.resource might be required to route the incoming POST to
> the proper RestController action. The POST request body for each of
> those actions can contain the set of resources and representations to
> act upon.
This is essentially the same as my proposal above, just with a different syntax.
--
Mike Orr <slugg...@gmail.com>