How to handle a list when the query param string is too long for a GET request

[guivalerio]

Hi all,

I am implementing an API method that should receive a list of facebook
IDs (e.g. 123,345,765,234) and return all users who have those IDs
linked to their account.

Initially I was using the users list method, which is also used when
running a search by name (i.e. /users?q=john+smith), but the facebook
IDs param can be reaaally long, as it will contain all the user's
friends IDs.

So there is my problem. A request on /users?facebook_ids={massive long
string with comma separated IDs} will break on a GET, so I need to
pass it via a POST.

I don't really want to use a POST request on /users, as that is for
creating a new user. For now I have made a temporary hack so it works
under something like /facebookfriends via a POST, which is horrible.

So the call returns a list and although it should be done via a GET
method, I can't, as the string with the list of facebook IDs is
massive. No really sure the best way forward.

Did anyone have any similar situation? How did you handle it?

Cheers,
G

[Mike Schinkel]

Agree, it's not ideal, nor is my suggestion. But this is one way to handle:

1.) POST the list of users to create a "user list" (a noun). Return 201 "Created" with a "Location" header to a URL like /userlists/{$userlist_id}

2.) GET /userlists/{$userlist_id} to retrieve the list of users.

3.) Also support DELETE /userlists/{$userlist_id} to explicitly purge the list.

4.) Optionally delete a user list after a length of time since last accessed (1 hour, 1 day, 30 days?), assuming no system would ever need to "bookmark" the list.

Hope this helps?

-Mike

http://about.me/mikeschinkel  

[Andy Berryman]

I had a similar problem and decided to go with the pattern ...

POST /users/search

So it doesnt have you doing a POST to the resource "/users" which would indicate a different operation.  Basically I'm coding for "search" as a specific resource "ID" that has special handling.  The drawback is that it means that no user could ever have the "ID" of "search".  Which seemed reasonable to me.

Andy

--

Andy Berryman
Technical Lead

919.228.4857 office
866.225.3085 fax


2701 Aerial Center Parkway
Morrisville | North Carolina | 27560

www.channeladvisor.com - Be Seen

[Daniel Roop]

While I didn't implement it, we had a similar situation come up and we were going to solve it using the user list suggestion, that Mike put out there. 

That being said, the URI length is a legacy browser restriction not a real spec restriction and most browsers don't enforce a limitation any more

http://www.ietf.org/rfc/rfc2616.txt

Section 3.2.1

The HTTP protocol does not place any a priori limit on the length of
   a URI. Servers MUST be able to handle the URI of any resource they
   serve, and SHOULD be able to handle URIs of unbounded length if they
   provide GET-based forms that could generate such URIs. A server
   SHOULD return 414 (Request-URI Too Long) status if a URI is longer
   than the server can handle (see section 10.4.15).

      Note: Servers ought to be cautious about depending on URI lengths
      above 255 bytes, because some older client or proxy
      implementations might not properly support these lengths.

[Ed Anuff]

Interesting, would love to see a compatibility matrix for who is enforcing URL length limits via XHR and JSONP.

Ed

[Darrel Miller]

On Mon, Apr 9, 2012 at 12:51 PM, guivalerio <ggu...@gmail.com> wrote:
> I don't really want to use a POST request on /users, as that is for
> creating a new user. For now I have made a temporary hack so it works
> under something like /facebookfriends via a POST, which is horrible.
>

I'm curious, why do you consider this solution horrible? You have
created a new resource /facebookfriends, which you are treating like a
data processing resource as is described in the HTTP spec. You are
passing a list of Ids to it and it is returning to you a list of
users.

From my perspective, I see nothing wrong with this. Is it just the
name of the resource that you don't like?

Darrel

[guivalerio]

Daniel and Mike,

I was thinking about implementing it that way. That would imply me
adding a table in the db to store the facebook friends resources, but
that is not too bad. I wouldn't really use it as a data cache, so
every the the API method is called, the list would be recreated. Only
thing I dislike is having to perform 2 calls to the API instead of one
(create list via POST and then retrieve it via a GET).

Daniel Roop, that's quite interesting, I didn't know about that. Not
sure if it is a good idea to pass thousands of characters in a GET
request though. Can you foresee any problems by doing that? (I shall
run a few tests! :))

Daniel M, I said it was horrible as I just created a non-resourceful
method (the facebook friends list is not created and stored anywhere,
there are no models for it either). I think it really belongs in the
API Users controller (/users/1/friends?facebook_list=id1,id2,id3...)
and that it should be called via a GET (as a POST request implies I am
creating something, which I am not.... unless I go with Mike's
suggestion).

Thanks for your replies so far!
Guilherme

[Darrel Miller]

On Wed, Apr 11, 2012 at 8:52 AM, guivalerio <ggu...@gmail.com> wrote:
>
> Darrel M, I said it was horrible as I just created a non-resourceful

> method (the facebook friends list is not created and stored anywhere,
> there are no models for it either).

I'm afraid I don't know what you mean by "a non-resourceful method".
However, there absolutely no requirement in the HTTP spec that says a
POST needs to create or store anything.

> I think it really belongs in the
> API Users controller (/users/1/friends?facebook_list=id1,id2,id3...)
> and that it should be called via a GET (as a POST request implies I am
> creating something, which I am not.... unless I go with Mike's
> suggestion).
>

A POST does not imply that you are creating something. A POST might
create something, or it might not. If you get back a 201 then it did
create something.

Here are some examples of the types of things you can do with POST
from the HTTP spec[1] :

o Annotation of existing resources;

o Posting a message to a bulletin board, newsgroup, mailing list, or
similar group of articles;

o Providing a block of data, such as the result of submitting a
form, to a data-handling process;

o Extending a database through an append operation.

If you look at the third option and paraphrasing, you are providing a
block of data to a data handling process. This is a completely valid
use of POST. The notion of /facebookfriends as a "data-handling"
resource is just as valid a resource as a UserList resource that you
can conceptually create and delete.

I frequently find people box themselves into a corner by mapping HTTP
methods to CRUD operations and then ask, how can I do non-CRUD stuff.
The answer is to stop mapping HTTP methods to CRUD.

Darrel

[1] http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-19#page-20

[Matthew Bishop]

Mike's suggestion is actually ideal to me. It expresses the very real
fact that the list of users is a resource that the caller can create
and use in different contexts that understand it.

Query parameters shouldn't be used to communicate business data or
resource scoping. In the API I am responsible for, query params are a
code smell that means you haven't thought through your resource
carefully enough. Often times it means another resource is required to
express the data and state of the feature being implemented.

[Sam Ramji]

The other nice thing about Mike's suggestion is that it's a pattern that can enable asynchronous processing on the request.  As we see APIs scale up, being able to decouple "request for data processing" from "response from data processing" is getting important.  Passing in a callback URL with the request is getting more common in these scenarios.

Cheers,

Sam