Where to put paging metadata in restful response?

[danieleli]

I'm building an app using backbonejs for client platform.  The backend is using rails.  When I query a large collection the backend is returning an object like this:

{
  count:70,
  current_page:1,
  data:collection_inserted_here,
  next:'url_to_next_page',
  number_of_pages:3
}

This is too bad as I can override the 'parse' function in backbone, but it smells funny. 

The pdf from apigee says to include this metadata but doesn't suggest where.  I was thinking it might go in response header or something that leaves the response body cleanly consumable.

Does anyone have any suggestions around this?

thx!

Dan

[landlessness]

there are some good thoughts on metadata in this thread:

https://groups.google.com/forum/?fromgroups=#!topic/api-craft/zt-bbxEG7vw

Kevin Swiber (active contributor to API Craft) and I will be working on a 3rd edition of API Design guidance soon.

[Daniel Roop]

we have landed on 

{

   <<meta-data-properties>>

   "entires" : [

       ...

    ]

}

Where entries is the actual list you got back, and the meta data like next/previosu links and counts live outside of that.  So in practice you end up with something like

{

    "offset" : 10,

    "limit" : 10,

    "total" : 100,

    "entries" : [

        ...

     ],

     "links" : {

         "next" : {

                "href" : "/collection?offset=20&limit=10"

         },

         "previous" : {

                "href" : "/collection?limit=10"

         }

     }

}

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group, send email to api-craft+...@googlegroups.com.
Visit this group at http://groups.google.com/group/api-craft?hl=en.
 
 

[Jon Moore]

Note that you can also use Link headers (RFC 5988) on the response, as in:

Link: <../collection?limit=25>; rel="first prev"
Link: <../collection?offset=50&limit=25>; rel="next"

This has the benefit of keeping your paging logic cleanly out of your
application semantics. As it happens, I just wrote a blog post about
managing lists with HTTP in a media-type-agnostic way (so long as the
media type can represent lists) that includes how to manage list
paging:

http://codeartisan.blogspot.com/2012/08/hypermedia-programming-lists.html

Jon

--
........
Jon Moore

[Mike Kelly]

Isn't pagination part of his "application semantics" though?

[Moore, Jonathan (CIM)]

Yes, you're right, poor choice of words by me. I meant that putting the pagination information in Link headers has the benefit of being agnostic to the particular media type in use, and hence could allow for generic intermediary processing (e.g. one that ran the links and prefetched them).

If a part of my application semantics can be expressed "natively" in HTTP, I usually prefer to do so in addition to/instead of in a media type to increase Visibility and to thus better take advantage of Layering.

Jon
........
Jon Moore

[Mike Kelly]

On Thu, Sep 6, 2012 at 10:58 AM, Moore, Jonathan (CIM)
<Jonatha...@comcast.com> wrote:
> Yes, you're right, poor choice of words by me. I meant that putting the pagination information in Link headers has the benefit of being agnostic to the particular media type in use, and hence could allow for generic intermediary processing (e.g. one that ran the links and prefetched them).
>
> If a part of my application semantics can be expressed "natively" in HTTP, I usually prefer to do so in addition to/instead of in a media type to increase Visibility and to thus better take advantage of Layering.
>

Your links will be visible from the body if you use a standardised
media type with link semantics, i.e. Atom, HAL, HTML, etc.

[Jon Moore]

........
Jon Moore

On Sep 7, 2012, at 5:46 AM, Mike Kelly <mikeke...@gmail.com> wrote:
> Your links will be visible from the body if you use a standardised
> media type with link semantics, i.e. Atom, HAL,

True, but then my intermediary needs to understand all those media types. If I put the paging info in Link headers it just needs to speak HTTP, and will be able to provide benefit even for messages carrying media types that haven't even been invented yet.

Jon

[Mike Kelly]

I find that a good argument for avoiding the proliferation of media
types that all do linking in their own special way, rather than one in
favour of sticking important parts of your application up in the http
headers. Also - the Link header isn't technically part of HTTP right?
Afaik, it's a separate specification?

Aside from all that, Link headers aren't that easy to work with since
the tool support is relatively thin on the ground, and of course the
killer is that there are practical limitations to the size of the HTTP
header block which you would hit fairly quickly.

Cheers,
M

[Jon Moore]

On Sep 7, 2012, at 6:14 AM, Mike Kelly <mikeke...@gmail.com> wrote:
> I find that a good argument for avoiding the proliferation of media
> types that all do linking in their own special way, rather than one in
> favour of sticking important parts of your application up in the http
> headers.

Right--isn't that what this thread is about? Where to put the paging links in a JSON response?

> Also - the Link header isn't technically part of HTTP right?
> Afaik, it's a separate specification?

Yes, it's a separate specification, but it operates at the HTTP protocol level, which makes it media-type agnostic.

> Aside from all that, Link headers aren't that easy to work with since
> the tool support is relatively thin on the ground,

But we're willing to write code to parse the links out of a media type?

> and of course the
> killer is that there are practical limitations to the size of the HTTP
> header block which you would hit fairly quickly.

What limitations have you encountered (I can believe they exist, just curious what they are)?

In any event, I agree a Link header is most useful for (1) standardized relations; (2) in common use. I'm suggesting that paging (next/prev/first/last) fits that description.

[Mike Kelly]

On Fri, Sep 7, 2012 at 11:57 AM, Jon Moore <jo...@jjmoore.net> wrote:
> On Sep 7, 2012, at 6:14 AM, Mike Kelly <mikeke...@gmail.com> wrote:
>> Aside from all that, Link headers aren't that easy to work with since
>> the tool support is relatively thin on the ground,
>
> But we're willing to write code to parse the links out of a media type?

parsing JSON is trivial in most languages. If your media type is well
designed getting the links out should also be equally trivial.

>> and of course the
>> killer is that there are practical limitations to the size of the HTTP
>> header block which you would hit fairly quickly.
>
> What limitations have you encountered (I can believe they exist, just curious what they are)?

Apache's default max size for headers is 8K:

https://httpd.apache.org/docs/2.2/mod/core.html#limitrequestfieldsize

Cheers,
M

[Pat Cappelaere]

I would tend to agree with Mike on this one.
Making links explicit in the message is a good practice and will help us move to full Hypermedia.
I would hate having to parse links in both headers and messages.
At least you are pretty much guaranteed that they will be go through all proxys and firewalls.
Pat.

[Peter Williams]

On Fri, Sep 7, 2012 at 5:18 AM, Mike Kelly <mikeke...@gmail.com> wrote:
> On Fri, Sep 7, 2012 at 11:57 AM, Jon Moore <jo...@jjmoore.net> wrote:
>> On Sep 7, 2012, at 6:14 AM, Mike Kelly <mikeke...@gmail.com> wrote:
>>> Aside from all that, Link headers aren't that easy to work with since
>>> the tool support is relatively thin on the ground,

I find them quite easy to work with, personally.

>> But we're willing to write code to parse the links out of a media type?
>
> parsing JSON is trivial in most languages. If your media type is well
> designed getting the links out should also be equally trivial.

Parsing link headers is also trivial. But with header i only have to
write and maintain that code once, rather than once per media type.

>> What limitations have you encountered (I can believe they exist, just curious what they are)?
>
> Apache's default max size for headers is 8K:
>
> https://httpd.apache.org/docs/2.2/mod/core.html#limitrequestfieldsize

That setting doesn't effect the response header, does it? In my
experience the response is where most link header fields go, not the
request.

Also, that setting seems to limit the size of individual header
fields, not the header as a whole. So each link field in the header
would be prevented from exceeding 8k. My links are rarely that large.
:)

I have never experienced, nor have i heard of anyone else experiencing
*any* issues related to intermediates messing with or stripping entity
header fields. I would be *shocked* if i ever do. Any intermediate
that did so would, imo, be inexcusably broken.

Peter
barelyenough.org

[Patrice Cappelaere]

US Government is broken (at least in that regard)

Pat

-- 
Patrice Cappelaere

Sent with Sparrow

[Peter Williams]

On Fri, Sep 7, 2012 at 11:49 AM, Patrice Cappelaere
<cappe...@gmail.com> wrote:
> US Government is broken (at least in that regard)

Can you be more specific about the behavior you have seen (and where)?

Peter
barelyenough.org