[stripe-api-announce] Updated Pagination API

[Jim Danz]

Hi folks,

We've overhauled how pagination works on lists returned by the Stripe
API and are moving from offset-based pagination to cursor-based
pagination.

If you use other modern web APIs, you've likely already encountered
cursor-based pagination. The change to cursor-based pagination is
useful in many ways. Most visibly, newly created objects won't make
you "lose your place" the way they could with offset-based pagination.

== New parameter `starting_after` for pagination ==
We've added `starting_after` to all API list methods: Pass an object
id as `starting_after` and you'll cursor to next object in the list,
"starting after" the object you specified.

Concretely, suppose you did a `GET /v1/charges` and got back 100
charges, with the 100th charge being `ch_foo`. If you then request
`GET /v1/charges?starting_after=ch_foo`, you'll get the next page of
charges -- those "starting after" `ch_foo` in the list.

== Stop returning `count` in responses ==
We observed that when someone is paginating through a list (especially
with cursor-based pagination rather than offset-based pagination),
it's not very helpful to receive a freshly computed `count` at each
step of the way. To add to the confusion, `count` sounds like it'd be
the count of the number of objects that you've gotten in your
particular response rather than the count of all objects in the
dataset.

Beginning with API version 2014-03-28, `count` is no longer returned
in list API calls. If you ever need the total count of all objects in
the dataset that you're paginating through, just specify
`&include[]=total_count` in your request.

== Rename `count` to `limit` in requests ==
To reduce ambiguity on the request side, we now use the parameter
`limit` instead of `count` to specify the limit of the number of
objects that you'd like to receive. (For backwards compatibility,
we'll continue to accept the `count` parameter.)

Finally, we've added a dedicated section of our API documentation to
summarize how pagination works: https://stripe.com/docs/api#pagination

Here's to faster and more reliable pagination for everybody. Feel free
to reply here if you have any questions or feedback about the changes.

Cheers,
Jim

--
You received this message because you are subscribed to the Google Groups "Stripe API Announcements" group.
To post to this group, send email to api-an...@lists.stripe.com.
Visit this group at http://groups.google.com/a/lists.stripe.com/group/api-announce/.

[Gabor Vitez]

Hi Jim,

A bit of problem with this approach is that it kills parallelism; you can now longer do parallel data fetches as sending out a new request depends on the previous one for the `starting_after` parameter. One use case for parallel bulk requests are stripe connect analytics apps; on the first login for a new user you'll want to use them to speed up the data import and thus improve the user experience. (Later you can possibly get away without this kind of parallelism as there won't be that much data to load anyway.)

If you could reintroduce the `offset` parameter to signify the offset from the `starting_after` object id, and modify the `count` response to return the object count after the `starting_after` id it would be the best of both worlds: both parallel loading and cursor based pagination would work.

Hope that makes sense.

regards,
Gabor

You received this message because you are subscribed to the Google Groups "Stripe API Discussion" group.
To post to this group, send email to api-d...@lists.stripe.com.
Visit this group at http://groups.google.com/a/lists.stripe.com/group/api-discuss/.

To unsubscribe from this group and stop receiving emails from it, send an email to api-discuss...@lists.stripe.com.

[Jim Danz]

Hi Gabor,

Thanks for writing in to raise this.

Among the goals of this new API is that we should be making it
strictly faster to fetch large datasets, so the issue that you bring
up is certainly germane to what we're trying to support here. I
actually think (and I'd be happy to do some of my own benchmarking to
prove/disprove) that for, eg, sucking down 100,000 charges on a
merchant, a single thread using this newstyle API will beat a
threadpool pounding the oldstyle API sliced by offset.

Another thing that I'll say is that the offset parameter is not going
away. It is undocumented now to nudge people in the direction of the
newstyle API, a decision that we can reverse if it causes too much
confusion. But we absolutely intend to support it indefinitely for
backwards compatibility reasons. So whatever reason(s) that you have
for needing offset, either as part of a transitional measure or even
as part of your final implementation, we will happily support. Offset
"composes" with `starting_after` in that asking for
`starting_after=ch_foo&offset=1000` will give you the thousandth
charge after `ch_foo`.

It's an interesting suggestion for `count` to return the object count
after `starting_after`. We want `starting_after` to be very obviously
a *pagination parameter* as opposed to a *filter*, so we didn't want
it to affect the total count returned (in the same way that increasing
the offset wouldn't affect the count returned). However, I could see
the value in being able to request the `remaining_count` of objects
after object `starting_after`. This is something that I'll float with
the team here.

Finally, I want to mention that while I feel that this API does indeed
make things incrementally better for the "suck down all the data"
connect application use case that you're describing, it is by no means
a final solution or the most ideal solution. This connect application
use case, and other related use cases, have suggested to us that we
should be exploring explicit export APIs. For instance, users of
Stripe are able to log into their dashboards and export CSV-formatted
data. The fact that this functionality is not available to connect
apps is much more of an implementation detail than it is a design
decision. It's too early to speak in terms of timelines, but I can
say that we've identified that there'd be a lot of value in providing
a hassle-free way for a connect app to receive a bulk export of data
as fast as possible, and it's something that we hope to address.

Cheers,
Jim

[Michaël Gallego]

Hi!

I've updated my ZfrStripe (unofficial PHP wrapper) so that it already supports it (it also supports the older API versions): https://github.com/zf-fr/zfr-stripe

It's kinda nice, as the new "has_more" allows to avoid the latest request, just to check if there are any more elements.

[Aaron Odom]

I've switched from using "count" and "offset" to using "starting_after" and "limit" for pagination but I'm having an problem understanding how a "Previous" button would work with these arguments.  For example, I'm displaying a limit of 10 transfers with a "next" and a "previous" button at the bottom of each page.  When the user clicks "next", the last transfer id is passed to the next page where it is used in the "starting_after" argument. This works great, however, there is no "ending_before" argument so I'm not sure how to get the "previous" button to work.  If someone clicks the previous button, how do I know what "starting_after" id to use?

 I've considered using a session variable to store the last used id but that's not a good solution because if someone shares the link of the 3rd page, there will be no session for the person who receives the link. Any help would be greatly appreciated. Thanks!  

[Dylan Witte]

This is something I'm currently struggling with myself since I'm updating my application to use the newer API.  I was thinking of storing the value in the URL myself, but that seems like a less than ideal solution.

[Jim Danz]

Hi Dylan,

Somehow the threads got split here, but we took Aaron's feedback and built a `ending_before` parameter the way he suggested. 

The parameter is now documented here:

https://stripe.com/docs/api#pagination

It's designed specifically with building "previous" buttons in mind, so let us know if it won't suit your needs.

Cheers,

Jim

You received this message because you are subscribed to the Google Groups "Stripe API Discussion" group.
To post to this group, send email to api-d...@lists.stripe.com.