Draft spec for API v2 - please comment

[maciej]

Here's the first half of my draft spec for the next-version API. This one covers general API features; I'll post the actual proposed method calls separately.

Please let me know your thoughts, good or bad, and any suggestions:

http://static.pinboard.in/api2_draft.pdf

[Dan Loewenherz]

Looking good!

I was a little confused about the "User Rate Limit" versus "Sitewide Rate Limit". Are sitewide limits also per-user, or are they allocated on a per-app basis?

Regarding certificate validity, where can we go to get the "Pinboard verified" SSL public key and signature?

--
You received this message because you are subscribed to the Google Groups "Pinboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pinboard-dev...@googlegroups.com.
To post to this group, send email to pinboa...@googlegroups.com.
Visit this group at http://groups.google.com/group/pinboard-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[maciej]

On Apr 25, 1:19 pm, Dan Loewenherz <d...@dlo.me> wrote:
> Looking good!
>
> I was a little confused about the "User Rate Limit" versus "Sitewide Rate
> Limit". Are sitewide limits also per-user, or are they allocated on a
> per-app basis?

Sitewide rate limits are per-user. Their purpose is to prevent the
site being clobbered by people who want to collect a lot of global
data.

>
> Regarding certificate validity, where can we go to get the "Pinboard
> verified" SSL public key and signature?

I'll publish it in the docs; thanks for pointing that out.

[maciej]

It's hard to say if they're low or high without talking about specific
use cases. I'm open to arguments in either direction.

On Apr 25, 1:11 pm, Mikael Konutgan <mkonut...@gmail.com> wrote:
> Everything sounds great, but aren't the sitewide rate limits a little low?

[Mikael Konutgan]

Great, 1000 per user is way more than enough.

[kijin]

Looks wonderful. I just have a couple of minor suggestions:

> In many cases, the server will return a 200 OK response even when something goes wrong.

Returning HTTP 200 OK and putting "not_ok" in the status doesn't look right. It would be much better if you reserved 200 OK responses to only those requests that actually succeeded. There are better codes to return when something is wrong with the request, such as an invalid URL. For example, I've seen a few APIs return 422 Unprocessable Entity when a request is syntactically correct but contains semantic errors. Also, I suppose every client should be prepared to encounter 500 Internal Server Error from time to time, so you might as well add it to the spec.

> Requests to certain parts of the site are subject to an additional rate limit

I think the X-Throttle-* headers are a neat trick, but it would be better if a client could tell whether it is being throttled for exceeding the "user rate limit" or for exceeding the "sitewide rate limit" (which I think are confusing labels to begin with, since you said that sitewide limits are actually per-user). Then a client could tell, for example, that although it can't request /site/recent/ anymore, it can still add a new bookmark if the "user rate limit" has not been reached.

Also, it looks like the rate limiting system works by measuring the number of requests in 15-minute "measuring periods", such as 12:00-12:15, 12:15-12:30, etc. Nothing wrong with that, but it would be useful to document this properly, so that there are no surprises. Some people might be under the (actually quite reasonable) assumption that you just SELECT COUNT(*) FROM api_requests WHERE timestamp > (NOW - 900).

15 minutes don't really matter either way, but this could be especially problematic in the case of the daily limit. For example, does the clock reset at 00:00 UTC or 00:00 PST/PDT each day? It would be great if you could specify a UTC time that takes into account most of your customers' usage pattern.

Finally, why use a custom header at all? Why not put the throttle information in the body of the JSON response?

Kijin

[maciej]

On Apr 26, 12:29 pm, kijin <kijins...@gmail.com> wrote:
> Looks wonderful. I just have a couple of minor suggestions:
>
> > In many cases, the server will return a 200 OK response even when
>
> something goes wrong.
>
> Returning HTTP 200 OK and putting "not_ok" in the status doesn't look
> right. It would be much better if you reserved 200 OK responses to only
> those requests that actually succeeded. There are better codes to return
> when something is wrong with the request, such as an invalid URL. For
> example, I've seen a few APIs return 422 Unprocessable Entity when a
> request is syntactically correct but contains semantic errors. Also, I
> suppose every client should be prepared to encounter 500 Internal Server
> Error from time to time, so you might as well add it to the spec.

I don't like the idea of shoehorning stuff into HTTP error codes when
there's not an obvious mapping. In the example you give (422
Unprocessable Entity) there would still need to be a meaningful error
message in the body of the response to give the client any clue about
what specifically had gone wrong.

In other cases, there is no meaningful error code to map to. For
example, a request to 'bookmarks/delta' will return a changeset since
a given timestamp. But if there are too many changes, it will return
an error that basically says 'you're too out of date, you need to
request everything'.

It's best to think of the HTTP error codes indicating transport
errors, and the JSON error codes indicating API errors. Even if
you're stubborn and ignore the HTTP status codes altogether, you
should do fine as long as you're checking JSON errors.

>
> > Requests to certain parts of the site are subject to an additional rate
>
> limit
>
> I think the X-Throttle-* headers are a neat trick, but it would be better
> if a client could tell whether it is being throttled for exceeding the
> "user rate limit" or for exceeding the "sitewide rate limit" (which I think
> are confusing labels to begin with, since you said that sitewide limits are
> actually per-user). Then a client could tell, for example, that although it
> can't request /site/recent/ anymore, it can still add a new bookmark if the
> "user rate limit" has not been reached.

User and sitewide refer to the information being requested, not the
entity making the requests.

I don't think multiple throttle headers are a good idea. It adds
complexity for no clear benefit. A client can always try different
API calls and back off if they're consistently throttled.

>
> Also, it looks like the rate limiting system works by measuring the number
> of requests in 15-minute "measuring periods", such as 12:00-12:15,
> 12:15-12:30, etc. Nothing wrong with that, but it would be useful to
> document this properly, so that there are no surprises. Some people might
> be under the (actually quite reasonable) assumption that you just SELECT
> COUNT(*) FROM api_requests WHERE timestamp > (NOW - 900).

This kind of detail would create an illusion of stability where I
don't want one. I meant it when I wrote "Rate limits can change
without warning"!

>
> 15 minutes don't really matter either way, but this could be especially
> problematic in the case of the daily limit. For example, does the clock
> reset at 00:00 UTC or 00:00 PST/PDT each day? It would be great if you
> could specify a UTC time that takes into account most of your customers'
> usage pattern.

I don't really understand the scenario where this is useful. You get
the number of seconds remaining until the reset as part of the
response, and can figure out the date from that, right?

> Finally, why use a custom header at all? Why not put the throttle
> information in the body of the JSON response?

So the entity doing the throttling doesn't have to know anything about
the API format.

[kijin]

On Saturday, April 27, 2013 7:51:39 AM UTC+9, maciej wrote:

I don't like the idea of shoehorning stuff into HTTP error codes when
there's not an obvious mapping.  In the example you give (422
Unprocessable Entity) there would still need to be a meaningful error
message in the body of the response to give the client any clue about
what specifically had gone wrong.

 

In other cases, there is no meaningful error code to map to.   For
example, a request to 'bookmarks/delta' will return a changeset since
a given timestamp.  But if there are too many changes, it will return
an error that basically says 'you're too out of date, you need to
request everything'.

My suggestion was *not* that you should return a unique status code for each type of error. My suggestion was only that you should return a non-200 status code when the API request is anything less than perfect, because 200 means "OK" and it's odd to return "not_ok" with it. I used 422 as an example because it implies client error while being broad enough to cover a wide range of errors.

 

It's best to think of the HTTP error codes indicating transport
errors, and the JSON error codes indicating API errors.  Even if
you're stubborn and ignore the HTTP status codes altogether, you
should do fine as long as you're checking JSON errors.

On the flipside, there are cases where a "stubborn" client that ignores the JSON response altogether might still do relatively fine as long as it checks HTTP status codes. For example, an app that does nothing but add new bookmarks (or some other write-only task) might only care whether the operation succeeded. It would display "Success" if it detects a 200 (or 201 Created) status code, and "Sorry, an error occurred" if it detects anything else.

Meanwhile, it's not like you're using HTTP error codes to convey transport errors only. You return 401 Unauthorized when a client fails to send a valid API key, but you probably don't add a valid WWW-Authenticate: header to this response, either. In principle, this is against RFC2616/2617, and therefore might be seen as (mis)using HTTP error codes to convey API errors. But this is OK if we all agree that it's sometimes useful to map the two kinds of errors to each other.

 

I don't think multiple throttle headers are a good idea. It adds
complexity for no clear benefit.   A client can always try different
API calls and back off if they're consistently throttled.

 

This kind of detail would create an illusion of stability where I

don't want one.  I meant it when I wrote "Rate limits can change
without warning"!

 
Thanks for the explanation. Still, I think the spec needs a better phrasing than "The X-Throttle-Reset header indicates how many seconds remain until the next measuring period starts." What about something along the lines of "These headers indicate that the client is permitted to make ${X-Throttle-Remaining} more requests (of the same kind as the current request, whether user or sitewide) in the next ${X-Throttle-Reset} seconds." IMO this is more relevant to developers while disclosing even less information about implementation details.

Kijin

[Dan Poirier]

Looking good overall. It'll be nice to have an updated API.

I agree that HTTP status codes should generally indicate whether the HTTP request had a problem. If the request was understood by the API but something went wrong while performing the request, a 200 is okay. (If we were trying for a pure REST API, I might rethink that, but clearly we aren't doing that here.)

Perhaps it would seem less incongruous, though, to not use "ok" in the status field.  I also like the suggestion someone else made that the value could just be a boolean. Maybe instead of

    status : "ok|not_ok",

it could return

    success: true|false,

Dan P.

[kijin]

On Saturday, April 27, 2013 9:44:58 PM UTC+9, Dan Poirier wrote:

If the request was understood by the API but something went wrong while performing the request, a 200 is okay.

I find it really annoying when a random PHP tutorial on the web tells the reader to just die("Error connecting to the database") with a "200 OK" status code. Maybe my excessive exposure to badly coded apps has permanently damaged my taste buds, but this gives me a bad taste every time I see this pattern.

Be forgiving in what you accept, and strict in what you produce. Failure is failure, and any failure contradicts "OK" no matter how you word it. Not to mention it feels really weird when you're trying to add a new bookmark and "201 Created" means success whereas "200 OK" means something might have gone wrong.

Nevertheless, if sticking to "200 OK" makes things significantly easier for maciej to manage for one reason or another, I won't complain about it anymore. There's plenty of room for API client implementations to hide minor oddities like this.

Kijin

[maciej]

It might be better to defer this until the discussion about specific
API calls, so we don't get too abstract. I do share your distaste for
the 200 + error pattern, so hopefully we can find ways to avoid it as
much as possible.

[Les Orchard]

On 4/25/13 3:52 PM, maciej wrote:
> Please let me know your thoughts, good or bad, and any suggestions:
>
> http://static.pinboard.in/api2_draft.pdf

Why not use HTTP Basic auth for the user + token authentication?

You could even combine the app ID into basic auth & parse it out on the server side - eg. base64(appID+user:token)

That might make things easier on the client, eg. no custom X-* headers. I know it can do funky things in browsers - ie. entice a user to enter user/pass in a phishing attempt (which happened on del.icio.us) But that might be less a concern since the user + token are not the same as user + password

--
Les Orchard
http://lmorchard.com/
{web,mad,computer} scientist; {tech,scifi} writer; home{brew,roast}er

[Les Orchard]

On 4/26/13 6:51 PM, maciej wrote:

> I don't like the idea of shoehorning stuff into HTTP error codes
> when there's not an obvious mapping.  In the example you give (422
> Unprocessable Entity) there would still need to be a meaningful
> error message in the body of the response to give the client any clue
> about what specifically had gone wrong.

HTTP error code scheme lends itself to shoehorning - at least, with respect to statuses 200, 400, and 500.

200 OK is the catch-all for success; 400 Bad Request is the catch-all for something the client did wrong; and 500 is the catch-all for a server failure. If you can't nail the status down to a more specific code within those ranges, it's up to the client to consult the response body (as you mention about JSON error codes).

But, at least 2xx/3xx/4xx/5xx gives an initial branching point for response handling, both for the end client and intermediaries. For example, the status influences caching + conditional GET - 200 OK for everything means caches (in-app or on the network) happily hang on to errors

[Les Orchard]

On 4/25/13 3:52 PM, maciej wrote:

One more thought: I'd love to see CORS enabled for this API, so I can use it from HTML5 web apps. (eg. in a browser, on a FirefoxOS phone, etc)

Sending a `Access-Control-Allow-Origin: *` header will cover GETs, but POST/PUT/DELETE requires a slightly more complicated dance:

https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS#Preflighted_requests

[Dan Loewenherz]

I'm personally a fan of the way the Stripe API utilizes HTTP status codes, not that I really consider myself a stickler for these things.

https://stripe.com/docs/api#errors

200 OK - Everything worked as expected.

400 Bad Request - Often missing a required parameter.

401 Unauthorized - No valid API key provided.

402 Request Failed - Parameters were valid but request failed.

404 Not Found - The requested item doesn't exist.

500, 502, 503, 504 Server errors - something went wrong on Stripe's end.

--
You received this message because you are subscribed to the Google Groups "Pinboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pinboard-dev...@googlegroups.com.
To post to this group, send email to pinboa...@googlegroups.com.

[Stephen Darlington]

Will there be a way in the API to "convert" a username and password to an API token? 

I ask because no matter what you put on-screen, users will enter their username and password. The only other API I've used with the same feature is bit.ly's, and I've lost count of the number of support emails telling me that my app is broken. (It isn't, at least not in this respect.)

Can't believe I'm even suggesting this, but have you considered OAuth or at least something like it? You could have a "flow" that apps follow -- open a web page, log in, get the API key -- that would make it much less error-prone.

Cheers,

Stephen

--
You received this message because you are subscribed to the Google Groups "Pinboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pinboard-dev...@googlegroups.com.
To post to this group, send email to pinboa...@googlegroups.com.
Visit this group at http://groups.google.com/group/pinboard-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

------------------------------------------------------------------------

                    Stephen Darlington (www.zx81.org.uk)

"I blame sex and paper for most of our current problems"

------------------------------------------------------------------------

[kijin]

API v1 has a call, /user/api_token, that returns a user's API token. Since API v1 uses the username and password for authentication, I suppose you could use it to obtain a user's API token, and then make API v2 requests with the token. Of course, this assumes that API v1 will remain operational after API v2 is released.

Kijin

[maciej]

For some reason I'm deeply ambivalent about making the API friendly for use from within web apps. I guess I need to look deep within myself and figure out why I have such a server-side bias.

[maciej]

I don't see a reason not to let people trade auth cookies or their credentials for the API token.  I'm sympathetic to your plight.

As far as oauth goes, I came close to implementing it, but got scared off by the complexity and the unfortunate resemblance to phishing.

[maciej]

Thanks to everyone who commented on the draft API spec. Here are the
changes I'll make based on your feedback:

- allow HTTP Basic Auth with the API token
- add an API call for fetching API token with a username/password pair
or website auth cookie
- remove X- from HTTP header names
- all API errors will be reflected in the HTTP status code, using 400
Bad Request as a generic catchall
- success field in the JSON envelope will have a value of 'true' or
'false' rather than 'ok' and 'not_ok'
- add pubkey and signature to API docs

Some things I'm still mulling over:

- CORS or JSONP support
- What to name the throttling headers.

I'll post some actual proposed API calls next. Of course discussion is
still welcome on any aspect of the API reboot.