API v3: all-JSON API response format and other changes

[Dan Stillman]

We've gotten a number of requests over the years for an all-JSON
response format from the API. I'm happy to announce that we're putting
the finishing touches on a new version of the API that will return
all-JSON responses by default, and I wanted to share some details.

First, a bit of background. We originally designed the API with Atom as
the default response format. Atom has a number of benefits, among them
implicit feed reader/library support, pagination, and a content
mechanism for returning different content types in the response (which
we've extended to support multiple types). But XML is heavy and hard to
read, and since content=json, which didn't exist originally, is now the
most common mode, most responses now require two separate parsers. With
all-JSON responses, we can have a single lightweight format that's easy
to read and easy to parse. It should also allow for better server-side
caching, which will help us speed up requests.

We're also taking this opportunity to clean up a bunch of other things
that bother us about the current API.

We're currently planning to keep format=atom around — for feed reader
support if nothing else — but we're going to be recommending that most
API consumers switch to format=json (and I suspect most will want to).
API versions 1 and 2 will remain available for the time being for
existing clients, and our goal is for existing format=atom code to
continue to work in v3 with very few changes.

Here's a sample of the new format in its current draft form:

https://gist.github.com/dstillman/7a6f7aa01840bce57f8a

Some notes about v3:

- format=json is the new default, with a single JSON object for
single-object requests and an array of objects for multi-object requests.

- All individual objects contain top-level 'key' and 'version'
properties and top-level 'links', 'library', and 'meta' objects. 'meta'
contains some of the non-editable system-generated properties like
createdByUser/lastModifiedByUser (for group items), creatorSummary, and
numChildren.

- There's a 'parsedDate' property in 'meta' that gives the full parsed
date in YYYY-MM-DD form, so that clients don't need to replicate our
date-parsing logic to get exact dates. In Atom v3, zapi:parsedDate
replaces zapi:year.

- zapi:numTags is removed in Atom v3, since it's unnecessary with the
'tags' array in the editable json.

- For format=json requests, include=fields has replaced Atom's
content=json and is now the default mode, with a top-level 'fields'
object containing the editable fields. As with 'content', additional
comma-separated types can be requested (e.g., include=fields,bib). The
requested types ('fields', 'bib', 'ris') will all be top-level
properties. content=html will remain the default in Atom.

- Total result count for multi-object responses is available in a new
custom HTTP header, Total-Results. zapi:totalResults will be removed
from Atom responses in v3.

- For format=json, rel=first/prev/next/last/alternate links for
multi-object responses will be in the Link HTTP header. Using the Link
header for pagination may allow us to support pagination on additional
requests, such as format=keys.

- Other Atom-specific feed properties (title, author, published,
updated) are omitted in format=json.

- The API key can be provided in the Authorization request header (exact
details forthcoming) instead of the 'key' query parameter to keep it out
of request URLs. Since API keys have never been included in the URLs
provided in responses, previously all provided URLs have had to be
modified for key-based access.

- The API version can be provided as a query parameter ('v=3') instead
of the Zotero-API-Version header for easier debugging and sharing of
requests, though both will remain supported.

- For formats other than Atom, dateModified descending is the default
sort instead of dateAdded descending.

- 'itemKey'/'itemVersion' (and similar properties on collections and
searches) in the editable JSON are now just 'key' and 'version' for
easier handling by clients. Most clients are likely just passing back
the edited JSON without touching those properties, so I'm hoping this
won't break anything. If you are storing the JSON, you'd need to modify
it before sending in v3.

- Related to the previous item, we will be recommending that clients
that do store editable JSON save the current API version alongside the
JSON. When the clients are later updated for an API version that changes
how passed JSON is handled, they'll need to make changes to the stored
JSON or retrieve it again before sending it up.

- dateAdded/dateModified are included in the 'fields' object in ISO 8601
form. Previously these timestamps were provided only in the Atom
published/updated elements, though in v2 they can be sent back in the
JSON as dateAdded/dateModified in SQL DATETIME format, interpreted as
UTC. In v3 write requests we'll accept either, though SQL DATETIME will
be deprecated.

- The 'Accessed' field, which is also currently an SQL DATETIME, will
also be changing to ISO 8601 in v3 (including in Atom) for both reading
and writing. SQL DATETIME will be accepted but deprecated.

- pprint=1 has been removed, and all JSON responses are now
pretty-printed. (pprint also triggered Content-Type: text/plain, but
browsers now display application/json by default, and we encourage
people to use cURL or similar for testing other requests.)

- There are a lot of 'links' objects — and the API responses would look
a lot cleaner without them — but we think they all provide some value,
either in user/group object requests or (for 'library' in particular) in
eventual global item requests.

- The feed-level pagination links (rel=self/first/prev/next/last) are
meant to be used directly, as long as you're using the Authorization
header. The rel=self links in individual objects are meant as base URIs
and do not include any query parameters (e.g., include=fields,bib). This
is a change from the current behavior, where the Atom entry rel="self"
links include all non-default provided parameters. (Among other things,
this allows for better caching.) But with the Authorization header and
include=fields as the new default, the base URI may be sufficient for
most individual-object requests.

- The 'newer' parameter is now 'since' for clarity. 'newer' is
deprecated but will continue to work.

- The 'order' parameter is now 'sort' and 'sort' is now 'direction' for
clarity, since even I can't keep 'sort' and 'order' straight.
order=<field> and sort=<asc/desc> are deprecated but will continue to work.

Also:

- We were previously escaping '<', '>', and '&' to \u.... in the
content=json objects (I guess to avoid having to XML-encode those
characters, though I'm not sure why that was any less desirable). To
avoid having to process JSON separately by API version throughout the
code, I'd like to stop escaping those characters for all API versions. I
don't think it should cause problems for existing clients, since these
characters have no special meaning in JSON and are just unnecessarily
escaped now, but I wanted to mention it. For JSON in Atom responses
those characters will be turned into XML numeric character references
instead.

Other changes we're considering:

- Use the HTTP Warning header to send clients non-fatal warnings — such
as deprecation warnings — that could be logged.

- Changes to 'relations', but that's probably a longer-term project

Whew! Other than that, nothing new to report.

Let me know if you have any questions or feedback on any of this, or if
there are any other pain points with the current API that you'd like
addressed. This is mostly already implemented, so we'll likely be
releasing v3 within a few weeks along with copious documentation of the
changes. v1 will remain the default for the time being, and we'll give
ample notice before changing the default, though we recommend using an
explicit version in released code. We'll be changing the default from v1
directly to v3. Poor v2 never got its chance to shine.

- Dan