api-based syncing

[Dan Stillman]

Happy New Year Zoterons,

We're planning a major server and client change that I wanted to share
with the list.

We're in the process of adding to the server API the functionality
necessary to fully and efficiently support API-based syncing from the
Zotero client and other third-party clients. Once this is done, we'll be
rewriting the Zotero client to use the API exclusively and stop using
the private sync protocol we've used up until now.

This is a major undertaking, but it will have significant benefits,
including enabling equivalent sync functionality in all clients, the
ability for Zotero client users to sync much larger libraries and have
better failure scenarios (e.g., a problem with one item in one library
won't break syncing of all items in all of a user's libraries, at the
cost of atomicity), a huge simplification of server code with easier
maintenance and better testing coverage, and API access to all library
data. Having a single interface that we use in the client ourselves will
allow us to focus on improvements that benefit all API consumers, most
notably with regard to performance.

We're a ways off from launching this, but we've created a document that
outlines the sync process we plan to use in the Zotero client as well as
some possible variations for different use cases:

http://www.zotero.org/support/dev/server_api/syncing

The document is incomplete and untested, and it may contain some logical
flaws, but it gives an idea of the new functionality that will be available.

I've been working on the new server code in a local branch, but within
the next few weeks I'm planning to push it to a branch on GitHub that
will be accessible to API clients via an HTTP header for testing. I'll
post here when that's available.

We think we can do this with minimal impact on existing clients, though
some clients may need a couple tweaks. We'll provide advance warning if
that's the case.

Let me know if you have any questions or feedback.

- Dan

[Katie]

This is fantastic news. Really looking forward to this feature. I was just about to post about this very issue. I hope to be able to request a list of items and their modifications (edit, add, removed) given a date. On a related note, it would be great to have item lists and collection/tag information on the same request page. Essentially I'm hoping to cut down on the number of requests made, and therefore cut down on sync/processing time.

[Alf Eaton]

On Wednesday, 2 January 2013 07:03:16 UTC, Dan Stillman wrote:

We're in the process of adding to the server API the functionality
necessary to fully and efficiently support API-based syncing from the
Zotero client and other third-party clients. Once this is done, we'll be
rewriting the Zotero client to use the API exclusively and stop using
the private sync protocol we've used up until now.

This sounds great, Dan. Are there any plans to add push messaging (e.g. Google's Cloud Messaging API) for sending updates to clients, so that they can update instantly in response to library changes on the server without having to poll too often?

Alf

[Dan Stillman]

On 1/18/13 6:51 AM, Alf Eaton wrote:

Are there any plans to add push messaging (e.g. Google's Cloud Messaging API) for sending updates to clients, so that they can update instantly in response to library changes on the server without having to poll too often?

As a matter of fact, yes. We'll be offering a WebSocket endpoint that will allow API clients to be notified of updates immediately. I have a proof-of-concept of this working internally, and I'll be posting documentation soon.

[Dan Stillman]

I'm happy to announce that the new server API version is now available.

Documentation is here:

http://www.zotero.org/support/dev/server_api/v2/

I believe I've covered most of the changes here:

http://www.zotero.org/support/dev/server_api/v2/changes_from_v1

If you're making read requests, you might need not to change anything
(beyond perhaps ignoring a few new JSON properties), though you should
be sure to read the section on conditional reads and caching. If you're
writing to the API, you'll need to make some adjustments.

As I noted in another thread, the new version is not currently the
default, and we've tried to keep the current version the same, so
existing API users shouldn't notice any changes as long as they're
making valid requests. If you do notice something not working in the old
version, please let us know.

To access the new API version, include the HTTP header
"Zotero-API-Version: 2" in all requests. We haven't yet decided when
exactly we'll be making version 2 the default, but it'll likely be
sometime around the beginning of April. I'm also not sure if or how long
we'll keep version 1 running once we make the switch, but we'll give
plenty of advance warning here in any case.

There may be some things that can be clarified or that still need to be
updated in the documentation, so let me know if you have any questions.
We're looking forward to seeing what people do with the new functionality.

- Dan