Some questions about the write API

[Rönkkö Mikko]

Hi

I have a couple of questions about the new api.

Question 1:

The documentation on the API states

"Notes and attachments can be made child items by assigning the parent item's key to the parentItem property. If parent and child items are created in the same request, the child items must appear after the parent item in the items array."

How would this work in practice? I cannot know the parent key before the parent is created on the server, right? So what should I use as parent key to create the parent and children in a single call?

Question 2:

The documentations says that you can PUT and PATCH items. PUT completely overwrites the item with the json and PATCH overwrites only the fields that are included in the json. Wouldn't this mean that PATCH and PUT do the same thing if the complete item json is included in the request. If yes, is there any reason to use the PUT request because the PATCH request can do the same and also supports partial downloads?

Question 3:

The new API supports throttling, and states when the client should pause starting requests. Are there any guidelines on how many request per second or how many parallel requests would be good default values to minimise the risk of being throttled?

Question 4:

The documentation states that write requests return 409 when the target library is locked. What does library locking mean and when does a library become locked or unlocked?

Mikko

[Dan Stillman]

All good questions.

> I have a couple of questions about the new api.
>
> Question 1:
>
> The documentation on the API states
>
> "Notes and attachments can be made child items by assigning the parent item's key to the parentItem property. If parent and child items are created in the same request, the child items must appear after the parent item in the items array."
>
> How would this work in practice? I cannot know the parent key before the parent is created on the server, right? So what should I use as parent key to create the parent and children in a single call?

This document is partially a specification for the Zotero client, where
object keys are created locally when the objects are created, such that
you have them both at upload time. I put off the question of whether
third-party clients should create keys locally as well, but if they do
we'll need to define valid keys in the spec. The alternative, obviously,
is to create top-level items, get back the keys, and then make follow-up
requests with child items. But we can probably just define the key
format in the spec.

> Question 2:
>
> The documentations says that you can PUT and PATCH items. PUT completely overwrites the item with the json and PATCH overwrites only the fields that are included in the json. Wouldn't this mean that PATCH and PUT do the same thing if the complete item json is included in the request. If yes, is there any reason to use the PUT request because the PATCH request can do the same and also supports partial downloads?

Partial uploads, you mean? Yes, if you're always sending the full item
JSON, with empty fields for empty items, then PUT and PATCH would be
equivalent. I think the only reason to use PUT would be if you didn't
want to store/send empty fields and also didn't want to track changes.
(It's useful for testing, at the very least.)

> Question 3:
>
> The new API supports throttling, and states when the client should pause starting requests. Are there any guidelines on how many request per second or how many parallel requests would be good default values to minimise the risk of being throttled?

Not yet. (We're not actually throttling yet.) We should have more
guidance later, but the general idea is that, as long as you're watching
for Backoff and not making too many parallel requests (which would
prevent you from responding quickly enough to a Backoff), your client
should be able to scale up and down automatically without hitting 429.
There might be some sort of abuse threshold that punishes clients more
severely, but mostly the Backoff would just put you at the maximum
average rate over a longer period.

We'll have to see how this works in practice, but I'd say 4-6 parallel
requests is a good place to start. (Firefox seems to limits max
persistent connections per server to 6 by default.)

> Question 4:
>
> The documentation states that write requests return 409 when the target library is locked. What does library locking mean and when does a library become locked or unlocked?

It means a classic sync is in progress for the target library. That will
likely go away once classic syncing is gone.

[Rönkkö Mikko]

Hi

A follow-up question

On Feb 22, 2013, at 11:32 PM, Dan Stillman <dsti...@zotero.org> wrote:

> All good questions.
>
>> I have a couple of questions about the new api.
>>
>> Question 1:
>>
>> The documentation on the API states
>>
>> "Notes and attachments can be made child items by assigning the parent item's key to the parentItem property. If parent and child items are created in the same request, the child items must appear after the parent item in the items array."
>>
>> How would this work in practice? I cannot know the parent key before the parent is created on the server, right? So what should I use as parent key to create the parent and children in a single call?
>
> This document is partially a specification for the Zotero client, where object keys are created locally when the objects are created, such that you have them both at upload time. I put off the question of whether third-party clients should create keys locally as well, but if they do we'll need to define valid keys in the spec. The alternative, obviously, is to create top-level items, get back the keys, and then make follow-up requests with child items. But we can probably just define the key format in the spec.

The key spec and the rules how to handle key collisions would be useful. This would simplify creating a parent and a child considerably.

Mikko

>
>> Question 2:
>>
>> The documentations says that you can PUT and PATCH items. PUT completely overwrites the item with the json and PATCH overwrites only the fields that are included in the json. Wouldn't this mean that PATCH and PUT do the same thing if the complete item json is included in the request. If yes, is there any reason to use the PUT request because the PATCH request can do the same and also supports partial downloads?
>
> Partial uploads, you mean? Yes, if you're always sending the full item JSON, with empty fields for empty items, then PUT and PATCH would be equivalent. I think the only reason to use PUT would be if you didn't want to store/send empty fields and also didn't want to track changes. (It's useful for testing, at the very least.)

>
>> Question 3:
>>
>> The new API supports throttling, and states when the client should pause starting requests. Are there any guidelines on how many request per second or how many parallel requests would be good default values to minimise the risk of being throttled?
>
> Not yet. (We're not actually throttling yet.) We should have more guidance later, but the general idea is that, as long as you're watching for Backoff and not making too many parallel requests (which would prevent you from responding quickly enough to a Backoff), your client should be able to scale up and down automatically without hitting 429. There might be some sort of abuse threshold that punishes clients more severely, but mostly the Backoff would just put you at the maximum average rate over a longer period.
>
> We'll have to see how this works in practice, but I'd say 4-6 parallel requests is a good place to start. (Firefox seems to limits max persistent connections per server to 6 by default.)

>
>> Question 4:
>>
>> The documentation states that write requests return 409 when the target library is locked. What does library locking mean and when does a library become locked or unlocked?
>
> It means a classic sync is in progress for the target library. That will likely go away once classic syncing is gone.
>

> --
> You received this message because you are subscribed to the Google Groups "zotero-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.
> To post to this group, send email to zoter...@googlegroups.com.
> Visit this group at http://groups.google.com/group/zotero-dev?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Rönkkö Mikko]

On Feb 25, 2013, at 9:06 AM, Rönkkö Mikko <mikko....@aalto.fi> wrote:

> Hi
>
> A follow-up question
>
> On Feb 22, 2013, at 11:32 PM, Dan Stillman <dsti...@zotero.org> wrote:
>
>> All good questions.
>>
>>> I have a couple of questions about the new api.
>>>
>>> Question 1:
>>>
>>> The documentation on the API states
>>>
>>> "Notes and attachments can be made child items by assigning the parent item's key to the parentItem property. If parent and child items are created in the same request, the child items must appear after the parent item in the items array."
>>>
>>> How would this work in practice? I cannot know the parent key before the parent is created on the server, right? So what should I use as parent key to create the parent and children in a single call?
>>
>> This document is partially a specification for the Zotero client, where object keys are created locally when the objects are created, such that you have them both at upload time. I put off the question of whether third-party clients should create keys locally as well, but if they do we'll need to define valid keys in the spec. The alternative, obviously, is to create top-level items, get back the keys, and then make follow-up requests with child items. But we can probably just define the key format in the spec.
>

> The key spec and the rules how to handle key collisions would be useful. This would simplify creating a parent and a child considerably.'

I am now implementing the API v2 in ZotPad and the key specification would be useful.

The documentation is a bit unclear on how creating an item and its children with a single call would be possible: The example request for creating an item in the server does not contain an item key. (http://www.zotero.org/support/dev/server_api/v2/write_requests#creating_an_item) This would mean that the item key would not be known at the time the request is made and could not be used as a parent key for the child.

Can you provide an example request that creates an item and a child for that item with a single call?

[Dan Stillman]

On 3/4/13 9:26 AM, Rönkkö Mikko wrote:
> On Feb 25, 2013, at 9:06 AM, Rönkkö Mikko <mikko....@aalto.fi> wrote:
>> On Feb 22, 2013, at 11:32 PM, Dan Stillman <dsti...@zotero.org> wrote:
>>
>>>> The documentation on the API states
>>>>
>>>> "Notes and attachments can be made child items by assigning the parent item's key to the parentItem property. If parent and child items are created in the same request, the child items must appear after the parent item in the items array."
>>>>
>>>> How would this work in practice? I cannot know the parent key before the parent is created on the server, right? So what should I use as parent key to create the parent and children in a single call?
>>> This document is partially a specification for the Zotero client, where object keys are created locally when the objects are created, such that you have them both at upload time. I put off the question of whether third-party clients should create keys locally as well, but if they do we'll need to define valid keys in the spec. The alternative, obviously, is to create top-level items, get back the keys, and then make follow-up requests with child items. But we can probably just define the key format in the spec.
>> The key spec and the rules how to handle key collisions would be useful. This would simplify creating a parent and a child considerably.'
> I am now implementing the API v2 in ZotPad and the key specification would be useful.
>
> The documentation is a bit unclear on how creating an item and its children with a single call would be possible: The example request for creating an item in the server does not contain an item key. (http://www.zotero.org/support/dev/server_api/v2/write_requests#creating_an_item) This would mean that the item key would not be known at the time the request is made and could not be used as a parent key for the child.

http://www.zotero.org/support/dev/server_api/v2/syncing#iv_upload_modified_data
links to
http://www.zotero.org/support/dev/server_api/v2/write_requests#updating_multiple_objects,
which shows how to create/update objects with keys. The example there
shows creating a child collection in the same request as its parent; the
process for items would be the same. (I'm not sure I have to test to
ensure this works, but it should in theory.)

I'll add keys to the documentation, but in the meantime see
Zotero.ID.getKey() for the characters we currently use:

https://github.com/zotero/zotero/blob/3.1/chrome/content/zotero/xpcom/id.js#L89

(Should be everything except 0, 1, and O, but I left a couple other
letters out by mistake. Once 3.0.11 is no longer actively syncing we can
start using the others.)

[Rönkkö Mikko]

On Mar 4, 2013, at 11:49 PM, Dan Stillman <dsti...@zotero.org>

This clarifies things. But I also have a couple of follow-up questions.

Are the keys globally unique:
- Can different libraries contain the same key?
- Can the same key be used for a collection and an item, or are the keys unique so that if a key is used for an item, it cannot be used for a collection any more?

How does Zotero deal with key collisions? It is theoretically possible that a client generates a key that already exists on the server and tries to upload that. How does the server respond in that case?

[Dan Stillman]

On 3/5/13 12:25 AM, Rönkkö Mikko wrote:
> On Mar 4, 2013, at 11:49 PM, Dan Stillman <dsti...@zotero.org>
> wrote:
>
>> http://www.zotero.org/support/dev/server_api/v2/syncing#iv_upload_modified_data links to http://www.zotero.org/support/dev/server_api/v2/write_requests#updating_multiple_objects, which shows how to create/update objects with keys. The example there shows creating a child collection in the same request as its parent; the process for items would be the same. (I'm not sure I have to test to ensure this works, but it should in theory.)
>>
>> I'll add keys to the documentation, but in the meantime see Zotero.ID.getKey() for the characters we currently use:
>>
>> https://github.com/zotero/zotero/blob/3.1/chrome/content/zotero/xpcom/id.js#L89
>>
>> (Should be everything except 0, 1, and O, but I left a couple other letters out by mistake. Once 3.0.11 is no longer actively syncing we can start using the others.)
> This clarifies things. But I also have a couple of follow-up questions.
>
> Are the keys globally unique:

No.

> - Can different libraries contain the same key?

Yes.

> - Can the same key be used for a collection and an item

Yes.

> How does Zotero deal with key collisions? It is theoretically possible that a client generates a key that already exists on the server and tries to upload that. How does the server respond in that case?

There'd be a version conflict, since the client would pass
[object]Version: 0 but there'd be a version on the server already. I've
clarified this a bit in the documentation, including indicating that
clients shouldn't use If-Unmodified-Since-Version for multi-object
requests if they don't know the full state of the server library for
that object type.

[Rönkkö Mikko]

On Mar 5, 2013, at 9:21 AM, Dan Stillman <dsti...@zotero.org>
wrote:

How does Zotero handle this? I would assume that you check the keys when receiving data from the server and recreate local keys if server returns a conflicting key? This would prevent the conflict happening, right?

[Dan Stillman]

On 3/5/13 2:32 AM, Rönkkö Mikko wrote:
>>> How does Zotero deal with key collisions? It is theoretically possible that a client generates a key that already exists on the server and tries to upload that. How does the server respond in that case?
>> There'd be a version conflict, since the client would pass [object]Version: 0 but there'd be a version on the server already. I've clarified this a bit in the documentation, including indicating that clients shouldn't use If-Unmodified-Since-Version for multi-object requests if they don't know the full state of the server library for that object type.
> How does Zotero handle this? I would assume that you check the keys when receiving data from the server and recreate local keys if server returns a conflicting key? This would prevent the conflict happening, right?

If the client has a local item with a given key, then either the server
already has that item (in which case another one can't come in) or it's
set to upload, which, since the data wouldn't match, would result in a
conflict being presented to the user. That conflict would be erroneous,
but the user would at least be alerted to the problem.

But if my math is correct, given the current set of characters there are
850 billion possible permutations, so the odds of this happening in a
given library are pretty low.

[Aurimas Vinckevicius]

So the only time when a key clash is possible is when two clients add an item with the same key before each is able to sync to the server. I understand that the odds of this are ridiculously low, but what if it does happen and the user is notified of the conflict. The items would, presumably, be completely different, so the current conflict resolution window would be of little help. Perhaps we should add an option to regenerate a new key for the local item. It could be an option for the local file that says "Treat as a separate item". This might also be useful in some cases where the items are actually the same, but perhaps a merge between the two would be more convenient.

--
You received this message because you are subscribed to the Google Groups "zotero-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+unsubscribe@googlegroups.com.

[Avram Lyon]

UI to deal with key collisions in a set of 850 billion is so unlikely to ever be used that it's simply not worth writing. It'd amount to unreachable code and would therefore be undertested and would be better off not being written.

To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.

[Dan Stillman]

On 3/5/13 3:32 AM, Aurimas Vinckevicius wrote:

So the only time when a key clash is possible is when two clients add an item with the same key before each is able to sync to the server. I understand that the odds of this are ridiculously low, but what if it does happen and the user is notified of the conflict. The items would, presumably, be completely different, so the current conflict resolution window would be of little help. Perhaps we should add an option to regenerate a new key for the local item. It could be an option for the local file that says "Treat as a separate item". This might also be useful in some cases where the items are actually the same, but perhaps a merge between the two would be more convenient.

I'm pretty unconcerned with the key clash scenario. In the incredibly unlikely chance that it happened, the user could always cancel and duplicate the local item, and chalk it up to a bug (albeit a disconcerting one).

If we thought a "Keep Both" made sense generally, we could add that, but I don't know that it's worth it. (We can't add it to the current sync code, in any case, so this would have to wait for API syncing.)

Not sure if you were suggesting it as a new feature, but a merge between the two versions is already possible in the merge window.

[Aurimas Vinckevicius]

Not exactly applicable to this "problem", but I found this post interesting and somewhat amusing, since the code in the answer is an exact replica of the code in Zotero.

http://stackoverflow.com/questions/2344312/how-is-randomness-achieved-with-math-random-in-javascript/3357323#3357323

--
You received this message because you are subscribed to the Google Groups "zotero-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.

[Rönkkö Mikko]

On Mar 5, 2013, at 10:55 AM, Dan Stillman <dsti...@zotero.org> wrote:

On 3/5/13 3:32 AM, Aurimas Vinckevicius wrote:

So the only time when a key clash is possible is when two clients add an item with the same key before each is able to sync to the server. I understand that the odds of this are ridiculously low, but what if it does happen and the user is notified of the conflict. The items would, presumably, be completely different, so the current conflict resolution window would be of little help. Perhaps we should add an option to regenerate a new key for the local item. It could be an option for the local file that says "Treat as a separate item". This might also be useful in some cases where the items are actually the same, but perhaps a merge between the two would be more convenient.

I'm pretty unconcerned with the key clash scenario. In the incredibly unlikely chance that it happened, the user could always cancel and duplicate the local item, and chalk it up to a bug (albeit a disconcerting one).

If we thought a "Keep Both" made sense generally, we could add that, but I don't know that it's worth it. (We can't add it to the current sync code, in any case, so this would have to wait for API syncing.)

I think that this would be really problematic to implement, because one of the items would need to have its key changed. If both items were already used in a document, the item keys stored in the documents would need to be changed as well. This could be of course achieved by changing the keys of both items, and then have the user to choose replacement items in the word processor plugin, but this is probably not worth the effort. 

Not sure if you were suggesting it as a new feature, but a merge between the two versions is already possible in the merge window.

On Tue, Mar 5, 2013 at 1:59 AM, Dan Stillman <dsti...@zotero.org> wrote:

On 3/5/13 2:32 AM, Rönkkö Mikko wrote:

How does Zotero deal with key collisions? It is theoretically possible that a client generates a key that already exists on the server and tries to upload that. How does the server respond in that case?

There'd be a version conflict, since the client would pass [object]Version: 0 but there'd be a version on the server already. I've clarified this a bit in the documentation, including indicating that clients shouldn't use If-Unmodified-Since-Version for multi-object requests if they don't know the full state of the server library for that object type.

How does Zotero handle this? I would assume that you check the keys when receiving data from the server and recreate local keys if server returns a conflicting key? This would prevent the conflict happening, right?

If the client has a local item with a given key, then either the server already has that item (in which case another one can't come in) or it's set to upload, which, since the data wouldn't match, would result in a conflict being presented to the user. That conflict would be erroneous, but the user would at least be alerted to the problem.

But if my math is correct, given the current set of characters there are 850 billion possible permutations, so the odds of this happening in a given library are pretty low.

--
You received this message because you are subscribed to the Google Groups "zotero-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.

[Rönkkö Mikko]

On Mar 5, 2013, at 9:21 AM, Dan Stillman <dsti...@zotero.org> wrote:

> On 3/5/13 12:25 AM, Rönkkö Mikko wrote:
>> On Mar 4, 2013, at 11:49 PM, Dan Stillman <dsti...@zotero.org>
>> wrote:
>>
>>> http://www.zotero.org/support/dev/server_api/v2/syncing#iv_upload_modified_data links to http://www.zotero.org/support/dev/server_api/v2/write_requests#updating_multiple_objects, which shows how to create/update objects with keys. The example there shows creating a child collection in the same request as its parent; the process for items would be the same. (I'm not sure I have to test to ensure this works, but it should in theory.)
>>>
>>> I'll add keys to the documentation, but in the meantime see Zotero.ID.getKey() for the characters we currently use:
>>>
>>> https://github.com/zotero/zotero/blob/3.1/chrome/content/zotero/xpcom/id.js#L89
>>>
>>> (Should be everything except 0, 1, and O, but I left a couple other letters out by mistake. Once 3.0.11 is no longer actively syncing we can start using the others.)
>> This clarifies things. But I also have a couple of follow-up questions.
>>
>> Are the keys globally unique:
>
> No.
>
>> - Can different libraries contain the same key?
>
> Yes.

While this may be the intention, this is probably not the case in the current Zotero code. The items table has itemID as the primary key, so having the same item key in two libraries would not work.

Two identical item keys in different libraries would also break file storage because the files are stored by the key only, not the library id.

>
>> - Can the same key be used for a collection and an item
>
> Yes.
>
>> How does Zotero deal with key collisions? It is theoretically possible that a client generates a key that already exists on the server and tries to upload that. How does the server respond in that case?
>
> There'd be a version conflict, since the client would pass [object]Version: 0 but there'd be a version on the server already. I've clarified this a bit in the documentation, including indicating that clients shouldn't use If-Unmodified-Since-Version for multi-object requests if they don't know the full state of the server library for that object type.
>

[Dan Stillman]

On 3/7/13 2:37 AM, Rönkkö Mikko wrote:
> On Mar 5, 2013, at 9:21 AM, Dan Stillman <dsti...@zotero.org> wrote:
>
>> On 3/5/13 12:25 AM, Rönkkö Mikko wrote:
>>> Are the keys globally unique:
>> No.
>>
>>> - Can different libraries contain the same key?
>> Yes.
> While this may be the intention, this is probably not the case in the current Zotero code. The items table has itemID as the primary key, so having the same item key in two libraries would not work.

itemID isn't the item key. It's just an integer that uniquely identifies
the item in the local database (quickly). `key` is the eight-character
string, and in combination with a libraryID it uniquely identifies the
item globally.

> Two identical item keys in different libraries would also break file storage because the files are stored by the key only, not the library id.

That much is true. The folder scheme predated libraries, and we never
dealt with this. But I was really referring to server-side uniqueness.
(We should fix this at some point, though.)

[Rönkkö Mikko]

On Mar 7, 2013, at 10:04 AM, Dan Stillman <dsti...@zotero.org> wrote:

> On 3/7/13 2:37 AM, Rönkkö Mikko wrote:
>> On Mar 5, 2013, at 9:21 AM, Dan Stillman <dsti...@zotero.org> wrote:
>>
>>> On 3/5/13 12:25 AM, Rönkkö Mikko wrote:
>>>> Are the keys globally unique:
>>> No.
>>>
>>>> - Can different libraries contain the same key?
>>> Yes.
>> While this may be the intention, this is probably not the case in the current Zotero code. The items table has itemID as the primary key, so having the same item key in two libraries would not work.
>
> itemID isn't the item key. It's just an integer that uniquely identifies the item in the local database (quickly). `key` is the eight-character string, and in combination with a libraryID it uniquely identifies the item globally.

My mistake, sorry.

>
>> Two identical item keys in different libraries would also break file storage because the files are stored by the key only, not the library id.
>
> That much is true. The folder scheme predated libraries, and we never dealt with this. But I was really referring to server-side uniqueness. (We should fix this at some point, though.)

Do you have any ideas on what the fix might be? Using the itemIDs for storage folder names?

[Dan Stillman]

On 3/7/13 4:42 AM, Rönkkö Mikko wrote:
> On Mar 7, 2013, at 10:04 AM, Dan Stillman <dsti...@zotero.org> wrote:
>
>> On 3/7/13 2:37 AM, Rönkkö Mikko wrote:
>>> Two identical item keys in different libraries would also break file storage because the files are stored by the key only, not the library id.
>> That much is true. The folder scheme predated libraries, and we never dealt with this. But I was really referring to server-side uniqueness. (We should fix this at some point, though.)
> Do you have any ideas on what the fix might be? Using the itemIDs for storage folder names?

We actually used to use itemIDs years ago, but we use keys because
they're stable. Among other things, if you do a Restore from Zotero
Server, items get new itemIDs, but they keep the same keys.

The fix would probably be to move the item directories into
library-specific subdirectories under 'storage'. But if we're changing
things, we should probably consider switching to human-readable
directory names at the same time, though that's obviously a
significantly harder problem to solve.

(We should probably also move the default data directory location out of
the Firefox profile and into the system Documents folder. With the two
different versions of Zotero, it makes much less sense for the data to
be in the profile directory by default. Also, I'm not sure if this was
always the case, but I realized recently that the Zotero data directory
isn't actually indexed by Spotlight on OS X.)

We should start a new thread to discuss this stuff further, though.