Help needed on attachment upload (II)
101 views
Skip to first unread message
Frank Bennett
Feb 20, 2020, 11:41:09 AM2/20/20
to zotero-dev
Starting a new thread for this, because I'm hitting fresh problems, and I'm completely flummoxed.
Following the steps at https://www.zotero.org/support/dev/web_api/v3/file_upload, I am able to complete an attachment upload without error, but there are serious anomalies:
1. The files uploaded (just two, for testing) show up on the server view greyed out, with metadata but no content.
2. Syncing a client to the library does not pull the new attachments. The old attachment (deleted via API) remains, but its file no longer exists.
So it's not working correctly, and I must be doing something wrong. Several things are not working as I understand they should from the documentation:
a. At steps 2 and 3(a)(ii), I can't use the If-None-Match: * header without getting an error message that the file already exists. If I in both I use If-Match: <hash>, where hash is the md5 of the file declared in step 1(a)(ii), the process clears -- but that's probably broken? I don't understand why it would see a file existing before the actual upload.
b. I was able to get past the file-size error in the other thread by casting the prefix and affix as node Buffer, catenating the buffer, and feeding that as the POST data in the node client I'm using (axios). By the axios docs, that is supposed to be legal; but I get no file content. It's a great mystery.
The attachments on the item used for testing may have gotten into a funny state that is compounding the confusion. Here is the item (showing one broken attachment here on sync, and two broken attachments in the online view):
If the item can be checked for sanity, that would be a great help. Also, if anyone has successfully used tools in node.js to complete an attachment upload, I would be glad to receive any advice. As Zotero is written in JS, I had assumed that this would be relatively straightforward, but it hasn't quite turned out that way.
FB
Stephan Hügel
Feb 20, 2020, 12:09:55 PM2/20/20
to zotero-dev
Speaking from (verging on the traumatic) personal experience, and noting that two people seem to be having similar problems with file uploads at the moment, I'd like to suggest two things:
1. Could the docs be amended in some way that makes it easier for implementors to debug their work? Perhaps with a reference file and correct expected values for hashes, file content, headers etc where appropriate. This is something I really struggled with when implementing uploads for Pyzotero.
2. Frank, if you want to post a reference file somewhere, I can instrument Pyzotero to show hashes, headers, content etc at each stage, and you could compare them to your own values in order to debug them?
Dan Stillman
Feb 20, 2020, 5:50:41 PM2/20/20
to zoter...@googlegroups.com
On 2/20/20 11:41 AM, Frank Bennett wrote:
> So it's not working correctly, and I must be doing something wrong.
> Several things are not working as I understand they should from the
> documentation:
>
> a. At steps 2 and 3(a)(ii), I can't use the If-None-Match: * header
> without getting an error message that the file already exists. If I in
> both I use If-Match: <hash>, where hash is the md5 of the file
> declared in step 1(a)(ii), the process clears -- but that's probably
> broken? I don't understand why it would see a file existing before the
> actual upload.
From 1(a)(ii):
> So it's not working correctly, and I must be doing something wrong.
> Several things are not working as I understand they should from the
> documentation:
>
> a. At steps 2 and 3(a)(ii), I can't use the If-None-Match: * header
> without getting an error message that the file already exists. If I in
> both I use If-Match: <hash>, where hash is the md5 of the file
> declared in step 1(a)(ii), the process clears -- but that's probably
> broken? I don't understand why it would see a file existing before the
> actual upload.
"md5 and mtime can be edited directly in personal libraries for
WebDAV-based file syncing. They should not be edited directly when using
Zotero File Storage, which provides an atomic method (detailed below)
for setting the properties along with the corresponding file."
By providing those values manually before the upload, you're putting it
into a state where If-Match is expected to match the existing value.
> The attachments on the item used for testing may have gotten into a
> funny state that is compounding the confusion. Here is the item
> (showing one broken attachment here on sync, and two broken
> attachments in the online view):
>
> https://www.zotero.org/groups/2318535/euro-expert/collections/VXLTUX65/items/5WRUGIFB
>
> If the item can be checked for sanity, that would be a great help.
trash) have an mtime and hash set with no file associated. Not sure
exactly what you did after 1(a)(ii) above, but the mtime and hash will
get set automatically when you register the file upload (or when you get
an 'exists' response).
Dan Stillman
Feb 20, 2020, 5:51:09 PM2/20/20
to zoter...@googlegroups.com
On 2/20/20 12:09 PM, Stephan Hügel wrote:
> Speaking from (verging on the traumatic) personal experience, and
> noting that two people seem to be having similar problems with file
> uploads at the moment, I'd like to suggest two things:
>
> 1. Could the docs be amended in some way that makes it easier for
> implementors to debug their work? Perhaps with a reference file and
> correct expected values for hashes, file content, headers etc where
> appropriate. This is something I really struggled with when
> implementing uploads for Pyzotero.
> 2. Frank, if you want to post a reference file somewhere, I can
> instrument Pyzotero to show hashes, headers, content etc at each
> stage, and you could compare them to your own values in order to debug
> them?
The main thing we'd need for debugging the vast majority of API issues
> Speaking from (verging on the traumatic) personal experience, and
> noting that two people seem to be having similar problems with file
> uploads at the moment, I'd like to suggest two things:
>
> 1. Could the docs be amended in some way that makes it easier for
> implementors to debug their work? Perhaps with a reference file and
> correct expected values for hashes, file content, headers etc where
> appropriate. This is something I really struggled with when
> implementing uploads for Pyzotero.
> 2. Frank, if you want to post a reference file somewhere, I can
> instrument Pyzotero to show hashes, headers, content etc at each
> stage, and you could compare them to your own values in order to debug
> them?
would be an HTTP request/response log, as you'd get with curl -v (minus
connection/TLS stuff), with API keys redacted. That should normally let
us determine pretty quickly whether it's a problem with the calling
code, the library, or the API.
Frank Bennett
Feb 20, 2020, 9:13:02 PM2/20/20
to zotero-dev
Many thanks for the advice on mtime and md5. I wasn't sure where the mtime and md5 values later in the steps come from, I guess they are returned in one of the JSON responses. Will chip away at it, but the problem with multiple deletes comes first.
The main thing we'd need for debugging the vast majority of API issues
would be an HTTP request/response log, as you'd get with curl -v (minus
connection/TLS stuff), with API keys redacted. That should normally let
us determine pretty quickly whether it's a problem with the calling
code, the library, or the API.
I've run the steps using curl -v -s, the results are below. In order:
01_library_version.txt
A call to the library containing the target item and attachments. It reports version "5"
02_item_data.txt
A call to the item that is parent of the attachments.
03_child_data.txt
A call to the children of the parent.
04_delete_attempt.txt
A DELETE call attempting to remove the first attachment, with a version threshold of "5".
Boom.
It's demanding version 3054, which doesn't correspond to the library, the parent item, or the target attachment.
^^^^^^^^^^
01_library_version.txt
A call to the library containing the target item and attachments. It reports version "5"
> GET /groups/2318535 HTTP/1.1
> Host: api.zotero.org
> User-Agent: curl/7.47.0
> Accept: */*
> Zotero-API-Key: XXX
>
< HTTP/1.1 200 OK
< Date: Fri, 21 Feb 2020 01:48:00 GMT
< Server: Apache/2.4.41 (Amazon)
...
< Zotero-API-Version: 3
< Zotero-Schema-Version: 3
< Last-Modified-Version: 5
< Vary: Accept-Encoding
< Content-Length: 877
< Content-Type: application/json
<
{ [877 bytes data]
* Connection #0 to host api.zotero.org left intact
{
"id": 2318535,
"version": 5,
"links": {
"self": {
"href": "https://api.zotero.org/groups/2318535",
"type": "application/json"
},
"alternate": {
"href": "https://www.zotero.org/groups/2318535",
"type": "text/html"
}
},
"meta": {
"created": "2019-04-16T13:06:40Z",
"lastModified": "2019-05-08T11:02:32Z",
"numItems": 3867
},
"data": {
"id": 2318535,
"version": 5,
"name": "euro-expert",
"owner": 6204,
"type": "Private",
"description": "",
"url": "",
"libraryEditing": "members",
"libraryReading": "members",
"fileEditing": "members",
"admins": [
5562730
],
"members": [
5713142,
5726168
]
}
}
^^^^^^^^^^
02_item_data.txt
A call to the item that is parent of the attachments.
> GET /groups/2318535/items/5WRUGIFB HTTP/1.1
> Host: api.zotero.org
> User-Agent: curl/7.47.0
> Accept: */*
> Zotero-API-Key: XXX
>
< HTTP/1.1 200 OK
< Date: Fri, 21 Feb 2020 01:48:11 GMT
< Server: Apache/2.4.41 (Amazon)
...
< Zotero-API-Version: 3
< Zotero-Schema-Version: 3
< Last-Modified-Version: 2749
< Vary: Accept-Encoding
< Content-Length: 4833
< Content-Type: application/json
<
{ [4833 bytes data]
{
"key": "5WRUGIFB",
"version": 2749,
"library": {
"type": "group",
"id": 2318535,
"name": "euro-expert",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/2318535",
"type": "text/html"
}
}
},
"links": {
"self": {
"type": "application/json"
},
"alternate": {
"type": "text/html"
}
},
"meta": {
"createdByUser": {
"id": 6204,
"username": "fbennett",
"name": "Frank Bennett",
"links": {
"alternate": {
"href": "https://www.zotero.org/fbennett",
"type": "text/html"
}
}
},
"parsedDate": "2009-11-25",
"numChildren": 6
},
"data": {
"key": "5WRUGIFB",
"version": 2749,
"itemType": "case",
"caseName": "Bibi Sitara v. Amanullah Khan",
"creators": [],
"abstractNote": "This is a civil revision related to an inheritance issue. ...",
"reporter": "",
"reporterVolume": "",
"court": "gbcc",
"docketNumber": "22/2000",
"firstPage": "",
"history": "",
"dateDecided": "2009/11/25",
"language": "en",
"shortTitle": "",
"url* Connection #0 to host api.zotero.org left intact
": "",
"accessDate": "",
"rights": "",
"extra": "mlzsync1:0131{\"extrafields\":{\"callNumber\":\"PK202\",\"jurisdiction\":\"019pk:gilgit.baltistanPakistan|PK|Gilgit Baltistan\",\"reign\":\"Civil Revision\"}}",
"tags": [
{
"tag": "AL:inheritance law"
},
{
"tag": "EP:written report"
},
{
"tag": "IB:expert appointed by the court"
},
{
"tag": "cn:PK"
}
],
"collections": [
"VXLTUX65"
],
"relations": {},
"dateAdded": "2019-11-04T02:37:09Z",
"dateModified": "2020-01-21T14:27:28Z"
}
}
^^^^^^^^^^
03_child_data.txt
A call to the children of the parent.
> GET /groups/2318535/items/5WRUGIFB/children HTTP/1.1
> Host: api.zotero.org
> User-Agent: curl/7.47.0
> Accept: */*
> Zotero-API-Key: XXX
>
< HTTP/1.1 200 OK
< Date: Fri, 21 Feb 2020 01:48:21 GMT
< Server: Apache/2.4.41 (Amazon)
...
< Zotero-API-Version: 3
< Zotero-Schema-Version: 3
< Total-Results: 2
< Link: <https://www.zotero.org/groups/2318535/items/5WRUGIFB/children>; rel="alternate"
< Last-Modified-Version: 3054
< Vary: Accept-Encoding
< Content-Length: 4207
< Content-Type: application/json
<
{ [4207 bytes data]
[
{
"key": "RBWA4Q3U",
"version": 3054,
"library": {
"type": "group",
"id": 2318535,
"name": "euro-expert",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/2318535",
"type": "text/html"
}
}
},
"links": {
"self": {
"type": "application/json"
},
"alternate": {
"type": "text/html"
},
"up": {
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 6204,
"username": "fbennett",
"name": "Frank Bennett",
"links": {
"alternate": {
"href": "https://www.zotero.org/fbennett",
"type": "text/html"
}
}
}
},
"data": {
"key": "RBWA4Q3U",
"version": 3054,
"parentItem": "5WRUGIFB",
"itemType": "attachment",
"linkMode": "imported_file",
"title": "Urdu Summary",
"accessDate": "2020-02-20T16:54:49Z",
"url": "",
"note": "",
"contentType": "application/pdf",
"charset": "",
"filename": "PK202us.pdf",
"md5": "c3b9f0c38e4543489fe1484318738afa",
"mtime": 1582090980000,
"tags": [
{
"tag": "LN:ur"
},
{
"tag": "TY:summary"
}
],
"relations": {},
"dateAdded": "2020-02-20T16:54:49Z",
"dateModified": "2020-02-20T16:54:49Z"
}
},
{
"key": "JWIKNPWD",
"version": 3053,
"library": {
"type": "group",
"id": 2318535,
"name": "euro-expert",
"links": {
"alternate": {
"href": "https://www.zotero.org/groups/2318535",
"type": "text/html"
}
}
},
"links": {
"self": {
"type": "application/json"
},
"alternate": {
"type": "text/html"
},
"up": {
"type": "application/json"
}
},
"meta": {
"createdByUser": {
"id": 6204,
"username": "fbennett",
"name": "Frank Bennett",
"links": {
"alternate": {
"href": "https://www.zotero.org/fbennett",
"type": "text/html"
}
}
}
},
"data": {
"key": "JWIKNPWD",
"version": 3053,
"parentItem": "5WRUGIFB",
"itemType": "attachment",
"linkMode": "imported_file",
"title": "English Judgment",
"accessDate": "2020-02-20T16:54:41Z",
"url": "",
"note": "",
"contentType": "application/pdf",
"charset": "",
"filename": "PK202.pdf",
"md5": "c1c876a254ca03a33d87bd631fbe6f09",
"mtime": 1582090980000,
"tags": [
{
"tag": "LN:en"
}
],
"relations": {},
* Connection #0 to host api.zotero.org left intact
"dateAdded": "2020-02-20T16:54:41Z",
"dateModified": "2020-02-20T16:54:41Z"
}
}
]
^^^^^^^^^^
04_delete_attempt.txt
A DELETE call attempting to remove the first attachment, with a version threshold of "5".
Boom.
> DELETE /groups/2318535/items?itemKey=RBWA4Q3U HTTP/1.1
> Host: api.zotero.org
> User-Agent: curl/7.47.0
> Accept: */*
> Zotero-API-Key: XXX
> If-Unmodified-Since-Version: 5
>
< HTTP/1.1 412 Precondition Failed
< Date: Fri, 21 Feb 2020 01:48:33 GMT
< Server: Apache/2.4.41 (Amazon)
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Zotero-API-Version: 3
< Zotero-Schema-Version: 3
< Vary: Accept-Encoding
< Content-Length: 74
< Content-Type: text/html; charset=UTF-8
<
{ [74 bytes data]
* Connection #0 to host api.zotero.org left intact
Library has been modified since specified version (expected 5, found 3054)
Frank Bennett
Feb 20, 2020, 9:50:39 PM2/20/20
to zotero-dev
(meanwhile deleting individual attachments in separate transactions
works okay, so I'll use that for now and move on to file uploads.)
works okay, so I'll use that for now and move on to file uploads.)
Frank Bennett
Feb 21, 2020, 12:03:30 AM2/21/20
to zotero-dev
No joy with uploads. The traces dumped from the node client (axios)
are below. Two things I don't understand:
1. We need to provide an md5 and mtime with the authorization request.
I'm deriving these values locally. Is that correct?
2. Final registration demands the mtime & md5. If I provide them, the
transaction goes through, but the file is not associated.
// Use template to create attachment
{
"method": "POST",
"headers": {
"content-type": "application/json",
"Zotero-API-Key": "XXX",
"Zotero-Write-Token": "iI9LtHEP7KrQZQK1I9D02L2f2u4rhEMh"
},
"url": "https://api.zotero.org/groups/2318535/items",
"responseType": "json",
"data": [
{
"note": "",
"relations": {},
"contentType": "application/pdf",
"charset": "binary",
"filename": "PK202.pdf",
"md5": null,
"mtime": null,
"parentItem": "5WRUGIFB"
}
]
}
status: 200
// Get upload authorization
{
"method": "POST",
"headers": {
"content-type": "application/x-www-form-urlencoded",
"Zotero-API-Key": "XXX",
"If-None-Match": "*"
},
"url": "https://api.zotero.org/groups/2318535/items/AXHPEVKM/file",
"responseType": "json",
"data": "md5=c1c876a254ca03a33d87bd631fbe6f09&filename=PK202.pdf&filesize=100182&mtime=1582090980000"
}
status: 200
// Upload file content to AWS
{
"method": "POST",
"headers": {
"content-type": "multipart/form-data;
boundary=---------------------------ec1690e492850a5b0ba0039435ce0753",
"Zotero-API-Key": "XXX"
},
"url": "https://zoterofilestorage.s3.us-east-1.amazonaws.com/",
"responseType": "json",
"data": {
"type": "Buffer",
"data": [
<catenated (prefix + contents + suffix) as buffer data>
]
}
}
status: 201
// Register attachment
{
"method": "POST",
"headers": {
"content-type": "application/x-www-form-urlencoded",
"Zotero-API-Key": "XXX",
"If-None-Match": "*"
},
"url": "https://api.zotero.org/groups/2318535/items/AXHPEVKM/file",
"responseType": "json",
"data": "uploadKey=bdb9a0eef0758c59831c73cd5b57a580"
}
Error: Request failed with status code 400
Detail: MD5 hash not provided
are below. Two things I don't understand:
1. We need to provide an md5 and mtime with the authorization request.
I'm deriving these values locally. Is that correct?
2. Final registration demands the mtime & md5. If I provide them, the
transaction goes through, but the file is not associated.
// Use template to create attachment
{
"method": "POST",
"headers": {
"content-type": "application/json",
"Zotero-API-Key": "XXX",
"Zotero-Write-Token": "iI9LtHEP7KrQZQK1I9D02L2f2u4rhEMh"
},
"url": "https://api.zotero.org/groups/2318535/items",
"responseType": "json",
"data": [
{
"itemType": "attachment",
"linkMode": "imported_file",
"title": "English Judgment",
"accessDate": "CURRENT_TIMESTAMP",
"linkMode": "imported_file",
"title": "English Judgment",
"note": "",
"tags": [
{
"tag": "LN:en"
}
],
"collections": [],
{
"tag": "LN:en"
}
],
"relations": {},
"contentType": "application/pdf",
"charset": "binary",
"filename": "PK202.pdf",
"md5": null,
"mtime": null,
"parentItem": "5WRUGIFB"
}
]
}
status: 200
// Get upload authorization
{
"method": "POST",
"headers": {
"content-type": "application/x-www-form-urlencoded",
"Zotero-API-Key": "XXX",
"If-None-Match": "*"
},
"url": "https://api.zotero.org/groups/2318535/items/AXHPEVKM/file",
"responseType": "json",
"data": "md5=c1c876a254ca03a33d87bd631fbe6f09&filename=PK202.pdf&filesize=100182&mtime=1582090980000"
}
status: 200
// Upload file content to AWS
{
"method": "POST",
"headers": {
"content-type": "multipart/form-data;
boundary=---------------------------ec1690e492850a5b0ba0039435ce0753",
"Zotero-API-Key": "XXX"
},
"url": "https://zoterofilestorage.s3.us-east-1.amazonaws.com/",
"responseType": "json",
"data": {
"type": "Buffer",
"data": [
<catenated (prefix + contents + suffix) as buffer data>
]
}
}
status: 201
// Register attachment
{
"method": "POST",
"headers": {
"content-type": "application/x-www-form-urlencoded",
"Zotero-API-Key": "XXX",
"If-None-Match": "*"
},
"url": "https://api.zotero.org/groups/2318535/items/AXHPEVKM/file",
"responseType": "json",
"data": "uploadKey=bdb9a0eef0758c59831c73cd5b57a580"
}
Error: Request failed with status code 400
Detail: MD5 hash not provided
Dan Stillman
Feb 21, 2020, 12:15:27 AM2/21/20
to zoter...@googlegroups.com
On 2/21/20 12:03 AM, Frank Bennett wrote:
> "data": "uploadKey=bdb9a0eef0758c59831c73cd5b57a580"
> }
> Error: Request failed with status code 400
> Detail: MD5 hash not provided
You're passing 'uploadKey' instead of 'upload'. In the absence of
> "data": "uploadKey=bdb9a0eef0758c59831c73cd5b57a580"
> }
> Error: Request failed with status code 400
> Detail: MD5 hash not provided
'upload', it just thinks you're making an authorization request, which
is why it's complaining about the absence of a hash. So you just haven't
been registering files at all.
Frank Bennett
Feb 21, 2020, 12:18:10 AM2/21/20
to zotero-dev
Success!
A stupid mistake, of course: I was setting "uploadKey=<uploadKey>", but API expects "upload=<uploadKey>".
Single-item deletions are working, and uploads come through on the server. Looks like I'm all set.
The only outstanding item is the version demanded for multiple deletions.
Thanks again for your guidance!
Frank
Dan Stillman
Feb 21, 2020, 12:22:57 AM2/21/20
to zoter...@googlegroups.com
On 2/20/20 9:13 PM, Frank Bennett wrote:
> It's demanding version 3054, which doesn't correspond to the library,
> the parent item, or the target attachment.
>
> ^^^^^^^^^^
>
> 01_library_version.txt
> A call to the library containing the target item and attachments.
> It reports version "5"
>
> > GET /groups/2318535 HTTP/1.1
> > Host: api.zotero.org
> > User-Agent: curl/7.47.0
> > Accept: */*
> > Zotero-API-Key: XXX
> >
> < HTTP/1.1 200 OK
> < Date: Fri, 21 Feb 2020 01:48:00 GMT
> < Server: Apache/2.4.41 (Amazon)
> ...
> < Zotero-API-Version: 3
> < Zotero-Schema-Version: 3
> < Last-Modified-Version: 5
The 5 isn't the library version. That's the group metadata version:
> It's demanding version 3054, which doesn't correspond to the library,
> the parent item, or the target attachment.
>
> ^^^^^^^^^^
>
> 01_library_version.txt
> A call to the library containing the target item and attachments.
> It reports version "5"
>
> > GET /groups/2318535 HTTP/1.1
> > Host: api.zotero.org
> > User-Agent: curl/7.47.0
> > Accept: */*
> > Zotero-API-Key: XXX
> >
> < HTTP/1.1 200 OK
> < Date: Fri, 21 Feb 2020 01:48:00 GMT
> < Server: Apache/2.4.41 (Amazon)
> ...
> < Zotero-API-Version: 3
> < Zotero-Schema-Version: 3
> < Last-Modified-Version: 5
https://www.zotero.org/support/dev/web_api/v3/syncing#sync_properties
Again, the library version is returned from multi-object endpoints —
e.g., …/items or …/collections.
Frank Bennett
Feb 21, 2020, 12:24:10 AM2/21/20
to zotero-dev
Thanks again, sorry for the confusion at this end.
FB
> --
> You received this message because you are subscribed to the Google Groups "zotero-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/010001706632e207-88cc9f18-7a17-49e9-ad17-229638fc7623-000000%40email.amazonses.com.
FB
> You received this message because you are subscribed to the Google Groups "zotero-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/010001706632e207-88cc9f18-7a17-49e9-ad17-229638fc7623-000000%40email.amazonses.com.
Frank Bennett
Feb 21, 2020, 12:46:47 AM2/21/20
to zotero-dev
> Again, the library version is returned from multi-object endpoints —
> e.g., …/items or …/collections.
Sorry to be dense, but what call should I make to obtain the library
> e.g., …/items or …/collections.
version of, say, /groups/2318535? The call to
/groups/2318535/items?format=versions returns a list of over three
thousand key/version pairs.
FB
Frank Bennett
Feb 21, 2020, 1:21:16 AM2/21/20
to zotero-dev
I think I've got it now. A call to a multi-object endpoint returns a "last-modified-version" header in the response, which seems to be the value we're after. Thanks again.
Frank
> > To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+unsubscribe@googlegroups.com.
Dan Stillman
Feb 21, 2020, 1:44:54 AM2/21/20
to zoter...@googlegroups.com
On 2/21/20 1:21 AM, Frank Bennett wrote:
> I think I've got it now. A call to a multi-object endpoint returns a
> "last-modified-version" header in the response, which seems to be the
> value we're after.
That's right, but one question here would be why you're making a
> I think I've got it now. A call to a multi-object endpoint returns a
> "last-modified-version" header in the response, which seems to be the
> value we're after.
multi-object delete request at all. Multi-object deletes are an outlier
among other write requests in that they require the library version,
which you wouldn't normally have in a non-sync workflow. If you're just
deleting a single item, you can just make a single-item delete request
using the item's version [1], which will ensure that the item wasn't
modified remotely since you downloaded it.
It's important to understand that the purpose of all the version
mechanisms for write requests is to avoid unwittingly clobbering remote
changes, and the versioning generally either is taken care of
automatically by the JSON version properties or follows logically from
the workflow — e.g., syncing and wanting to make sure no remote updates
are missed.
There's no great reason to make an extra multi-object GET request just
to get a library version to send with the multi-object deletion, because
you're not actually tracking the library version or caring about whether
other remote changes were made. If you found yourself needing to delete
a large number of items at once, it might make sense to do that for
performance reasons, but that would be a case where you were
deliberately overriding the API's built-in protections.
[1]
https://www.zotero.org/support/dev/web_api/v3/write_requests#deleting_an_item
Frank Bennett
Feb 21, 2020, 9:28:13 AM2/21/20
to zotero-dev
Thanks, that makes sense. I just wanted to be sure that I understood how that one worked.
I have one more issue, and it's an odd one. I've run a script to remove existing attachments and upload 353 new ones, against a library of 169 items. The process completed fine, with only a couple of failures for a couple of massive files (~25megs apiece) that I moved aside. The files all show on the server, and can be displayed in the online view. But when I synced the local client, all of the attachments on the 169 items disappeared, and the new files did not turn up. Repeating sync has no effect. I tried resetting file sync history for the group library containing the items, and there was no change, the attachments still don't appear locally.
Thinking that it might be some bizarre issue limited to Jurism, I joined the group library from another account and did a fresh sync, and the attachments turn up there (in Jurism). So it's some sort of problem specific to my main client database. It's not an urgent issue: the project that will consume the upload will pull updates via the Zotero API, so they should be all set. But this is weird, and I'll need to sort it out.
I'm not sure how to go about debugging. Have you seen this ever?
FB
Dan Stillman
Feb 21, 2020, 12:40:02 PM2/21/20
to zoter...@googlegroups.com
On 2/21/20 9:28 AM, Frank Bennett wrote:
> Thanks, that makes sense. I just wanted to be sure that I understood
> how that one worked.
>
> I have one more issue, and it's an odd one. I've run a script to
> remove existing attachments and upload 353 new ones, against a library
> of 169 items. The process completed fine, with only a couple of
> failures for a couple of massive files (~25megs apiece) that I moved
> aside. The files all show on the server, and can be displayed in the
> online view. But when I synced the local client, all of the
> attachments on the 169 items disappeared, and the new files did not
> turn up. Repeating sync has no effect. I tried resetting file sync
> history for the group library containing the items, and there was no
> change, the attachments still don't appear locally.
>
> Thinking that it might be some bizarre issue limited to Jurism, I
> joined the group library from another account and did a fresh sync,
> and the attachments turn up there (in Jurism). So it's some sort of
> problem specific to my main client database. It's not an urgent issue:
> the project that will consume the upload will pull updates via the
> Zotero API, so they should be all set. But this is weird, and I'll
> need to sort it out.
>
> I'm not sure how to go about debugging. Have you seen this ever?
You'd want to review the debug output for the first sync after the items
> Thanks, that makes sense. I just wanted to be sure that I understood
> how that one worked.
>
> I have one more issue, and it's an odd one. I've run a script to
> remove existing attachments and upload 353 new ones, against a library
> of 169 items. The process completed fine, with only a couple of
> failures for a couple of massive files (~25megs apiece) that I moved
> aside. The files all show on the server, and can be displayed in the
> online view. But when I synced the local client, all of the
> attachments on the 169 items disappeared, and the new files did not
> turn up. Repeating sync has no effect. I tried resetting file sync
> history for the group library containing the items, and there was no
> change, the attachments still don't appear locally.
>
> Thinking that it might be some bizarre issue limited to Jurism, I
> joined the group library from another account and did a fresh sync,
> and the attachments turn up there (in Jurism). So it's some sort of
> problem specific to my main client database. It's not an urgent issue:
> the project that will consume the upload will pull updates via the
> Zotero API, so they should be all set. But this is weird, and I'll
> need to sort it out.
>
> I'm not sure how to go about debugging. Have you seen this ever?
were modified via the API.
Frank Bennett
Feb 21, 2020, 8:03:16 PM2/21/20
to zotero-dev
That's the deal, thanks. It wasn't reproducible. I isolated one item, re-ran the upload script over it, and the attachment turned up in the client after sync. I then reran the entire job, omitting the two large files that had failed, and sync brought all of the attachments into the client as per normal. Joy!
Not sure what happened there. The only difference was to avoid the two files that violated size constraints, but since the upload script processes attachments to completion one item at a time, those crashes of individual uploads seem irrelevant. In any case, all smiles and smooth sailing.
Thanks again for responding to my series of questions about the API. It's been a huge help.
Frank
Reply all
Reply to author
Forward
0 new messages