ZotServer addon – local http server to access Zotero database

[Shmavon Gazanchyan]

Hi all,

I am working on addon called ZotServer that provides HTTP access to Zotero database locally.

I've posted about it on the forums and learned that there are already efforts to have similar functionality built-in.

The addon nowhere close to its final state but it already provides functionality need for my use case (search database for an item).

Happy to hear your feedback and contribute to the official implementation of the HTTP access.

Shmavon

[Christian Boulanger]

Hi, Dominic and I have been working towards a similar goal: https://github.com/Dominic-DallOsto/zotero-api-endpoint - we should probably join forces! Probably some of the APIs of our project could be integrated into yours if your project is more actively maintained than ours (which isn't at all at the moment).

Christian

[Shmavon Gazanchyan]

Hi Cristian,

This is great, looks like our implementations are using similar approach as well!

I would be very interested to learn more about your use case and why did you start working on such addon.

I am happy to join forces and build API addon together!

Would be great to check with Dan Stillman on the plans for API functionality in Zotero core and potentially help there as well.

I am new to Zotero development, what is the best way to continue conversation - in this mailing list, forums or in Slack/Discord/Telegram/anything else that supports real-time messaging?

Thanks,

Shmavon

[Christian Boulanger]

HI Shmavon,

I have pinged you on https://gitter.im - we can exchange ideas there with Dominic.

Best,

Christian

[Christian Boulanger]

We have assembled some thoughts on improving the local Zotero HTTP API here: https://pad.gwdg.de/s/CD-kt9Pjh .

[Dan Stillman]

Thanks for writing this up.

So I think there's a much more straightforward way to approach this, and it's the way we've long planned to do this in Zotero proper: to have the Zotero app essentially mirror the web API, such that tools designed to work with the web API could trivially be adapted to work against a local installation, immediately benefitting from private data storage and/or lower latency. Some basic extensions could allow for any other functionality to be performed by local tools.

There are already two Zotero APIs: the web API and the local JavaScript API. The web API, which is used by Zotero itself for syncing, provides a cleanly defined API for accessing all data in Zotero — it's well documented and widely supported by an array of libraries and tools. The JavaScript API is less well documented outside of the Zotero source code but lets you do essentially anything you want in Zotero, and it's obviously used by an increasingly large ecosystem of Zotero plugins.

I think creating a whole other local API, with ad hoc endpoints for specific tasks, would be a mistake, splintering development resources and reproducing existing work for no real reason.

Zotero can already generate web-API-style JSON for all data objects (because that's what's used for syncing), and we already have the beginnings of code to parse an incoming API-style path and return API JSON. Select a collection and run this to see an example:

var c = ZoteroPane.getSelectedCollection();
var path = `library/collections/${c.key}/items`; // would also support 'users/1234' for web API compatibility
var params = Zotero.API.Data.parsePath(path);
var results = await Zotero.API.getResultsFromParams(params);
return results;

Finishing that work and hooking it up to the built-in HTTP server would be fairly trivial. Write requests will take some more work, but the behavior is all clearly defined. (It could probably even share part of a test suite with the dataserver.)

In the end, most existing libraries or tools could likely be pointed at a local installation just by changing the host, perhaps with a few other tweaks to remove various assumptions. So then if you were more comfortable writing Python than JavaScript (or didn't want to dig through Zotero source code), you could use pyzotero to script operations to clean up a local Zotero database.

The primary objection, I assume, will be that the web API is more limited than local Zotero. That's true, but I think there are better ways to address that. I see basically three categories of limitations:

1) Functionality that's not yet implemented in the web API but could be. E.g., the web API currently supports "Title/Creator/Year" search and "Title/Creator/Year + Full-Text" search, but that's just because we've never reproduced the Zotero app's advanced search functionality on the dataserver. But since it already exists locally, it's easy to expose. Similarly, there's no way to get the actual item results from saved searches online, but supporting …/searches/{searchKey}/items locally would be trivial.

2) Functionality that's not implemented in the web API because it's about actions that only exist in the app, but that still applies to data objects referenced by the API. E.g., selecting items or opening a PDF doesn't make sense via the web API, but I could imagine using an HTTP header to trigger a Zotero UI action on a set of objects defined by the scope of the URL, so that POST /library/items/B2FDK42M with Zotero-Action: select would select a single item (equivalent to launching zotero://select/library/items/B2FDK42M), and POST /library/items?tag=to%20read&sort=dateAdded&direction=desc&limit=5 with Zotero-Action: open-pdf would open the PDFs belonging to the five most recently added items with the "to read" tag.

3) Functionality that's not implemented in the web API because it involves Zotero app internals. This could be addressed just by adding an endpoint to run arbitrary privileged JavaScript, the same as BBT's debug-bridge. We've never done this, but with appropriate security precautions, we could, allowing local apps to kick off whatever functionality they wanted and optionally return results.

Related to (3), we've put in some work towards being able to save JS snippets and trigger them either manually or via keyboard shortcuts (the latter similar to the old keyconfig extension for Firefox and Thunderbird). I think this would better address some of the repeated tasks mentioned in the linked doc that have no real reason to run outside of Zotero. Ultimately, the language of the Zotero app is JavaScript, and if effort is going to be put towards improving Zotero's scriptability, the first priority would be generating better JS API documentation from the Zotero source code so that it's easier for people to do what they want to do, whether from a plugin, manually from within Zotero, or from an external app POSTing some JS and getting a response.

We can try to hook up some of the above soon to get this started, and if others would like to help implement parts of the web API after that, that'd be great. But I'd strongly discourage folks from creating a whole other dev environment — at least one that you encourage others to use — rather than building off of the two very powerful, widely supported APIs we already have.

- Dan

[Christian Boulanger]

Dan:

thanks so much for your detailed answer. It makes total sense to me to proceed the way you suggest, i.e.

1) mirroring the web API locally to provide a 100% compatible offline/local endpoint 

2) adding Zotero actions via header to extend the API with app-specific features

3) running Javascript remotely and returning the result for any other use case not covered by 1) and 2)

I am very happy if I can avoid having to write (and have users install) plugins, so I particularly like 3) to give me total freedom to access the internal API.

[Christian Boulanger]

As to security - what do you think about interactively getting user permission for 1) and separately with some serious warnings for 2) and 3) but then removing the CORS restriction so that a locally running web app can interact with Zotero?

[Christian Boulanger]

Hi Dan, do you have a sense for what a rough timeline would be for implementing the basics of what you have described? I think 3) would get us already a long way since this would allow to do basically anything withing Zotero (with safety precautions, of course). Any thoughts on the CORS issue?

[POB TTT]

I came up with the idea to create a local http server plugin for zotero this morning and then I found this thread.

If I got it right, did Dan mean you think we should clone web api behavior rather than write a whole new api system?

[Dan Stillman]

We've implemented nearly the entire read-only portion of the Zotero web API, and we'll be merging it soon.

I'll announce it here when it's ready for use.

[Christian Boulanger]

Great news Dan - looking forward to this. Until then, and until the write part has been implemented, https://github.com/Dominic-DallOsto/zotero-api-endpoint works well enough (if you don't mind that the API is not compatible with the Zotero Web API) and is still accepting additions. Happy to see this extension be made redundant soon.

[Christian Boulanger]

Hi Dan, just out of curiosity - is the work on this feature in a public branch in the GitHub repo? It would be interesting to see what you are working on...

Dan Stillman schrieb am Dienstag, 18. Oktober 2022 um 11:10:27 UTC+2:

[adamsmith]

Yes, the relevant PR is here: https://github.com/zotero/zotero/pull/2842 (remember that PRs on Zotero's gh are rarely the place for discussion except for technical feedback based on tests of the PR branch)

[Dan Stillman]

The local API is now available in the Zotero 7 beta:

https://groups.google.com/g/zotero-dev/c/ElvHhIFAXrY/m/fA7SKKwsAgAJ

Thanks for your patience!

[Shmavon Gazanchyan]

Thanks Dan, this is great news!

-- 

You received this message because you are subscribed to a topic in the Google Groups "zotero-dev" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/zotero-dev/eJ9bdtiCvJE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zotero-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/010001901b754a00-a3c97f36-1813-42dc-8f9d-5e6cd4c33ae1-000000%40email.amazonses.com.

[Abe Jellinek]

Try

let [path] = searchParams.keys();

On Jun 16, 2024, at 5:22 PM, Emiliano Heyns <emilian...@iris-advies.com> wrote:



I think this comes with a change in parameters available to server endpoints -- query is now searchParams, correct? I used to be able to retrieve the "path" passed to an url such as /better-bibtex/pull?some/path/here as request.query[''], but that shows up in search params as key = some/path/here, value = "". Can the bare query string be made available on the request method again?

On Saturday, June 15, 2024 at 12:33:14 PM UTC+2 Dan Stillman wrote:

The local API is now available in the Zotero 7 beta:

https://groups.google.com/g/zotero-dev/c/ElvHhIFAXrY/m/fA7SKKwsAgAJ

Thanks for your patience!

--

You received this message because you are subscribed to the Google Groups "zotero-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/5639995f-ea7a-4053-b7ce-5b98aa0c6c6dn%40googlegroups.com.

[Emiliano Heyns]

Thx! I found searchParams.toString() works for my purposes. 

You received this message because you are subscribed to a topic in the Google Groups "zotero-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zotero-dev/eJ9bdtiCvJE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zotero-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/B4FE30D4-C89F-48A8-BE21-1BC0EE25365D%40berkeley.edu.

[Abe Jellinek]

Keep in mind that the output of toString() will be URL-encoded.

To view this discussion on the web visit https://groups.google.com/d/msgid/zotero-dev/CAHOSGO6mPJEAmK3H9-NXWpAAtGPbAFyXiW6%3Dyj3DrypZkBWFCg%40mail.gmail.com.