External data source and future of Zotero?

[Karl Fritsche]

First of all thanks for all the support and work over the past years on this project!

Today I was trying to catching up with the latest info about Zotero, which in my mind is not that easy with the spread of the forum, this discussion board, and GitHub issues.

But mostly I will relate now to https://groups.google.com/d/msg/zotero-dev/N9ncp0776uc/Izek_RIqAgAJ and https://groups.google.com/d/msg/zotero-dev/qvBK2YEIj_c/FgY4YrLCAQAJ .

Our main use case always was to have a private data source. This is not because we do not want to support you or do not want to pay, its basically because of having our data in our hand and not "somewhere" in the cloud.

But while reading through some posts I also have some general questions about the future of Zotero. So I split up my questions in status quo regarding external data sources and the future of Zotero.

Questions status quo:

1. In the past with V4 we used http://git.27o.de/dataserver/. Does anyone have experience if this works with the new V5? Are there any big changes on the API level between V4 and V5? Is the branch 2015.02 in that repo more stable as last commits there suggest something about API version 3, which it seems is used in V5?
   
2. "fact that until recent changes in 5.0, local sync servers were always rather awkward as they require patched clients"
   This quote was from 07.07.17 and I can't find a way in the current V5 that this had changed and the client still needs a patched version. Was this idea dropped at the end?
   
3. "We're working on a Docker-based setup that will make small installations of the data server much easier."
   Any updates on this one? Is this still planned or dropped because of other constraints?
   
4. It seems that the most reliable solution is to continue with V4 if there were big changes on the API between V4 and V5? I would always like to prefer to run on a newer version, still using the git.27o.de which wasn't updated since 3-5 years feels awkward.
   

Questions future Zotero:

1. Is there a public roadmap available?
   
2. Are there any plans to implement a way to support external data sources?
   
3. Are there any plans to support WebDAV for groups? In parallel, I try to negotiate with the company I'm working for if using api.zotero.org for the metadata/DB would be an option and using WebDAV on our server infrastructure. But using official API would mean we couldn't use workarounds like everybody using the same account. As for example, a user would still have access even after she left the company. And/or its hard to give people temporary access to the DB. With external data sources this is not an issue, if you can only access the server if you are in-house. That's why WebDav for groups would be required in that case.
   
4. I read that electron will be the new way to go, but from the sound of some comments, I get the feeling nothing much is done in that direction yet. This is from a security point of view a very big issue. As it seems currently Firefox 54 is used, which is not an ESR version and can have critical bugs and security issues. It can't be upgraded due to the XLU issue and Firefox ESR 52 goes end of life latest August this year (6 months left!) with the qualifying phase starting in May (2 months left!). So installing Zotero will be soonish a security issue. Is there any battle plan to get the switch to electron done till then? For me, with the stuff I've seen till now, this sounds like a very ambitious plan.
   
5. An alternative idea for supporting external data sources would make the sync pluggable. Meaning api.zotero.org would be the default plugin for sync, which can be changed to an alternative plugin. So somebody could write an alternative data source without AWS usage to have a lightweight in-house alternative. But thinking in this way doesn't make sense till the electron upgrade is not done.
   

[Dan Stillman]

On 2/23/18 11:20 AM, Karl Fritsche wrote:
> Questions status quo:
>
> In the past with V4 we used http://git.27o.de/dataserver/. Does anyone
> have experience if this works with the new V5? Are there any big
> changes on the API level between V4 and V5? Is the branch 2015.02 in
> that repo more stable as last commits there suggest something about
> API version 3, which it seems is used in V5?

I wouldn't recommend using a server version that old with Zotero 5.
While the basics of API-based syncing might work, we've made made
changes since then, including things we were able to change or fix
without breaking backward compatibility.

> "fact that until recent changes in 5.0, local sync servers were always
> rather awkward as they require patched clients"  This quote was from
> 07.07.17 and I can't find a way in the current V5 that this had
> changed and the client still needs a patched version. Was this idea
> dropped at the end?

https://github.com/zotero/zotero/commit/084f5a72df24912820724e8fe46b39b837deb554

> "We're working on a Docker-based setup that will make small
> installations of the data server much easier."  Any updates on this
> one? Is this still planned or dropped because of other constraints?

Still planned, hopefully for later this year, but nothing available yet.

> It seems that the most reliable solution is to continue with V4 if
> there were big changes on the API between V4 and V5? I would always
> like to prefer to run on a newer version, still using the git.27o.de
> which wasn't updated since 3-5 years feels awkward.
>
> Questions future Zotero:
>
> Is there a public roadmap available?

No, but then, there's not a private roadmap either.

> Are there any plans to implement a way to support external data sources?

Same as we've said: we're hoping to offer a Docker-based version that,
along with the dropping of 4.0 sync later this year, should be much
easier for others to run, both for custom installations and for
development. (Depending on the state of AWS service clones and how our
architecture evolves, it may still require AWS for complete
functionality, but it could be under an AWS account that you control.
Our goal (as it currently stands) is to make the dataserver runnable by
others, not to provide an on-prem solution.)

> Are there any plans to support WebDAV for groups? In parallel, I try
> to negotiate with the company I'm working for if using api.zotero.org
> for the metadata/DB would be an option and using WebDAV on our server
> infrastructure. But using official API would mean we couldn't use
> workarounds like everybody using the same account. As for example, a
> user would still have access even after she left the company. And/or
> its hard to give people temporary access to the DB. With external data
> sources this is not an issue, if you can only access the server if you
> are in-house. That's why WebDav for groups would be required in that case.

I don't think we'll support WebDAV in groups — configuration would be
just too much of a mess. It's more likely that we'll support setting a
Linked Attachment Base Directory on a per-group basis and allowing
linked files in groups.

> I read that electron will be the new way to go, but from the sound of
> some comments, I get the feeling nothing much is done in that
> direction yet. This is from a security point of view a very big issue.
> As it seems currently Firefox 54 is used, which is not an ESR version
> and can have critical bugs and security issues. It can't be upgraded
> due to the XLU issue and Firefox ESR 52 goes end of life latest August
> this year (6 months left!) with the qualifying phase starting in May
> (2 months left!). So installing Zotero will be soonish a security
> issue. Is there any battle plan to get the switch to electron done
> till then? For me, with the stuff I've seen till now, this sounds like
> a very ambitious plan.

Windows and Linux builds are based on Firefox 52 ESR. The current Mac
build is based on Firefox 54.0.1, but the next release will be on 52 ESR
as well.

We've mostly already updated Zotero to run on Firefox 60 ESR
(https://github.com/zotero/zotero/tree/fx60) and should have that
finished by the summer. (Third-party XUL-based plugins will still work
with minor adjustments.)

Electron/React is still the plan, but Firefox 60 ESR should get us
through to 2019.

- Dan