new to zotero-dev; a few questions
Robin Paulson
i've have been using zotero for about 3 years now, am new to this list
and to zotero dev, but have some experience of the tools used.
i'm interested in developing the dataserver part of zotero to a
stable, easy-to-install state, so the complete stack can be
self-hosted, without relying on zotero.org.
is this the correct list to continue discussing the data server, or is
there another location, and if so could someone let me know?
i've read through some of the posts, and found these pages, which look relevant:
http://www.zotero.org/support/dev/dataserver_setup
https://www.zotero.org/svn/dataserver/trunk/
assuming this is the correct list, i have further questions, if you don't mind:
is the data server under current development?
is the page above the active code repository?
are there any other pages which would be helpful to read before working on it?
is there an active code repository for a modified version of the
zotero extension, to allow arbitrary sync servers to be specified?
thanks for your time,
--
robin
http://fu.ac.nz - Auckland's Free University
mronkko
On Oct 31, 10:07 am, Robin Paulson <robin.paul...@gmail.com> wrote:
> hi all,
> i've have been using zotero for about 3 years now, am new to this list
> and to zotero dev, but have some experience of the tools used.
>
> i'm interested in developing the dataserver part of zotero to a
> stable, easy-to-install state, so the complete stack can be
> self-hosted, without relying on zotero.org.
>
> is this the correct list to continue discussing the data server, or is
> there another location, and if so could someone let me know?
> i've read through some of the posts, and found these pages, which look relevant:http://www.zotero.org/support/dev/dataserver_setuphttps://www.zotero.org/svn/dataserver/trunk/
> assuming this is the correct list, i have further questions, if you don't mind:
> is the data server under current development?
> is the page above the active code repository?
server code does not seem to be there yet.
> are there any other pages which would be helpful to read before working on it?
> is there an active code repository for a modified version of the
> zotero extension, to allow arbitrary sync servers to be specified?
network) but this for is not about custom servers. You should start a
new fork. (Read the github tutorial on that if you are not familiar
with github)
Mikko
Avram Lyon
> On Oct 31, 10:07 am, Robin Paulson <robin.paul...@gmail.com> wrote:
>> i've read through some of the posts, and found these pages, which look relevant:http://www.zotero.org/support/dev/dataserver_setuphttps://www.zotero.org/svn/dataserver/trunk/
>> are there any other pages which would be helpful to read before working on it?
There's nothing else currently available, but there are people on this
mailing list (I'm not one of them) who have gotten the data server
running with syncs and all. So consult the list archives, and feel
free to add info and clarifications to the documentation wiki at
zotero.org.
>> is there an active code repository for a modified version of the
>> zotero extension, to allow arbitrary sync servers to be specified?
>
> Github lists one active fork (https://github.com/zotero/zotero/
> network) but this for is not about custom servers. You should start a
> new fork. (Read the github tutorial on that if you are not familiar
> with github)
As Zotero has just moved to Github, there was previously no easy way
identify third-party development forks and branches. The teams running
local dataserver instances presumably have internal code repositories
for this-- maybe someone can put that code on Github as a fork?
Avram
Avram Lyon
> i've read through some of the posts, and found these pages, which look relevant:
> http://www.zotero.org/support/dev/dataserver_setup
> https://www.zotero.org/svn/dataserver/trunk/
>
> assuming this is the correct list, i have further questions, if you don't mind:
> is the data server under current development?
> is the page above the active code repository?
The dataserver code has now moved to Github:
https://github.com/zotero/dataserver
Avram
Robin Paulson
> The dataserver code has now moved to Github:
> https://github.com/zotero/dataserver
excellent, thanks for the help and links guys.
i'm also interested in the zotero addon itself; is there a fork of
that maintained by anyone, which allows users to change the address of
the sync server they wish to use?
Avram Lyon
> i'm also interested in the zotero addon itself; is there a fork of
> that maintained by anyone, which allows users to change the address of
> the sync server they wish to use?
No such fork has been announced publicly, although I have to assume
that those who have set up local dataserver installs have also forked
the client code. It would be great if someone made such a fork public
on GitHub.
Avram
Robin Paulson
> On Thu, Nov 10, 2011 at 3:29 AM, Robin Paulson <robin....@gmail.com> wrote:
>> i'm also interested in the zotero addon itself; is there a fork of
>> that maintained by anyone, which allows users to change the address of
>> the sync server they wish to use?
>
> No such fork has been announced publicly, although I have to assume
> that those who have set up local dataserver installs have also forked
i get the impression that anyone running their own sync server is
manually editing the code which sets the server location. i'd like to
make it more configurable that that.
> the client code. It would be great if someone made such a fork public
> on GitHub.
i've been digging around that part of the code myself, and have got a
good idea of how to do it. if no-one else has started that fork, i'll
do it myself
Bruce D'Arcus
> On 10 November 2011 17:01, Avram Lyon <ajl...@gmail.com> wrote:
>> On Thu, Nov 10, 2011 at 3:29 AM, Robin Paulson <robin....@gmail.com> wrote:
>>> i'm also interested in the zotero addon itself; is there a fork of
>>> that maintained by anyone, which allows users to change the address of
>>> the sync server they wish to use?
>>
>> No such fork has been announced publicly, although I have to assume
>> that those who have set up local dataserver installs have also forked
>
> i get the impression that anyone running their own sync server is
> manually editing the code which sets the server location. i'd like to
> make it more configurable that that.
And would be good for this to be folded into Zotero mainline.
Bruce
>> the client code. It would be great if someone made such a fork public
>> on GitHub.
>
> i've been digging around that part of the code myself, and have got a
> good idea of how to do it. if no-one else has started that fork, i'll
> do it myself
>
> --
> robin
>
> http://fu.ac.nz - Auckland's Free University
>
> --
> You received this message because you are subscribed to the Google Groups "zotero-dev" group.
> To post to this group, send email to zoter...@googlegroups.com.
> To unsubscribe from this group, send email to zotero-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/zotero-dev?hl=en.
>
>
Avram Lyon
[..]
>> i get the impression that anyone running their own sync server is
>> manually editing the code which sets the server location. i'd like to
>> make it more configurable that that.
>
> And would be good for this to be folded into Zotero mainline.
It's a little early to say whether this is necessary-- if there comes
a day when something like federated data servers come to be common,
then that might make sense. But I think we can do without an exposed
preference for now. Regardless, this is an exciting time to be working
in the Zotero ecosystem; the next year or two should be fun.
Avram
Kieren Diment
I absolutely disagree. The name of the sync server should absolutely be a hidden preference item changeable in about:config. My (limited) understanding of the zotero codebase suggests to me that this should be a two or three line patch to zotero mainline to implement. Hardcoding the server in the javascript is a strong indication that zotero's server functionality is intended as a walled garden only. While I think it's absolutely reasonable for the dev team not to support independent servers, the combination of this and the hardcoding of the server sends a strong signal which I think is not the right one to be sending.
I have other issues with the server which I hope to deal with in due course, but this is the main one right now.
Avram Lyon
> On 11/11/2011, at 9:32 AM, Avram Lyon wrote:
>> It's a little early to say whether this is necessary-- if there comes
>> a day when something like federated data servers come to be common,
>> then that might make sense. But I think we can do without an exposed
>> preference for now. Regardless, this is an exciting time to be working
>> in the Zotero ecosystem; the next year or two should be fun.
>
>
> I absolutely disagree. The name of the sync server should absolutely be a hidden preference item changeable in about:config. My (limited) understanding of the zotero codebase suggests to me that this should be a two or three line patch to zotero mainline to implement. Hardcoding the server in the javascript is a strong indication that zotero's server functionality is intended as a walled garden only. While I think it's absolutely reasonable for the dev team not to support independent servers, the combination of this and the hardcoding of the server sends a strong signal which I think is not the right one to be sending.
Fair criticism. Perhaps someone out there with a working patched
version can issue a pull request on GitHub with this change. I don't
know if the team will land it, but it would at the very least
demonstrate how simple the change is.
Perhaps Sean or Dan can comment on the team's current views of
independent servers and such a patch?
Avram
Kieren Diment
Thanks for the prompt reply. While I understand that the server name in a hidden config has implications with respect to support and governance, I do not think that the issues it causes are insurmountable (some clearly documented pointer that independent server faults require a test case demonstration on the central server, or a commercial support contract should suffice as an indicator).
Robin Paulson
> It's a little early to say whether this is necessary-- if there comes
> a day when something like federated data servers come to be common,
> then that might make sense. But I think we can do without an exposed
> preference for now. Regardless, this is an exciting time to be working
> in the Zotero ecosystem; the next year or two should be fun.
to be honest, i don't see any difference between allowing the user to
change the sync server, and change the storage server, the latter
already being exposed for anyone to change.
anyway, i'm making the changes as i find it useful and think others
might too. whether it gets pulled into mainline or not is beyond my
call.
i've completed most of what i'm after, and have got it all working
apart from translations to non-english, will post it on github soon
which brings me to my next point:
i'm trying to enable the hiding and showing of sync options in the
preferences dialogue, similar to how the attachment options are hidden
or shown, dependent on whether the "Sync attachment file in My Library
using" option is set to "Zotero" or "WebDAV". i've dug through
preferences.js (i assume it's in here) file looking for the code to
hide these options so i can copy it rather than writing it again. but
i can't find it, can anyone point me to where it it?
any help appreciated. please ask me to clarify, if the questions sounds muddled
cheers
Robin Paulson
>>> I absolutely disagree. The name of the sync server should absolutely be a hidden preference item changeable in
> about:config. My (limited) understanding of the zotero codebase suggests to me that this should be a two or three
> line patch to zotero mainline to implement. Hardcoding the server in the javascript is a strong indication that zotero's > server functionality is intended as a walled garden only. While I think it's absolutely reasonable for the dev team not
> to support independent servers, the combination of this and the hardcoding of the server sends a strong signal which I
> think is not the right one to be sending.
i don't think this is the case at all. the code is in the open, we're
working on it, and it is trivial to change, as i found out
add that to the fact the sync server is not at a point where it's as
simple to set up as webdav for attachment syncing, and i don't see
anything wrong with what they're doing
i see that the version of zotero in the github repo is version 3.0b3.
can someone point me to the repo for the stable version? sorry if this
is a dumb question, i'm new to the idea of repositories
Dan Stillman
Our goal is to provide the best possible experience for users, and we do
that by offering a client that works with the server infrastructure we
run. We can't guarantee that same experience for a version of Zotero
syncing with an independent data server, and we want the distribution
mechanism to reflect that. Any organization capable of running and
supporting its own data server would have the ability to modify Zotero
to sync with it, and members of that organization should know they're
not using the official version of Zotero.
On 11/15/11 3:18 AM, Kieren Diment wrote:
> While I understand that the server name in a hidden config has implications with respect to support and governance, I do not think that the issues it causes are insurmountable (some clearly documented pointer that independent server faults require a test case demonstration on the central server, or a commercial support contract should suffice as an indicator).
It wouldn't be a matter of simply not supporting sync errors with
independent servers. Bug fixes and new features quite often involve
complex sets of changes across particular versions of Zotero server and
client code that, if not done just right, could easily cause all sorts
of both server-side and client-side problems. (As just one example, in
3.0b2 there's a bug with resolving merged duplicates in group libraries
from word processor documents. That's actually an error related to
syncing, and it will require choreographed client-side changes,
server-side changes, and server-side processing at the next client
release to fix.)
There may be a time when we're comfortable with this being a hidden
pref, but at this time we're not.
On 11/16/11 6:54 AM, Robin Paulson wrote:
> to be honest, i don't see any difference between allowing the user to
> change the sync server, and change the storage server, the latter
> already being exposed for anyone to change.
They're not comparable. Zotero syncs files with our own storage service
or with a WebDAV server. We run the former, and the latter uses a very
basic subset of a fairly simple spec that mostly hasn't changed since
1999. If a WebDAV server is broken, file syncing just won't work. If
there's a problem with a data server, it has the potential to destroy
someone's database or break their client in subtle or major ways.
Dan Stillman
> i see that the version of zotero in the github repo is version 3.0b3.
> can someone point me to the repo for the stable version? sorry if this
> is a dumb question, i'm new to the idea of repositories
The GitHub repo has all the code. There's a 2.1.10 tag that contains
code from that release. We'll be creating a 3.0 branch very soon and
continuing post-3.0 development on the master branch.
Read up on Git branches and tags for more info.
Robin Paulson
> They're not comparable. Zotero syncs files with our own storage service or
> with a WebDAV server. We run the former, and the latter uses a very basic
> subset of a fairly simple spec that mostly hasn't changed since 1999. If a
> WebDAV server is broken, file syncing just won't work. If there's a problem
> with a data server, it has the potential to destroy someone's database or
> break their client in subtle or major ways.
ok.
are there any build scripts for the zotero extension? i guess i could
put it together manually, but it seems like the kind of thing someone
here might have scripted
cheers,
Frank Bennett
>
> > They're not comparable. Zotero syncs files with our own storage service or
> > with a WebDAV server. We run the former, and the latter uses a very basic
> > subset of a fairly simple spec that mostly hasn't changed since 1999. If a
> > WebDAV server is broken, file syncing just won't work. If there's a problem
> > with a data server, it has the potential to destroy someone's database or
> > break their client in subtle or major ways.
>
> ok.
>
> are there any build scripts for the zotero extension? i guess i could
> put it together manually, but it seems like the kind of thing someone
> here might have scripted
build script I use for the multilingual version is available here:
https://github.com/fbennett/zotero/blob/master/build.sh
If you use this one, you'll probably need to adjust some values, will
probably want to change the filename of the output xpi.
Frank
>
> cheers,
Robin Paulson
> It wouldn't be a matter of simply not supporting sync errors with
> independent servers. Bug fixes and new features quite often involve complex
> sets of changes across particular versions of Zotero server and client code
> that, if not done just right, could easily cause all sorts of both
> server-side and client-side problems. (As just one example, in 3.0b2 there's
> a bug with resolving merged duplicates in group libraries from word
> processor documents. That's actually an error related to syncing, and it
> will require choreographed client-side changes, server-side changes, and
> server-side processing at the next client release to fix.)
<snip>
> They're not comparable. Zotero syncs files with our own storage service or
> with a WebDAV server. We run the former, and the latter uses a very basic
> subset of a fairly simple spec that mostly hasn't changed since 1999. If a
> WebDAV server is broken, file syncing just won't work. If there's a problem
> with a data server, it has the potential to destroy someone's database or
> break their client in subtle or major ways.
Thanks for going into detail to explain this Dan, I've had time to
digest what you said and I can see the logic behind a lot of it now.
As you say, the sync protocol is rather different to WebDAV at
anything other than a trivial level.
Bearing in mind the issues this thread has raised, and the question by
Mikko on the thread about documentation, I can see immense value in a
well-documented, stable protocol available for use for syncing across
multiple clients by multiple authors, without having to rely on Zotero
using something which is edging towards being a proprietary protocol.
Is this stability and documentation something Zotero are going to do?
I gather from the above that at the moment the protocol is unstable
and liable to change at any time, as the developers wish, and
important subtleties and nuances are thus somewhat out-of-sight of
anyone outside the Zotero group. I realise the code for extension and
server is out in the open, which is something the CHNM should be
commended for, and thus could technically be reverse-engineered, but
there are logistical reasons around time and accuracy why this will
not necessarily turn out the best results, as you have indicated.
What are the plans for the protocol?
Thanks,
Dan Stillman
We don't have any plans to document or stabilize the sync protocol,
which is tightly coupled with client and server development and
maintenance. I would strongly discourage third-party clients from trying
to use it, since they're pretty much guaranteed to break as we make changes.
The server API, on the other hand, is documented and stable and designed
for third-party use, and it encourages a safer (and simpler) style of
interaction with the data server, where the server remains the
authoritative store and clients provide a sometimes-cached view of that
data. Third-party clients designed around the API would likely be able
to work with independent data servers without much trouble. As we add in
some of the recently requested API functionality, it will become easier
for API clients to cache and update data efficiently.
The Zotero client itself has different requirements�a fully offline app,
where the client's data is as authoritative as the server's, and
transactional, all-or-nothing syncing�that aren't really compatible with
the API. We also have to handle a much greater volume of traffic, which
we do with asynchronous queuing of the atomic sync requests. This model
isn't particular suited for third-party clients or third-party servers,
but, again, our concern here is providing the best experience for the
majority of Zotero users.
adamsmith
> Mikko on the thread about documentation, I can see immense value in a
> well-documented, stable protocol available for use for syncing across
> multiple clients by multiple authors, without having to rely on Zotero
> using something which is edging towards being a proprietary protocol.
> Is this stability and documentation something Zotero are going to do?
released, devs and project directors were adamant that it's not going
to receive Zotero support and that includes, I assume, documentation.
More generally, writing something like this is not helpful at all. I'm
not much of a coder, but I feel strongly about the politics of FOSS.
Code released under AGPL is _never_ "bordering on proprietary". There
is no EULA that constrains you. There are not patent lawsuits that you
have to watch out for. There is no licensing fee that you have to pay,
nor a licensing agreement that you have to adhere to etc.
The idea that software needs to be fully documented and stable to be
considered free (let alone non-proprietary) is completely at odds with
the both the history and the customs of free software.
More practically, I'm wondering if you have any thoughts on the
business side of this. Who is going to pay for it?
Robin Paulson
> We don't have any plans to document or stabilize the sync protocol, which is
> tightly coupled with client and server development and maintenance. I would
> strongly discourage third-party clients from trying to use it, since they're
> pretty much guaranteed to break as we make changes.
thanks for the explanation dan.
> The server API, on the other hand, is documented and stable and designed for
> third-party use, and it encourages a safer (and simpler) style of
> interaction with the data server, where the server remains the authoritative
> store and clients provide a sometimes-cached view of that data. Third-party
> clients designed around the API would likely be able to work with
> independent data servers without much trouble. As we add in some of the
> recently requested API functionality, it will become easier for API clients
> to cache and update data efficiently.
great, i'll check it out. cheers for the assistance