Zotero Dataserver: problem with Sync.inc.php

1250 views
Skip to first unread message

simonth

unread,
Jul 14, 2011, 9:08:00 AM7/14/11
to zotero-dev
Hello,

I asked this in an older message but for more clearness I make a new
post.

So, I have a problem with 1406 to 1408 lines in Sync.inc.php (in /
model/ directory):

> // Make sure no other upload sessions use this same timestamp
> // for any of these libraries, since we return >= 1 as the next
> // last sync time
> if (!Zotero_Libraries::setTimestampLock($affectedLibraries, $timestamp)) {
> throw new Exception("Library timestamp already used",
> Z_ERROR_LIBRARY_TIMESTAMP_ALREADY_USED);
>}

When I put these lines in comment, it works fine (till now)...
Otherwise, there is an error message in "upload" daemon:

> 2011-07-14 08:57:03.0059 Starting upload processor daemon
> 2011-07-14 08:57:03.0472 0 processors, 0 queued processes
> 2011-07-14 08:57:09.3387 0 processors, 1 queued process
> 2011-07-14 08:57:09.3392 Starting 1 new processor
> 2011-07-14 08:57:09.8894 [168660644] Starting sync processor
> 2011-07-14 08:57:10.0542 exception 'Exception' with message 'Library timestamp already used' in /home/simon/dataserver/model/Sync.inc.php:1407
> Stack trace:
> #0 /home/simon/dataserver/model/Sync.inc.php(422): Zotero_Sync::processUploadInternal(1, Object(SimpleXMLElement), 449232451, '168660644')
> #1 /home/simon/dataserver/model/Processor.inc.php(128): Zotero_Sync::processUploadFromQueue('168660644')
> #2 /home/simon/dataserver/model/Processor.inc.php(51): Zotero_Upload_Processor->processFromQueue()
> #3 /home/simon/dataserver/processor/upload/processor.php(19): Zotero_Processor->run('168660644')
> #4 {main}
> 2011-07-14 08:57:10.0666 [168660644] Exiting on lock error
> 2011-07-14 08:57:10.0683 LOCK received — waiting 2 seconds

Thanks for your help.

Simon

Avram Lyon

unread,
Jul 14, 2011, 10:43:08 AM7/14/11
to zoter...@googlegroups.com
What precisely happens when you comment out those lines? What goes
wrong? They appear to be to prevent multiple initiations of a sync--
it seems that basic syncing should work even without them.

Also take a look at the timestamp data on your server-- perhaps you
can find an explanation or odd data in the database?

Avram

> --
> You received this message because you are subscribed to the Google Groups "zotero-dev" group.
> To post to this group, send email to zoter...@googlegroups.com.
> To unsubscribe from this group, send email to zotero-dev+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/zotero-dev?hl=en.
>
>

simonth

unread,
Jul 14, 2011, 2:43:22 PM7/14/11
to zotero-dev
When I comment out those lines, synchronization never ends and I've
got the error message in upload daemon loop every 2 seconds...

Amin Hedjazi

unread,
Jul 14, 2011, 4:33:29 PM7/14/11
to zoter...@googlegroups.com
disable memcach caching in your configurations and try again 
or flush all of memcache data 

i had this problem before i think it was some thing to do with memcache 

Regards Amin

On Thu, Jul 14, 2011 at 11:13 PM, simonth <thiber...@gmail.com> wrote:
When I comment out those lines, synchronization never ends and I've
got the error message in upload daemon loop every 2 seconds...

Amin Hedjazi

unread,
Jul 14, 2011, 4:38:36 PM7/14/11
to zoter...@googlegroups.com
oh i remembered the problem occurred when i refreshed the database but not memcache , this code is running here for me 
the problem is not the code you should not comment any thing out 

simonth

unread,
Jul 15, 2011, 4:15:59 AM7/15/11
to zotero-dev
Amin,

Thanks for your answer.

In my config.inc.php I already put this :

public static $MEMCACHED_ENABLED = false;
public static $MEMCACHED_SERVERS = array('memcached1.localdomain:
11211:2', 'memcached2.localdomain:11211:1');

I don't know what to do to disable memcache caching otherwise.

What do you mean by "flush all of memcache data"? Where are these
data?

Simon

Amin Hedjazi

unread,
Jul 15, 2011, 5:10:36 AM7/15/11
to zoter...@googlegroups.com
i took a better look and it seems our problems are not the same 

i suggest you do some debugging
you could debug your processors by using xdebug and eclipse remotely 


and memcache could be flushed by

telnet localhost 1211
flush_all
quit

but flushing probably would not work in your scenario 

Regards Amin




--

Sturmgewehr

unread,
Aug 29, 2012, 5:29:17 PM8/29/12
to zoter...@googlegroups.com
Well, you are not the only one who experienced this problem. I guess the problem is in the obsolete "test_reset" script.
The database structure has been modified (look at the "master.sql" and "shard.sql") but the changes were not reflected to the "test_reset" file.
Properly created "zoterotest_master", "zoterotest1" databases work fine with the rest of the code.
I spent two weeks figuring out what is wrong with both old SVN and new github versions of the server. Eventually, I modified this file and inserted some other changes. All these changes are available in the forked data server project https://github.com/Panzerkampfwagen/dataserver
The modifications are incremental and signed by me so it kind of obeys the license. If there is any infringement of copyright (copyleft), I'll delete the forked project.
Detailed installation instructions are available for Arch Linux (look at the "misc/Zotero_Data_Server_Installation.pdf" file).
Basically, it is an updated and supplemented tutorial published in http://www.zotero.org/support/dev/dataserver_setup
The tutorial is still uncompleted (was written two months ago and then abandoned) and contains many grammatical errors but sufficient for proper installation.
Client is a patched Zotero Firefox extension ("misc/zotero-3.0.8-patched.xpi" file).
The changes are available in https://github.com/Panzerkampfwagen/zotero
They allow you to add your custom data server (like http://127.0.0.1:85) to the preferences without modifying "/ets/hosts", deceiving client and messing up with SSL certificates. You can run this client even on Windows :)
Tested on Arch Linux (x86_64). Works fine with small libraries. You can capture traffic between client and server with Wireshark and see what is going on. When the library size very big (>3000 items), MySQL backend cannot handle such huge amount information and crashes.


Hi Simon,
 
have you solved your problem in the meantime?
 
I'm currently working on setting up a dataserver for our researcher group,
following the instructions from the zotero website, but i'm hanging at the
memcache thing as well. but with other symptoms.
 
In the current ./test_reset skript i don't overcome the following
 
ERROR 1452 (23000) at line 1: Cannot add or update a child row: a foreign key constraint fails (`zoterotest_master`.`libraries`, CONSTRAINT `libraries_ibfk_1` FOREIGN KEY (`shardID`) REFERENCES `shards` (`shardID`))
ERROR 1452 (23000) at line 1: Cannot add or update a child row: a foreign key constraint fails (`zoterotest_master`.`libraries`, CONSTRAINT `libraries_ibfk_1` FOREIGN KEY (`shardID`) REFERENCES `shards` (`shardID`))
ERROR 1452 (23000) at line 1: Cannot add or update a child row: a foreign key constraint fails (`zoterotest_master`.`libraries`, CONSTRAINT `libraries_ibfk_1` FOREIGN KEY (`shardID`) REFERENCES `shards` (`shardID`))
ERROR 1452 (23000) at line 1: Cannot add or update a child row: a foreign key constraint fails (`zoterotest_master`.`users`, CONSTRAINT `users_ibfk_1` FOREIGN KEY (`libraryID`) REFERENCES `libraries` (`libraryID`) ON DELETE CASCADE)
ERROR 1452 (23000) at line 1: Cannot add or update a child row: a foreign key constraint fails (`zoterotest_master`.`users`, CONSTRAINT `users_ibfk_1` FOREIGN KEY (`libraryID`) REFERENCES `libraries` (`libraryID`) ON DELETE CASCADE)
ERROR 1452 (23000) at line 1: Cannot add or update a child row: a foreign key constraint fails (`zoterotest_master`.`groups`, CONSTRAINT `groups_ibfk_1` FOREIGN KEY (`libraryID`) REFERENCES `libraries` (`libraryID`) ON DELETE CASCADE)
ERROR 1452 (23000) at line 1: Cannot add or update a child row: a foreign key constraint fails (`zoterotest_master`.`groupUsers`, CONSTRAINT `groupUsers_ibfk_1` FOREIGN KEY (`groupID`) REFERENCES `groups` (`groupID`) ON DELETE CASCADE)
ERROR 1050 (42S01) at line 3: Table 'collections' already exists
PHP Notice:  MemcachePool::get(): Server localhost (tcp 11211, udp 0) failed with: Connection refused (111) in /home/UTILISATEUR/dataserver/include/Memcached.inc.php on line 99
Notice: MemcachePool::get(): Server localhost (tcp 11211, udp 0) failed with: Connection refused (111) in /home/UTILISATEUR/dataserver/include/Memcached.inc.php on line 99
PHP Fatal error:  Uncaught exception 'Exception' with message 'Shard 1 not found' in /home/UTILISATEUR/dataserver/include/Shards.inc.php:57
Stack trace:
#0 /home/UTILISATEUR/dataserver/include/DB.inc.php(74): Zotero_Shards::getShardInfo(1)
#1 /home/UTILISATEUR/dataserver/include/DB.inc.php(315): Zotero_DB->getShardLink(1, true)
#2 /home/UTILISATEUR/dataserver/misc/test_setup(21): Zotero_DB::query('DELETE FROM sha...', false, 1)
#3 {main}
  thrown in /home/UTILISATEUR/dataserver/include/Shards.inc.php on line 57
Fatal error: Uncaught exception 'Exception' with message 'Shard 1 not found' in /home/UTILISATEUR/dataserver/include/Shards.inc.php:57
Stack trace:
#0 /home/UTILISATEUR/dataserver/include/DB.inc.php(74): Zotero_Shards::getShardInfo(1)
#1 /home/UTILISATEUR/dataserver/include/DB.inc.php(315): Zotero_DB->getShardLink(1, true)
#2 /home/UTILISATEUR/dataserver/misc/test_setup(21): Zotero_DB::query('DELETE FROM sha...', false, 1)
#3 {main}
  thrown in /home/UTILISATEUR/dataserver/include/Shards.inc.php on line 57
 
I'm not quite sure about the proper configuration of memcache but there
isn't a firewall problem in our ubuntu box. But still i receive that connection refused thing.
Obviously memcache itself is working ...
ro...@research:/home/UTILISATEUR/dataserver/misc# telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
stats
STAT pid 2285
STAT uptime 3078
STAT time 1346251570
STAT version 1.4.5
STAT pointer_size 64
STAT rusage_user 0.080005
STAT rusage_system 0.000000
STAT curr_connections 10
STAT total_connections 12
STAT connection_structures 11
STAT cmd_get 1
STAT cmd_set 1
STAT cmd_flush 0
STAT get_hits 1
STAT get_misses 0
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 110
STAT bytes_written 112
STAT limit_maxbytes 67108864
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT bytes 135
STAT curr_items 1
STAT total_items 1
STAT evictions 0
STAT reclaimed 0
END
version
VERSION 1.4.5
quit
Connection closed by foreign host.
 
Can you help me on that?
 
Thanks in Advance for any help on that
 
mkl @ TU Berlin

adamsmith

unread,
Aug 29, 2012, 6:02:23 PM8/29/12
to zoter...@googlegroups.com
Sidenote: I don't see how a fork of AGPL licensed code could possibly violate any license, so I wouldn't worry about that.
I'd encourage you or anyone else who feels like it to update  http://www.zotero.org/support/dev/dataserver_setup - those instructions where created by a user and are not maintained by Zotero devs, but improvements and updates are certainly welcome. It's a wiki.

Avram Lyon

unread,
Aug 29, 2012, 6:05:08 PM8/29/12
to zoter...@googlegroups.com
To edit, add ?do=login:


--
You received this message because you are subscribed to the Google Groups "zotero-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/zotero-dev/-/3ycqZRT_IIEJ.

Sturmgewehr

unread,
Aug 31, 2012, 2:25:45 PM8/31/12
to zoter...@googlegroups.com
Hi, mkl.
As I understood from your logs, posting request from the console using curl works fine for 127.0.0.1:85.
It is good. You almost succeeded, because you can initiate the session and get session ID.
When you post a request to "http://yourdomain:85" (doesn't matter from where: curl or Zotero client), the host name of the server ("yourdomain") is being sent in the HTTP request regardless the resolved IP address. So the request goes to 127.0.0.1:85 with stored "yourdomain" host name.
Examine 
"$Zotero_Data_Server_Root/include/config/routes.inc.php" file. 
Here the data server compares the host name received in the HTTP request (inside a packet) with $SYNC_DOMAIN variable defined in "$Zotero_Data_Server_Root/include/config/config.inc.php". If they are equivalent, then synchronization action is executed. 
If not, then application programming interface (API) is activated.
if ($_SERVER['HTTP_HOST'] == Z_CONFIG::$SYNC_DOMAIN) {
$router->connect('/:action/*', array('controller' => 'Sync'));
}
Despite the fact that "yourdomain" corresponds to "127.0.0.1", variable $_SERVER['HTTP_HOST'] is equal to "yourdomain:85".
As you can see, 
$_SERVER['HTTP_HOST']  is not equal to Z_CONFIG::$SYNC_DOMAIN. So synchronization action is not executed.
However,  when you specify directly the IP address instead of the domain name no problem occurs.
Alternatively you can set 
$SYNC_DOMAIN variable to "yourdomain" name.
Or you can just remove this check and set synchronization as a default action :)

Are you using patched version of Zotero Firefox extension? You have to use the patched one, otherwise it won't work.
As adamsmith and Avram have mentioned, the Wiki is a product of the community. Everyone can contribute. You could write the issues and problems you have encountered during installation process. And I just don't have time to copy-paste tutorial from LaTeX to the Wiki page.

Regards,
Stg44


Hi Sturmgewehr (cool nickname btw)

thank you very much for your help ... i'd like to add serveral issues
to your manual concerning php extensions if you would allow that. 
I was overcomming serveral problems i have experienced before, 
but now i'm hanging at the following point.
My manual test was working from the console, but the 
connection to the server from the plugin in firefox is hanging.

By debugging on the server i see the following happening
mkluge@researcher:~$ curl -X POST -d "version=9&username=testuser&password=testuser" http://OURIPAdress:85/login
<h1>Not Found</h1>
<p>The page you requested could not be found.</p>
mkluge@researcher:~$ curl -X POST -d "version=9&username=testuser&password=testuser" http://127.0.0.1:85/login
<?xml version="1.0"?>
<response version="9" timestamp="1346403529"><sessionID>32cfc109c6a9c3eabd98768f66c8f2f0</sessionID></response>
mkluge@researcher:~$ curl -X POST -d "version=9&username=testuser&password=testuser" http://OURFullQualifiedDomainName:85/login
<h1>Not Found</h1>
<p>The page you requested could not be found.</p>
mkluge@researcher:~$

in the ==> /var/log/apache2/Zotero-Data-Server-Access.log <==
OURIPAdress - - [31/Aug/2012:10:55:36 +0200] "POST /login HTTP/1.1" 404 288
127.0.0.1 - - [31/Aug/2012:10:55:58 +0200] "POST /login HTTP/1.1" 200 327
OURIPAdress - - [31/Aug/2012:10:56:40 +0200] "POST /login HTTP/1.1" 404 288

In my understanding i'm not aware of the things there happening.
The apache2 is configured to listen on port 85
.....
NameVirtualHost *:85
Listen 85
.....
 
and the default file is as well working
NameVirtualHost *:85
<VirtualHost *:85>
ServerName *:85
        DocumentRoot "/home/Zotero/ZoteroDataServer/htdocs"
        ErrorLog "/var/log/apache2/Zotero-Data-Server-Error.log"
        CustomLog "/var/log/apache2/Zotero-Data-Server-Access.log" common
        <Directory "/home/Zotero/ZoteroDataServer/htdocs">
                Options FollowSymLinks MultiViews
                AllowOverride All
                Order allow,deny
                Allow from all
        </Directory>
        AllowEncodedSlashes On
</VirtualHost>

Where is the point at which the webserver distinguishes between a request made towards 127.0.0.1 from the console 
and an external request using dns names ... 

That is the point at wich i'm currently hanging.

Any help in that particular thing would be great.
Thanks in advance
/mkl

mkl@tuB

unread,
Sep 3, 2012, 12:50:34 PM9/3/12
to zoter...@googlegroups.com
Hi Stg44,

i have currently fixed by my understanding the server routing problem.
Thanks for your tip on that.
The customized plugin get's access towards the server.

Due to our requirement to enable team users to get direct access 
to the publications we decided to store all data inside the zotero server
database.
Unfortunatly all publication data from the local zotero directory is stored 
during the synchronisation in a reproducable manner only without any  
attachments. 

I can acces from a different machine the testuser again but during the download
towards another client there aren't any attachments at all. That's not particular the
way i can convince other department members to switch to our new server ,-(

On top of that we receive on the "uploader" client side a spinning wheel with
the info screen later on : 

JavaScript Error: "[Exception... "'File sync login failed Check your username 
and password in the Sync pane of the Zotero preferences.' when calling method: [nsIDOMEventListener::handleEvent]"
nsresult: "0x8057001e (NS_ERROR_XPC_JS_THREW_STRING)"  location:
"<unknown>"  data: no]"]

Is that due to the fact that we didn't use WebDAV for the attachments?
I had the understanding that the Zotero is capable, if working in a local area high speed network should be capable
of handling the blobs inside the db.

ps: i have my Texlipse , so it should be a problem to update your  Tex file .-)

Sturmgewehr

unread,
Sep 3, 2012, 5:00:24 PM9/3/12
to zoter...@googlegroups.com
Hi, mkl@tuB.
You have almost succeeded in setting up the data server.
According to the http://www.zotero.org/support/sync article, the data server synchronizes only metadata.
As it is stated, attached files should be stored using WebDAV server. You can you use cloud storage and free online file sharing services such as Yandex Disk. However, you can set up your own WebDAV  server. Actually syncing works fine but a little bit buggy. Metadata and files get uploaded. However, you should reset file sync history every time (Zotero Preferences -> Sync -> Reset -> File Syncing -> Reset File Sync History) before uploading. Once attachments get uploaded, you can check their existence on the WebDAV server using "cadaver" client as described in the tutorial.
On the other client machine you can just restore everything from the data server (Zotero Preferences -> Sync -> Reset -> Zotero Sync Server -> Restore from Zotero Server). It will show you that metadata and attachments are downloaded, though it is not completely correct: the attachments are not downloaded. To actually download files from the WebDAV server you should just press green "sync" button. Then it may take a while to download all the attached files.

понедельник, 3 сентября 2012 г., 19:50:34 UTC+3 пользователь mkl@tuB написал:

mkl@tuB

unread,
Sep 4, 2012, 9:00:08 AM9/4/12
to zoter...@googlegroups.com
болшое спасибо,

теперь я понял на конец что в нашем режиме уже в плане не было правильным.

/mkl@tuB

Julia Werner

unread,
Apr 5, 2013, 6:21:26 AM4/5/13
to zoter...@googlegroups.com
We fixed this error and zebulon2 posted a patched Zotero 4.0.3 so there's no problem there anymore.
But new problems emerged *sigh* I didn't find this error anywhere else and will post an answer here if we can fix it. 
Error is below.
Regards 
Julia 

SessionStartup: _ensureInitialized: [object Object]

SessionStartup: _ensureInitialized: [object Object]

SessionStartup: _ensureInitialized: [object Object]

SessionStartup: _ensureInitialized: null

SessionStartup: _ensureInitialized: null

[JavaScript Error: "Kein Element gefunden" {file: "http://testuser:********@IP/zotero/zotero-test-file" line: 1 column: 2 source: " "}]

[JavaScript Error: "Kein Element gefunden" {file: "http://testuser:********@IP/zotero/zotero-test-file" line: 1 column: 2 source: " "}]

Julia Werner

unread,
Apr 8, 2013, 8:43:11 AM4/8/13
to zoter...@googlegroups.com
Another Problem solved, in this case the daemons weren't running properly, we restarted them with daemon tools and our patched client syncs like a charm. But there is another, hopefully, minor problem with Group Libraries. We are trying to implement our own local Zotero because of copyright conserns while using the original Zotero Storage. We have set up users and groups in the mysql database. Users and their individual libraries sync properly with all their atteached files and file types. But the Zotero client doesn't show our group libraries. Our question is: Is there anything in the patched client preventing group libraries from beeing synced? Again we couldn't find anything within the groups but hopefully someone has encountered a simmliar problem or can tell us that its just not possible, which would be odd since the Zotero Forum suggest a local installation as a workaround for the official Zotero Storage. 

Can anyone help us?
Thanks in advance for your time.
Julia 

Klaus Flittner

unread,
Apr 8, 2013, 2:41:54 PM4/8/13
to zoter...@googlegroups.com, wern...@gmail.com
Julia Werner wrote:

> Another Problem solved, in this case the daemons weren't running properly,
> we restarted them with daemon tools and our patched client syncs like a
> charm. But there is another, hopefully, minor problem with Group Libraries.
> We are trying to implement our own local Zotero because of copyright
> conserns while using the original Zotero Storage. We have set up users and
> groups in the mysql database. Users and their individual libraries sync
> properly with all their atteached files and file types. But the Zotero
> client doesn't show our group libraries. Our question is: Is there anything
> in the patched client preventing group libraries from beeing synced? Again
> we couldn't find anything within the groups but hopefully someone has
> encountered a simmliar problem or can tell us that its just not possible,
> which would be odd since the Zotero Forum suggest a local installation as a
> workaround for the official Zotero Storage.

You need to update the dateModified field in the groups table if you
create or change settings of a group. If you leave it at the default
value of 0 the group will not be visible in the client.

But you won't get file syncing for groups with just a webdav server.
For filesyncing with groups you need either an Amazon S3 account for
file storage or a local server supporting part of the Amazon S3 REST
API.

Klaus

Klaus Flittner

unread,
Apr 9, 2013, 2:08:50 PM4/9/13
to zoter...@googlegroups.com, alexande...@ibi.hu-berlin.de, wern...@gmail.com
Alexander Struck wrote:

> On Monday, April 8, 2013 8:41:54 PM UTC+2, Klaus Flittner wrote:
>
> [...]
>
> > But you won't get file syncing for groups with just a webdav server.
> > For filesyncing with groups you need either an Amazon S3 account for
> > file storage or a local server supporting part of the Amazon S3 REST
> > API.
>
> Could you please point us to an informative resource here?

I'm not really sure what kind of resource you want. Regarding the
requirement of Amazon S3 or compatible server as backend for group
file storage i can only point you to the sourcecode of the dataserver.

That group file syncing with webdav is not possible was mentioned in
the forum:
https://forums.zotero.org/discussion/9485

In our group we use a local server which supports the S3 API for file
syncing with our local dataserver. With this setup no webdav server is
required and file syncing works for personal as well as group libraries.

Klaus

adamsmith

unread,
Apr 9, 2013, 2:17:32 PM4/9/13
to zoter...@googlegroups.com, alexande...@ibi.hu-berlin.de, wern...@gmail.com, kl...@flittner.org
Klaus (and others),

since you have something running, maybe you have an idea -
I was approached by a medical consulting group who're looking to set-up a local Zotero server for a client and are interested in hiring a developer for that.
Would someone from your group be interested or do you know someone else who might be? I can't have done some other Zotero work for them and it has been a very pleasant experience, but this is above my pay grade.
You can e-mail me directly at karcher - at - u.northwestern.edu and I'll make the contact.
Best,
Sebastian

Klaus Flittner

unread,
Apr 10, 2013, 2:23:44 PM4/10/13
to zoter...@googlegroups.com, alexande...@ibi.hu-berlin.de, wern...@gmail.com
Alexander Struck wrote:

> Please correct me if I'm wrong but I was under the impression that we would
> not need S3 if we wanted to keep things in our LAN. A source code snippet
> respectively the file name that proves me wrong would indeed be appreciated
> if you happen to be aware of the location.

All the file upload/download request from the client are handled in
controllers/ItemsController.php in the function _handleFileRequest.

A normal file upload happens as follows:

1. The client asks the server for upload authorization. For this the
client sends the filename, filesize, mtime, content md5 hash, content
type and charset to the server.

2. The server checks your quota or if the file is already present. If a
file with the same content but a different name is already present then
the dataserver asks S3 to just copy the file so the client doesn't need
to upload it.
If the file is new then the server replies with the upload url and the
upload parameters for Amazon S3.

3. The client uploads the file to S3 with the parameters it got from
the server.

4. The client notifies the server that the file was uploaded successfully.

5. The server checks if the file is present in S3.

The file itself is never uploaded to the dataserver itself but directly
to S3. So if you want file syncing with a local server you also need a
local storage server that is compatible with the S3 API or you would
need to change the dataserver and the client to support a different
storage method.

Klaus

Stefan Meir

unread,
Apr 11, 2013, 7:06:37 AM4/11/13
to zoter...@googlegroups.com
I'm one more researcher who is highly interested in setting up something
like this for his research group.
Klaus, Could you go into the details on what kind of S3-compatible
server you are running?

Best regards,

Stefan



On 04/10/2013 08:23 PM, Klaus Flittner wrote:
> Alexander Struck wrote:
>

Michael Kluge

unread,
Apr 11, 2013, 9:25:38 AM4/11/13
to zoter...@googlegroups.com
Klaus, please can you put me too on that knowledge Distribution;-)

I stopped our efforts at berlin because of that missing piece of information.

Sent from campus Wedding with my ALCATEL one touch Android.

Stefan Meir <herr...@gmail.com> schrieb:

>--
>You received this message because you are subscribed to a topic in the Google Groups "zotero-dev" group.
>To unsubscribe from this topic, visit https://groups.google.com/d/topic/zotero-dev/mrARKo4b4Zw/unsubscribe?hl=en.
>To unsubscribe from this group and all its topics, send an email to zotero-dev+...@googlegroups.com.


>To post to this group, send email to zoter...@googlegroups.com.

>Visit this group at http://groups.google.com/group/zotero-dev?hl=en.
>For more options, visit https://groups.google.com/groups/opt_out.
>
>

Klaus Flittner

unread,
Apr 13, 2013, 12:06:14 PM4/13/13
to zoter...@googlegroups.com, Stefan Meir, Michael Kluge
Stefan Meir wrote:

> I'm one more researcher who is highly interested in setting up something
> like this for his research group.
> Klaus, Could you go into the details on what kind of S3-compatible
> server you are running?

We use a modified version of the dataserver, which allows you to
configure the S3 url. This version is available on github [1].

As we setup our server we didn't find a S3 API that supported all the
API zotero uses. Especially the POST API was not supported. The last
time i checked ceph [2] should support all required API parts.

We are currently using a perl application that just implements the
absolute minimum required by zotero. I pushed the code to github [3].
There is probably a lot that could be improved and it only implements
just enough of the S3 API to be useable with zotero. For our use case
this is sufficient at the moment.

There is currently no howto for installation of the modified dataserver
nor for the storage server. If i find time i will write a short
installation instruction.

Klaus

[1] github.com/sualk/dataserver
[2] ceph.com/docs/master/radosgw/s3/
[3] github.com/sualk/zss

Kieren Diment

unread,
Apr 14, 2013, 7:36:03 AM4/14/13
to zoter...@googlegroups.com


On 14/04/2013, at 2:06 AM, Klaus Flittner wrote:

> Stefan Meir wrote:
>
>> I'm one more researcher who is highly interested in setting up something
>> like this for his research group.
>> Klaus, Could you go into the details on what kind of S3-compatible
>> server you are running?
>
> We use a modified version of the dataserver, which allows you to
> configure the S3 url. This version is available on github [1].
>
> As we setup our server we didn't find a S3 API that supported all the
> API zotero uses. Especially the POST API was not supported. The last
> time i checked ceph [2] should support all required API parts.
>
> We are currently using a perl application that just implements the
> absolute minimum required by zotero. I pushed the code to github [3].
> There is probably a lot that could be improved and it only implements
> just enough of the S3 API to be useable with zotero. For our use case
> this is sufficient at the moment.
>

It looks like a pretty good piece of reasonably modern perl. Improvements would be pretty trivial:

1. Move the whole thing into a proper CPAN dist with Makefile.PL, lib directory etc. If you install Module::Starter by cpan (or better, cpanm), then run this script it should be pretty self explanatory:

module-starter --mi --module=$1 --author="Me Example" --email=m...@example.com

After that you need some very basic documentation. The last thing needed is some pretty basic tests to sanity check it.


I have a perl script that implements a webdav server cleanly as a standalone HTTP server (see below) but I've never had it interact successfully with zotero under a real web server (my choice is nginx - the PHP webdav server seems not to be happy without apache too).


Between some docs to indicate how to run a standalone server, your AWS impersonation code and the webdav server, it seems that a pretty complete standalone zotero server infrastructure is not far off.

> There is currently no howto for installation of the modified dataserver
> nor for the storage server. If i find time i will write a short
> installation instruction.
>
> Klaus
>
> [1] github.com/sualk/dataserver
> [2] ceph.com/docs/master/radosgw/s3/
> [3] github.com/sualk/zss
>
> --
> You received this message because you are subscribed to the Google Groups "zotero-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to zotero-dev+...@googlegroups.com.
> To post to this group, send email to zoter...@googlegroups.com.
> Visit this group at http://groups.google.com/group/zotero-dev?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Perl webdav server works wtih zotero except if running with server as front end proxy:

#!/usr/bin/env perl
use warnings;
use strict;
use Filesys::Virtual::Plain;
use Net::DAV::Server;
use HTTP::Daemon;
use FindBin qw/$Bin/;
my $filesys = Filesys::Virtual::Plain->new({root_path => $Bin});
my $webdav = Net::DAV::Server->new();
$webdav->filesys($filesys);

my $d = HTTP::Daemon->new(
LocalAddr => 'localhost',
LocalPort => 9998,
ReuseAddr => 1) || die;
print "Please contact me at: ", $d->url, "\n";
while (my $c = $d->accept) {
while (my $request = $c->get_request) {
my $response = $webdav->run($request);
$c->send_response ($response);
}
$c->close;
undef($c);
}





mkl@tuB

unread,
Apr 23, 2013, 3:29:20 PM4/23/13
to zoter...@googlegroups.com, kl...@flittner.org
Hi Klaus, 
i tried to merge our efforts towards a group attachment 
usable local storage server here in Berlin, but the 
intended partners at the other university went back 
to the paid option of zotero.

That's unfortunatly not an option for us.

Can we talk to each other somehow to investigate an adoption of your solution. 
br from campus wedding
/mkl
Reply all
Reply to author
Forward
0 new messages