very slow webdav experience

[Martin Bosner]

I am using webdav with seafile and it works well until i got more than around 100 files in one directory. As soon as i have that many files things are getting very very slow. Browsing the directory itself is fast but as soon as i try to download or upload a file (3mb picture, tested with firefox and chrome on different machines) it takes up to 30 seconds before i get a response from the server and then it starts downloading the file at normal speed. Nothing in the logs but the access itself.

nginx:
        location /webdav  {
                client_max_body_size 0;
                fastcgi_pass    127.0.0.1:8080;
                fastcgi_param   SCRIPT_FILENAME     $document_root$fastcgi_script_name;
                fastcgi_param   PATH_INFO           $fastcgi_script_name;

                fastcgi_param   SERVER_PROTOCOL     $server_protocol;
                fastcgi_param   QUERY_STRING        $query_string;
                fastcgi_param   REQUEST_METHOD      $request_method;
                fastcgi_param   CONTENT_TYPE        $content_type;
                fastcgi_param   CONTENT_LENGTH      $content_length;
                fastcgi_param   SERVER_ADDR         $server_addr;
                fastcgi_param   SERVER_PORT         $server_port;
                fastcgi_param   SERVER_NAME         $server_name;

                fastcgi_param   HTTPS               off;
                fastcgi_param HTTP_SCHEME http;

                access_log      /var/log/nginx/seafdav.access.log;
                error_log       /var/log/nginx/seafdav.error.log;
        }

cat seafdav.conf
[WEBDAV]
enabled = true
port = 8080
fastcgi = true
share_name = /webdav

server tcpdump / the first package is the initial client request for a file:

tcpdump -i eth0 port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
04:43:06.228762 IP client.hostname.57820 > server.ip.https: Flags [S], seq 440665803, win 65535, options [mss 1452,nop,wscale 4,nop,nop,TS val 1251172942 ecr 0,sackOK,eol], length 0
04:43:06.228796 IP server.ip.https > client.hostname.57820: Flags [S.], seq 4096996172, ack 440665804, win 28960, options [mss 1460,sackOK,TS val 883456 ecr 1251172942,nop,wscale 7], length 0
04:43:06.270242 IP client.hostname.57820 > server.ip.https: Flags [.], ack 1, win 8280, options [nop,nop,TS val 1251172984 ecr 883456], length 0
04:43:06.271481 IP client.hostname.57820 > server.ip.https: Flags [P.], seq 1:216, ack 1, win 8280, options [nop,nop,TS val 1251172984 ecr 883456], length 215
04:43:06.271510 IP server.ip.https > client.hostname.57820: Flags [.], ack 216, win 235, options [nop,nop,TS val 883466 ecr 1251172984], length 0
04:43:06.272691 IP server.ip.https > client.hostname.57820: Flags [P.], seq 1:885, ack 216, win 235, options [nop,nop,TS val 883467 ecr 1251172984], length 884
04:43:06.315268 IP client.hostname.57820 > server.ip.https: Flags [.], ack 885, win 8224, options [nop,nop,TS val 1251173027 ecr 883467], length 0
04:43:06.319897 IP client.hostname.57820 > server.ip.https: Flags [P.], seq 216:378, ack 885, win 8224, options [nop,nop,TS val 1251173031 ecr 883467], length 162
04:43:06.320426 IP client.hostname.57820 > server.ip.https: Flags [P.], seq 378:886, ack 885, win 8224, options [nop,nop,TS val 1251173031 ecr 883467], length 508
04:43:06.320449 IP server.ip.https > client.hostname.57820: Flags [.], ack 886, win 252, options [nop,nop,TS val 883478 ecr 1251173031], length 0
04:43:06.320936 IP server.ip.https > client.hostname.57820: Flags [P.], seq 885:1127, ack 886, win 252, options [nop,nop,TS val 883479 ecr 1251173031], length 242
04:43:06.364914 IP client.hostname.57820 > server.ip.https: Flags [.], ack 1127, win 8209, options [nop,nop,TS val 1251173076 ecr 883479], length 0
04:43:16.482287 IP client.hostname.57820 > server.ip.https: Flags [.], ack 1127, win 8209, length 0
04:43:16.482298 IP server.ip.https > client.hostname.57820: Flags [.], ack 886, win 252, options [nop,nop,TS val 886019 ecr 1251173076], length 0
04:43:26.637419 IP client.hostname.57820 > server.ip.https: Flags [.], ack 1127, win 8209, length 0
04:43:26.637429 IP server.ip.https > client.hostname.57820: Flags [.], ack 886, win 252, options [nop,nop,TS val 888558 ecr 1251173076], length 0
04:43:36.801288 IP client.hostname.57820 > server.ip.https: Flags [.], ack 1127, win 8209, length 0
04:43:36.801298 IP server.ip.https > client.hostname.57820: Flags [.], ack 886, win 252, options [nop,nop,TS val 891099 ecr 1251173076], length 0
04:43:42.644439 IP server.ip.https > client.hostname.57820: Flags [.], seq 1127:4007, ack 886, win 252, options [nop,nop,TS val 892559 ecr 1251173076], length 2880
04:43:42.644515 IP server.ip.https > client.hostname.57820: Flags [.], seq 4007:6887, ack 886, win 252, options [nop,nop,TS val 892559 ecr 1251173076], length 2880
04:43:42.644545 IP server.ip.https > client.hostname.57820: Flags [.], seq 6887:9767, ack 886, win 252, options [nop,nop,TS val 892559 ecr 1251173076], length 2880
04:43:42.644550 IP server.ip.https > client.hostname.57820: Flags [.], seq 9767:12647, ack 886, win 252, options [nop,nop,TS val 892559 ecr 1251173076], length 2880
04:43:42.644578 IP server.ip.https > client.hostname.57820: Flags [.], seq 12647:15527, ack 886, win 252, options [nop,nop,TS val 892559 ecr 1251173076], length 2880
04:43:42.720582 IP server.ip.https > client.hostname.57820: Flags [.], seq 15527:16967, ack 886, win 252, options [nop,nop,TS val 892579 ecr 1251173076], length 1440
04:43:42.886845 IP client.hostname.57820 > server.ip.https: Flags [.], ack 4007, win 8102, options [nop,nop,TS val 1251209364 ecr 892559], length 0
04:43:42.886874 IP server.ip.https > client.hostname.57820: Flags [.], seq 16967:18407, ack 886, win 252, options [nop,nop,TS val 892620 ecr 1251209364], length 1440
04:43:42.886881 IP server.ip.https > client.hostname.57820: Flags [.], seq 18407:21287, ack 886, win 252, options [nop,nop,TS val 892620 ecr 1251209364], length 2880
04:43:42.887499 IP client.hostname.57820 > server.ip.https: Flags [.], ack 6887, win 8012, options [nop,nop,TS val 1251209364 ecr 892559], length 0
04:43:42.887527 IP server.ip.https > client.hostname.57820: Flags [.], seq 21287:24167, ack 886, win 252, options [nop,nop,TS val 892620 ecr 1251209364], length 2880

[Shuai Lin]

Hi Martin, What's the version of your seafile server?

--
You received this message because you are subscribed to the Google Groups "seafile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seafile+u...@googlegroups.com.
To post to this group, send email to sea...@googlegroups.com.
Visit this group at http://groups.google.com/group/seafile.
For more options, visit https://groups.google.com/d/optout.

[Martin Bosner]

Sorry - it is 3.0.3 on Ubuntu 14.04 server 64bit

[JiaQiang Xu]

Is the library created before you upgrade to 3.0?

I don't understand, webdav doesn't now support upload from a browser. How do you do that?

[Martin Bosner]

Ok - i should be a little bit more verbose:

It is a fresh 3.0.2 installation upgraded to 3.0.3.

We don't have any problems using the normal seafile client - everything is working fine and fast.
But - for photo uploads via webdav from our mobiles we are using foldersync on android. At first everything worked fine but after uploading around 100 files into one directory we experienced very slow connections. For testing the seafile side we just went to a browser and tried to download a file. Browsing the directories is fast but downloading a file is slow.

Here is the wget debug output - i create a red mark where it hangs:

time wget --debug --no-check-certificate --http-user=username --http-password=passwordxy https://seafile.server/webdav/Bilderupload/20131205_194536.jpg
Setting --check-certificate (checkcertificate) to 0
Setting --http-user (httpuser) to username
Setting --http-password (httppassword) to passwordxy
DEBUG output created by Wget 1.13.4 on linux-gnu.

URI encoding = `UTF-8'
--2014-05-15 10:53:24--  https://seafile.server/webdav/Bilderupload/20131205_194536.jpg
Host `seafile.server' has not issued a general basic challenge.
Resolving seafile.server (seafile.server)... 1.1.1.1
Caching seafile.server => 1.1.1.1
Connecting to seafile.server (seafile.server)|1.1.1.1|:443... connected.
Created socket 3.
Releasing 0x0000000000cdaff0 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x0000000000cdb250
certificate:
  subject: /C=DE/ST=Some-State/O=Internet Widgits Pty Ltd/CN=seafile.server
  issuer:  /C=DE/ST=Some-State/O=Internet Widgits Pty Ltd/CN=seafile.server
WARNING: cannot verify seafile.server's certificate, issued by `/C=DE/ST=Some-State/O=Internet Widgits Pty Ltd/CN=media.x-ion.de':
  Self-signed certificate encountered.

---request begin---
GET /webdav/Bilderupload/20131205_194536.jpg HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: */*
Host: seafile.server
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Not Authorized
Server: nginx/1.4.6 (Ubuntu)
Content-Type: text/html
Content-Length: 142
Connection: keep-alive
WWW-Authenticate: Basic realm="Seafile Authentication"
Date: Thu, 15 May 2014 08:53:28 GMT

---response end---
401 Not Authorized
Registered socket 3 for persistent reuse.
Skipping 142 bytes of body: [<html><head><title>401 Access not authorized</title></head>
<body>
<h1>401 Access not authorized</h1>
</body>
</html>
        ] done.
Inserted `seafile.server' into basic_authed_hosts
Reusing existing connection to seafile.server:443.
Reusing fd 3.

---request begin---
GET /webdav/Bilderupload/20131205_194536.jpg HTTP/1.1
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: */*
Host: seafile.server
Connection: Keep-Alive
Authorization: Basic bS5iasduZXJAeC1pb24uZGU6sdasdasdLWE1KOEEyZFQ=

---request end---
HTTP request sent, awaiting response...
wget is waiting at this point for about 35 seconds.
---response begin---
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Content-Type: image/jpeg
Content-Length: 4618983
Connection: keep-alive
Last-Modified: Tue, 29 Apr 2014 23:56:41 GMT
Date: Thu, 15 May 2014 08:54:04 GMT
ETag: "f720a2cfasdasdasdf4ea5ea011fcdc888"

---response end---
200 OK
Length: 4618983 (4.4M) [image/jpeg]
Saving to: `20131205_194536.jpg'

100%[====================================================================================================================================================================================================>] 4,618,983   14.4M/s   in 0.3s

2014-05-15 10:54:01 (14.4 MB/s) - `20131205_194536.jpg' saved [4618983/4618983]


real  0m36.572s
user  0m0.032s
sys 0m0.044s

[Martin Bosner]

Any news on this? Webdav is pretty important to me.

[Martin Bosner]

Will the fix discussed in https://groups.google.com/forum/#!topic/seafile/SSeeG0N4Zjs also help with this issue?

[JiaQiang Xu]

I'll do some testing.

Can you observe the delay in logs/seafdav.log?

[Martin Bosner]

i activated verbose logs wherever i could and got a little more. I tested with firefox this time but it is the same behavior with chrome, wget or curl.

When i request a file (i can see the request in tcpdump) i get the following instant logentries:

seafile.log:

[05/21/14 02:10:52] repo-mgr.c(593): Commit b03db0efd676ee9552b31fe7463072da748969b8 is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit becc90a1c65f9ffa7f69dd2ffefbe414286b707e is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit 6f30557cdc7f6cf9f6f18205f3d732d82767f3ea is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit dba3e25f0ec0789077672d4c3172a41a36d4df14 is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit b63019ea9922e500dd0c9a7a78214af88e03e750 is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit 15bcd275f1abfd1e72bea8d4b7a01934972b7008 is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit 3f31101d65a85ba5d43276bd6e7c2bfda2f78fbe is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit b03db0efd676ee9552b31fe7463072da748969b8 is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit becc90a1c65f9ffa7f69dd2ffefbe414286b707e is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit 6f30557cdc7f6cf9f6f18205f3d732d82767f3ea is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit dba3e25f0ec0789077672d4c3172a41a36d4df14 is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit b63019ea9922e500dd0c9a7a78214af88e03e750 is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit 15bcd275f1abfd1e72bea8d4b7a01934972b7008 is missing
[05/21/14 02:10:52] repo-mgr.c(593): Commit 3f31101d65a85ba5d43276bd6e7c2bfda2f78fbe is missing

then i don't see any response in any logfile for about 40 seconds (check timestamps) - after that i get this:



seafile.log:

[05/21/14 02:11:30] ccnet_processor_handle_update: [Proc] Shutdown processor threaded-rpcserver-proc(-1642) for bad update: 515 peer down


nginx - seafdav.access.log:

185.27.182.2 - my.e...@address.com [21/May/2014:02:11:29 +0200] "GET /webdav/Bilderupload%20Martin/20140517_162902.jpg HTTP/1.1" 200 98047 "https://seafile.web.server/webdav/Bilderupload%20Martin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0"
185.27.182.2 - my.e...@address.com [21/May/2014:02:11:42 +0200] "GET /webdav/Bilderupload%20Martin/20140517_162902.jpg HTTP/1.1" 200 5645794 "https://seafile.web.server/webdav/Bilderupload%20Martin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0"



controller.log:

[05/21/14 02:11:33] seafile-controller.c(419): pid file /data/fileshare/seafile/versions/pids/seafdav.pid does not exist
[05/21/14 02:11:33] seafile-controller.c(450): seafdav need restart...
[05/21/14 02:11:33] seafile-controller.c(74): spawn_process: /usr/bin/python2.7 -m wsgidav.server.run_server runfcgi --log-file /data/fileshare/seafile/versions/logs/seafdav.log --pid /data/fileshare/seafile/versions/pids/seafdav.pid --port 8080
[05/21/14 02:11:33] seafile-controller.c(89): spawned /usr/bin/python2.7, pid 15809
[05/21/14 02:11:43] seafile-controller.c(419): pid file /data/fileshare/seafile/versions/pids/seafdav.pid does not exist
[05/21/14 02:11:43] seafile-controller.c(450): seafdav need restart...
[05/21/14 02:11:43] seafile-controller.c(74): spawn_process: /usr/bin/python2.7 -m wsgidav.server.run_server runfcgi --log-file /data/fileshare/seafile/versions/logs/seafdav.log --pid /data/fileshare/seafile/versions/pids/seafdav.pid --port 8080
[05/21/14 02:11:43] seafile-controller.c(89): spawned /usr/bin/python2.7, pid 15874
[05/21/14 02:11:53] seafile-controller.c(419): pid file /data/fileshare/seafile/versions/pids/seafdav.pid does not exist
[05/21/14 02:11:53] seafile-controller.c(450): seafdav need restart...
[05/21/14 02:11:53] seafile-controller.c(74): spawn_process: /usr/bin/python2.7 -m wsgidav.server.run_server runfcgi --log-file /data/fileshare/seafile/versions/logs/seafdav.log --pid /data/fileshare/seafile/versions/pids/seafdav.pid --port 8080
[05/21/14 02:11:53] seafile-controller.c(89): spawned /usr/bin/python2.7, pid 15939
...restarting over and over

this started with the first filerequest and did not stop for a while (at least 15 minutes - and it is not firefox sending further requests - i closed firefox )


ccnet.log:

[05/21/14 02:11:30] ../common/peer.c(942): Local peer down
[05/21/14 02:11:30] ../common/processor.c(219): [Proc] Shutdown processor threaded-rpcserver-proc(-1002) for bad update: 515 peer down
[05/21/14 02:11:30] ../common/processor.c(219): [Proc] Shutdown processor service-proxy-proc(-1001) for bad update: 515 peer down
[05/21/14 02:11:30] ../common/processor.c(219): [Proc] Shutdown processor service-stub-proc(1642) for bad update: 515 peer down
[05/21/14 02:11:30] ../common/peer.c(942): Local peer down
[05/21/14 02:11:30] ../common/processor.c(219): [Proc] Shutdown processor threaded-rpcserver-proc(-1001) for bad update: 515 peer down



seafdav.log:

[2014-05-21 02:11:29,711]:   - my.e...@address.com - "GET /Bilderupload Martin/20140517_162902.jpg" depth=0, connection="keep-alive", agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0", elap=50.464sec -> 200 OK
[2014-05-21 02:11:34,967]:  Default encoding: ascii (file system: UTF-8)
[2014-05-21 02:11:42,330]:   - my.e...@address.com - "GET /Bilderupload Martin/20140517_162902.jpg" depth=0, connection="keep-alive", agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0", elap=49.805sec -> 200 OK
[2014-05-21 02:11:43,540]:  Default encoding: ascii (file system: UTF-8)
[2014-05-21 02:11:53,550]:  Default encoding: ascii (file system: UTF-8) (this message is repeating over and over with every webdav restart)


hope that helps helping me.

[Martin Bosner]

i just played around a bit - is seafserv-gc  still up to date?

i get the following output:

/seafile/bin/seafserv-gc -c /data/fileshare/seafile/versions/ccnet -d /data/fileshare/seafile/seafile-data --verify
[05/22/14 01:31:39] Commit b03db0efd676ee9552b31fe7463072da748969b8 is missing
[05/22/14 01:31:39] Repo e86d227c is corrupted.
[05/22/14 01:31:39] Failed to get repo e86d227c.

[Michele Innocenti]

This only happens with folders full of images?
Can you download all the files?
During some tests, I have uploaded some pictures from an Android client.
One of these was not saved correctly (according to the client instead everything was ok).
By accessing the webdav folder via pc, I noticed slowdowns due precisely to the image that the computer was unable to download to generate the preview.

[Martin Bosner]

Ok i just tested a little bit more:

created a new lib
added some hundret pdfs via 3.0.3 seafile mac os client (450mb)
had no problems downloading a file from there using webdav.

i will now do the same just for images - but this will take some time:
@JiaQiang Xu had some time taking a look into this?

[Joachim Schmid]

Did you find a solution? I have the same problem. Look likes webdav is very slow for folders containing subfolders and a lot of other documents. Accessing such a file (100KB) via webdav takes 18seconds and via Seafile webinterface below 1 second.

Everything was uploaded and accessed using version 3.1.6

Performance is also very good using the iPad version. Only webdav performance is very very poor for exactly the same file.

[Daniel Pan]

Hi All,

We are working on improve the performance of WebDAV. Hopefully we will publish a new version in the next week.