Slow Upload Speed in the Client

[Jason Tsang]

The upload speed of the seafile client is quite slow. Download speed is ok.

I have not set any bandwidth limit in the seafile client. What's the bottleneck of the seafile client?

This is the seafile.log of a 20Mb file test.

A "Test" library is created

[06/12/13 15:40:42] clone-mgr.c(306): Transition clone state for f80e428d from [init] to [fetch].

[06/12/13 15:40:42] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'init') --> ('normal', 'check')

[06/12/13 15:40:43] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'check') --> ('normal', 'commit')

[06/12/13 15:40:43] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'commit') --> ('normal', 'data')

[06/12/13 15:40:43] repo-mgr.c(120): Worktree for repo 'Testing'(f80e428d) is not set.

[06/12/13 15:40:44] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'data') --> ('finished', 'finished')

[06/12/13 15:40:44] clone-mgr.c(306): Transition clone state for f80e428d from [fetch] to [checkout].

[06/12/13 15:40:45] clone-mgr.c(306): Transition clone state for f80e428d from [checkout] to [done].

Put the 20Mb file to the "Test Folder". From the log, it takes 6mins to upload a 20Mb file from the client.

[06/12/13 15:41:15] sync-mgr.c(425): Repo 'Testing' sync state transition from 'synchronized' to 'committing'.

[06/12/13 15:41:15] sync-mgr.c(425): Repo 'Testing' sync state transition from 'committing' to 'initializing'.

[06/12/13 15:41:15] sync-mgr.c(425): Repo 'Testing' sync state transition from 'initializing' to 'uploading'.

[06/12/13 15:41:16] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'init') --> ('normal', 'check')

[06/12/13 15:41:16] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'check') --> ('normal', 'commit')

[06/12/13 15:41:16] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'commit') --> ('normal', 'fs')

[06/12/13 15:41:18] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'fs') --> ('normal', 'data')

[06/12/13 15:47:34] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'data') --> ('normal', 'update-branch')

[06/12/13 15:47:34] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'update-branch') --> ('finished', 'finished')

[06/12/13 15:47:34] sync-mgr.c(425): Repo 'Testing' sync state transition from 'uploading' to 'synchronized'.

I try the web interface upload during this peroid. It only takes about 1min to upload via the web interface.

And 5 seconds to download back to the computer.

[06/12/13 15:51:45] sync-mgr.c(425): Repo 'Testing' sync state transition from 'initializing' to 'downloading'.

[06/12/13 15:51:46] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'init') --> ('normal', 'check')

[06/12/13 15:51:46] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'check') --> ('normal', 'commit')

[06/12/13 15:51:47] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'commit') --> ('normal', 'fs')

[06/12/13 15:51:47] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'fs') --> ('normal', 'data')

[06/12/13 15:51:47] transfer-mgr.c(446): Transfer repo 'f80e428d': ('normal', 'data') --> ('finished', 'finished')

[06/12/13 15:51:47] sync-mgr.c(425): Repo 'Testing' sync state transition from 'downloading' to 'merging'.

[06/12/13 15:51:48] Fast forward.

[06/12/13 15:51:48] sync-mgr.c(654): [Sync mgr] Merged repo Testing(f80e428d).

[06/12/13 15:51:48] sync-mgr.c(425): Repo 'Testing' sync state transition from 'merging' to 'synchronized'.

[06/12/13 15:51:51] sync-mgr.c(425): Repo 'Testing' sync state transition from 'synchronized' to 'committing'.

[06/12/13 15:51:51] sync-mgr.c(425): Repo 'Testing' sync state transition from 'committing' to 'synchronized'.

Nothing happen to the ccnet.log and applet.log. So I'm not going to upload them here.

[Jason Tsang]

I'm using the latest client and server.

[JiaQiang Xu]

Is it in a LAN or WAN? What's the upload speed? Can you stably reproduce it?

The download is fast because all the blocks are already on the client cache. So actually no data is downloaded.

2013/6/12 Jason Tsang <tsangw...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "seafile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seafile+u...@googlegroups.com.
To post to this group, send email to sea...@googlegroups.com.
Visit this group at http://groups.google.com/group/seafile?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jason Tsang]

It is a WAN.

Upload speed is around 60kbps for the client and 300kbps for the web upload.

It happens on every upload via the seafile client so I can reproduce it easily.

The main question is why is there a big difference between the client & web upload?

[JiaQiang Xu]

In my WAN environment I can get 100-200KB/s (full speed of my network) when uploading 20MB file with latest client and server. Do you have any network restriction on the server side?

2013/6/12 Jason Tsang <tsangw...@gmail.com>

[Jason Tsang]

There's no network restriction on the server side. I setup a speedtest mini and I can upload files to the server at a speed of 10Mbps(at work) and 1Mbps(at home).

However, both my work computer & home computer can only achieve the best speed of 60kbps with the client upload.

-------------------------------------------

This is the 2nd test case.

File A - 16,556KB - upload via client, takes 5mins

File B - 19,590KB - upload via web interface, takes 1.5min

File B is larger than File A but it takes shorter time to upload.

Create a "Test" library

[06/12/13 16:29:38] clone-mgr.c(306): Transition clone state for 643b1467 from [init] to [fetch].

[06/12/13 16:29:39] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'init') --> ('normal', 'check')

[06/12/13 16:29:39] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'check') --> ('normal', 'commit')

[06/12/13 16:29:40] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'commit') --> ('normal', 'data')

[06/12/13 16:29:40] repo-mgr.c(120): Worktree for repo 'Test'(643b1467) is not set.

[06/12/13 16:29:40] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'data') --> ('finished', 'finished')

[06/12/13 16:29:41] clone-mgr.c(306): Transition clone state for 643b1467 from [fetch] to [checkout].

[06/12/13 16:29:41] clone-mgr.c(306): Transition clone state for 643b1467 from [checkout] to [done].

Put File A to the folder, takes 5mins to upload

[06/12/13 16:29:58] sync-mgr.c(425): Repo 'Test' sync state transition from 'synchronized' to 'committing'.

[06/12/13 16:29:59] sync-mgr.c(425): Repo 'Test' sync state transition from 'committing' to 'initializing'.

[06/12/13 16:29:59] sync-mgr.c(425): Repo 'Test' sync state transition from 'initializing' to 'uploading'.

[06/12/13 16:29:59] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'init') --> ('normal', 'check')

[06/12/13 16:30:00] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'check') --> ('normal', 'commit')

[06/12/13 16:30:00] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'commit') --> ('normal', 'fs')

[06/12/13 16:30:01] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'fs') --> ('normal', 'data')

[06/12/13 16:34:51] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'data') --> ('normal', 'update-branch')

[06/12/13 16:34:51] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'update-branch') --> ('finished', 'finished')

[06/12/13 16:34:52] sync-mgr.c(425): Repo 'Test' sync state transition from 'uploading' to 'synchronized'.

As soon as file A finishes upload, I upload File B via the web. It takes 1.5mins to upload on the web interface.

It takes 30sec for File B to download

[06/12/13 16:36:22] sync-mgr.c(425): Repo 'Test' sync state transition from 'initializing' to 'downloading'.

[06/12/13 16:36:23] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'init') --> ('normal', 'check')

[06/12/13 16:36:24] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'check') --> ('normal', 'commit')

[06/12/13 16:36:24] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'commit') --> ('normal', 'fs')

[06/12/13 16:36:25] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'fs') --> ('normal', 'data')

[06/12/13 16:36:48] transfer-mgr.c(446): Transfer repo '643b1467': ('normal', 'data') --> ('finished', 'finished')

[06/12/13 16:36:49] sync-mgr.c(425): Repo 'Test' sync state transition from 'downloading' to 'merging'.

[06/12/13 16:36:49] Fast forward.

[06/12/13 16:36:49] sync-mgr.c(654): [Sync mgr] Merged repo Test(643b1467).

[06/12/13 16:36:49] sync-mgr.c(425): Repo 'Test' sync state transition from 'merging' to 'synchronized'.

[06/12/13 16:36:54] sync-mgr.c(425): Repo 'Test' sync state transition from 'synchronized' to 'committing'.

[06/12/13 16:36:54] sync-mgr.c(425): Repo 'Test' sync state transition from 'committing' to 'synchronized'.

[JiaQiang Xu]

By network restriction, I mean is there upload bandwidth limit for non-http ports on the server side?

Seafile use port 12001 for the client upload/download.

I cannot reproduce your problem though.

2013/6/12 Jason Tsang <tsangw...@gmail.com>

[Jason Tsang]

All settings are default Ubuntu settings and I'm not aware there's any bandwidth limit on the server side.

I owns the VM in a datacentre. The server can stream youtube without any problem via VPN, which is already on a non-http port.

Is it possible to have more detailed logs to diagnose the bottleneck?

[darkbl...@gmail.com]

@JiaQiang Xu, jason tsang

what if setup seafile with nginx as reverse proxy? does it improve the transfer performance?

在 2013年6月12日星期三UTC+8下午3时44分57秒，Jason Tsang写道：

[JiaQiang Xu]

Unfortunately not. Have you tried scp? You can also setup a local VM to test.

2013/6/12 Jason Tsang <tsangw...@gmail.com>

[JiaQiang Xu]

Have you tried seacloud.cc?

2013/6/12 JiaQiang Xu <xjqki...@gmail.com>

[Jason Tsang]

Just signed up and tested..

upload speed is around 25~30KB/s

Regards,
Jason Tsang © 1997 - 2013

--
You received this message because you are subscribed to a topic in the Google Groups "seafile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/seafile/HIRnPcrbnWU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to seafile+u...@googlegroups.com.

[Jason Tsang]

tried... doesn't improve seafile client transfer performance.

Regards,
Jason Tsang © 1997 - 2013

--
You received this message because you are subscribed to a topic in the Google Groups "seafile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/seafile/HIRnPcrbnWU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to seafile+u...@googlegroups.com.

[Jason Tsang]

upload speed is around 25~30KBs with the client and 100KBs with the web interface.

The web interface upload is more closer to my home's upload speed limit.

Regards,
Jason Tsang © 1997 - 2013

[JiaQiang Xu]

Try install 1.6.1 client and see what happens.

https://code.google.com/p/seafile/downloads/list 

2013/6/12 Jason Tsang <tsangw...@gmail.com>

[Jason Tsang]

35~40KBs

Regards,
Jason Tsang © 1997 - 2013

[JiaQiang Xu]

What OS are you using?

2013/6/12 Jason Tsang <tsangw...@gmail.com>

--

You received this message because you are subscribed to the Google Groups "seafile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seafile+u...@googlegroups.com.

[Jason Tsang]

Windows 7 64Bit

[JiaQiang Xu]

Weird enough. I get 100-200KB/s on seacloud.cc. Where are you? I'm in Beijing.

2013/6/12 Jason Tsang <tsangw...@gmail.com>

--

You received this message because you are subscribed to the Google Groups "seafile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seafile+u...@googlegroups.com.

[Jason Tsang]

I know.. :(

I'm in Sydney.

[JiaQiang Xu]

OK, I tested on Win 7 64bit. The same result as you. I'll look into this problem.

2013/6/12 Jason Tsang <tsangw...@gmail.com>

[Jason Tsang]

Awesome.. Thank you.

[JiaQiang Xu]

This is a TCP performance problem. On WIndows, the default TCP send buffer size is too small for high latency network. You can ping your own server to see the RTT. I've fixed this in the code and will give you a new package to try.

2013/6/12 Jason Tsang <tsangw...@gmail.com>

[Jason Tsang]

The rtt is 150ms to the server.

Thanks for your help.. looking forward to the new package.

[Shuai Lin]

Hi, here is the new package. We have tested it on Windows XP and Windows 7 64bit. Please try it.

https://seacloud.cc/f/7cb3b0315e/

--
You received this message because you are subscribed to the Google Groups "seafile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seafile+u...@googlegroups.com.
To post to this group, send email to sea...@googlegroups.com.

Visit this group at http://groups.google.com/group/seafile.

[JiaQiang Xu]

Hi Jason,

Any result?

2013/6/17 Shuai Lin <linshu...@gmail.com>

[Jason Tsang]

The client upload speed varies and fluctuate. 

The pattern is a bit like the additive increase multiplicative decrease algorithm of the TCP Congestion Avoidance.

[JiaQiang Xu]

That's possible if the connection is lossy. But what's the top speed? Is it reasonable now?

2013/6/18 Jason Tsang <tsangw...@gmail.com>

[Jason Tsang]

Despite the fluctuation, the top speed is more reasonable now...

[Netflix Boundaries]

Same issue with Seafile Client for OS X  1.8

Please advise.

[JiaQiang Xu]

We'll have a look.

2013/10/5 Netflix Boundaries <netflix.b...@gmail.com>

[Jan Ecker]

Hi,

I'm new to seafile and everything seems to work well. But I have also issues with very slow upload speeds witht he seafile client. I'm accessing my Seafile Server from local Network and the Upload speed is around 7-14 kb/s. The Seafile Server is running on my virtualized Home Server. With other Services on the same machine I have no Problem and can reach full Gigabit-Ethernet Speed. The Seafile Server is running in an on fresh Debian VM.

The only thing i could figure out using Wireshark is, that there are a lot of droped TCP Packages. There is a factor 2 between valid and invalid packages, means 2 packages get droped while 1 is valid.

I have this issu on two different machines using Windows 7 64bit. If this would help, i could try using a Linux Client as well.

[JiaQiang Xu]

This doesn't look like a problem of the software itself. Can you send me the log messages from the client? It's in C:\users\<username>\ccnet\logs\seafile.log

2013/10/8 Jan Ecker <in...@jan-ecker.eu>

[Jan Ecker]

I allready found the error in my setup. The Problem was, that i used a mounted NFS share to store the seafile data. I found out that the porformance of that is very poor in connection with seafile. That was the bottleneck! After moving storage to a local drive I have now nice sync speeds.

Thank you for your help anyway!

[Lingtao Pan]

Hi,  if you use a mounted NFS to store seafile data, you should use MySQL backend. If you use the default sqlite3 backend, the sqlite database file will be saved in the NFS and has a very poor performance.

[JiaQiang Xu]

We've tested OS X client in a high deley-bandwidth product environment, and it has good upload speed.

This upload speed problem only appears in such environment.

Can you upload a big file via the web (seahub) to see the real network speed?

2013/10/5 Netflix Boundaries <netflix.b...@gmail.com>