Slow Transfers? + MD5 Test

174 views
Skip to first unread message

fogWraith

unread,
Oct 12, 2008, 3:16:22 PM10/12/08
to Python FTP server library - Discussion group
I noticed when testing locally that downloads were going up and down
between 300-700kb/s, stable around 450-ish or so, but wildly jumping.

Changed the chunk size in ftpserver.py (line 790 + 791) from 8192 to
4096, and the speed jumped drastically to around 3Mb/s or more. Which
for me, at least seems more optimal than the previous setting. Just a
note / heads up.

Also, last but not least, the md5_ftpd.py, slapped me in the face with
message saying MD5 has been deprecated, so instead... hashlib perhaps?
Entire code below (should work just fine)

#!/usr/bin/env python
# md5_ftpd.py

"""A basic ftpd storing passwords as hash digests (platform
independent).
"""

import hashlib
import os

from pyftpdlib import ftpserver


class DummyMD5Authorizer(ftpserver.DummyAuthorizer):

def validate_authentication(self, username, password):
hash = hashlib.md5(password).hexdigest()
return self.user_table[username]['pwd'] == hash

if __name__ == "__main__":
# get a hash digest from a clear-text password
hash = hashlib.md5('12345').hexdigest()
authorizer = DummyMD5Authorizer()
authorizer.add_user('user', hash, os.getcwd(), perm='elradfmw')
authorizer.add_anonymous(os.getcwd())
ftp_handler = ftpserver.FTPHandler
ftp_handler.authorizer = authorizer
address = ('', 21)
ftpd = ftpserver.FTPServer(address, ftp_handler)
ftpd.serve_forever()

Giampaolo Rodola'

unread,
Oct 12, 2008, 3:42:02 PM10/12/08
to Python FTP server library - Discussion group
> I noticed when testing locally that downloads were going up and down
> between 300-700kb/s, stable around 450-ish or so, but wildly jumping.
>
> Changed the chunk size in ftpserver.py (line 790 + 791) from 8192 to
> 4096, and the speed jumped drastically to around 3Mb/s or more. Which
> for me, at least seems more optimal than the previous setting. Just a
> note / heads up.

Uhmmm are you sure? With that change the test suite runs 0.5 secs
slower.

> Also, last but not least, the md5_ftpd.py, slapped me in the face with
> message saying MD5 has been deprecated, so instead... hashlib perhaps?
> Entire code below (should work just fine)

Oh, you're right. It's been deprecated in Python 2.5.
Would you mind to open a report in the issue tracker?

fogWraith

unread,
Oct 12, 2008, 3:56:41 PM10/12/08
to Python FTP server library - Discussion group
I haven't tried the test suite, which may have been bad of me to
forget.
Either way, I've been trying downloading files with only 4096 byte
chunks for now, when grabbing larger files over the network aswell as
over the net it's faster than 8192 byte chunks, which would seem odd.
It applies to my system though and I don't know about others, so I
suppose more tests will have to be run on other system configs.

Giampaolo Rodola'

unread,
Oct 13, 2008, 1:15:18 PM10/13/08
to Python FTP server library - Discussion group
I made a bunch of other tests on Windows and Linu and it seems that
using 8192 bytes is faster.
Anyway, I talked with a friend of mine and he said that conducting
tests on localhost is not reliable because usually localhost uses a
different subsystem than connecting remotely.
A more reliable benchmark would be given by two computers on a LAN,
one using wget as a client, the other serving FTP through pyftpdlib.
Unfortunately right now I have no other machine except the one I'm
currently typing from.

Giampaolo Rodola'

unread,
Oct 13, 2008, 1:22:25 PM10/13/08
to Python FTP server library - Discussion group
By the way, could you give more details about the tests you conducted?
Platform? Python version? Is it faster only for downloads (RETR) or
also for uploads (STOR)?
Personally I'm using this simple benchmark script to conduct the
tests:

import ftplib, time
f = ftplib.FTP()
f.connect('127.0.0.1', 21)
f.login()
t = time.time()
f.retrbinary("retr testfile", lambda x: x)
print time.time() - t



On 12 Oct, 21:56, fogWraith <fogWra...@gmail.com> wrote:

fogWraith

unread,
Oct 13, 2008, 2:52:08 PM10/13/08
to Python FTP server library - Discussion group
For the time being, only a couple tests have been made from within the
LAN and via wireless clients further away, most seem to benefit from
the new settings, oddly enough... Though no upload tests have been
made just yet, I'll have some testing done with that later on.

All clients are using different FTP clients, but the result is always
the same as above, I'm going to have a few runs with other chunksizes
aswell.

The test platform for the server is running Windows Vista X64 with
Python 2.6 X64 installed.

NotAnAdminAccount

unread,
Oct 13, 2008, 4:31:39 PM10/13/08
to Python FTP server library - Discussion group
Ultimately the question of block size depends on platform as well as
network interconnect. I've seen really funny things happen on
wireless (I had a macbook pro for a while that couldn't break
100kbytes/second over WPA2 wireless), and have seen performance peak
across a gbit lan using anywhere from 4k to 32k block sizes depending
on the host and client configurations.

Generally, it's hard to go too wrong with 4k block sizes, from both a
performance and overhead perspective. It's significantly faster than
512 bytes (as was the default with asyncore/asynchat in Python 2.5),
and in testing that I have done (typically windows <-> windows on
10mbit, 100mbit, and 1gbit links) is within 10-20% of whatever block
size was the peak. Sure, you can take 8 more passes with 4k than 32k,
but if you really feel like going crazy, tossing in a loop like the
following might give a transfer rate increase...

...
first = 1
while first or sent == len(block):
first = 0
block = self.outq.popleft()
sent = self.send(block)
# handle cleanup
...

- Josiah

fogWraith

unread,
Oct 14, 2008, 2:42:06 AM10/14/08
to Python FTP server library - Discussion group
Aha, I'll keep that noted and have the clients test this tonight when
I set up the different chunksizes, run a couple of tests with small,
medium and large files... see where it goes well and where it doesn't,
hopefully to get an optimal value.

On Oct 13, 10:31 pm, NotAnAdminAccount <josiah.carl...@gmail.com>
wrote:

Giampaolo Rodola'

unread,
Oct 27, 2008, 9:02:16 AM10/27/08
to Python FTP server library - Discussion group
I made some tests on a 100 Mb Intranet between two Sun Solaris
workstations and it seems that by using 4096 the transfers are a bit
faster.
I haven't noticed the drastical difference FogWraith reported above
but given some infos I discovered on Internet I think it'd be better
using 4096 as default.

@FogWraith: could you please open a new report on the bug tracker?
Reply all
Reply to author
Forward
0 new messages