Slow (100kb/sec) Amazon S3 upload speeds from Australia

[James Collins]

Hi everyone,

We've recently started having an issue when uploading files to the US standard Amazon S3 region from our Australian dedicated server, and I was hoping one of you would be able to give us some advice.

Uploads used to go at 500+kb/sec, but are now only getting 98 - 100kb/sec (since approx 10 January 2011).

Uploads from our other dedicated severs (in the US and UK) still seem to be fine.

All of our servers are running CentOS 5.x.

We have been using the s3sync.rb script (http://s3sync.net/) to perform this upload/sync since March 2008 and have had very few issues. The command that we've been using is:

 # s3sync.rb -v --ssl -r /backups/daily $HOSTNAME-backups:

(where $HOSTNAME is the server's host name)

Running the sync in debug mode:

 # s3sync.rb -v --progress --debug --ssl  -r /backups/daily $HOSTNAME-backups:

reveals an upload speed of no more than 100kb/sec. We also seem to receive the occasional "broken pipe" error.

The files we're uploading are .tar.gz files that range from 1MB to 2.5GB in file size.

Our web host has eliminated the possibility of problems at their end (they forced our server to use a different uplink provider and it didn't help much).

In reality, the s3sync.rb script no longer seems to be actively maintained, so if anyone knows of any better ways to sync one or more directories on a Linux server with an Amazon S3 bucket then I'd be happy to give it a try.

In particular, I know that s3sync.rb doesn't support Multipart uploads, so if there was another tool out there that did, then that might help us with the upload speeds.

Thanks very much.

James