Slow uploads to EC2

[Erin N]

Hi there,

I am new to using EC2 for QIIME, and to test the waters I tried to upload a modestly large file (130MB) into a microinstance just to see how long it would take.

It is incredibly slow (60 Kbps), which is strange since when I do a speed test supposedly my realtime upload speed is 500 Kbps (theoretical max is 1.5 Mbps according to AT&T).  

Any clue why this is so slow?  Do larger instances have faster upload speeds?  Or do I need to find a faster internet connection and/or use Dropbox?

Thanks for your help,

Erin

[Daniel McDonald]

There are a lot of factors that go into throughput. Where are you
located in the world?
-Daniel

> --
>
>
>

[Erin N]

I'm in San Francisco, CA.  Seems like that should be fine?

Erin

[Antonio González Peña]

Yes that should be fine and the speed sounds normal for EC2, which are
lower than what we normally see in other environments.

https://forums.aws.amazon.com/message.jspa?messageID=376269

> --
>
>
>



--
Antonio González Peña
Research Assistant, Knight Lab
University of Colorado at Boulder
https://chem.colorado.edu/knightgroup/
http://scholar.google.com/citations?user=d5EXd78AAAAJ

[Erin N]

Hi Antonio,

If these are typical upload speeds, it would take hours (a day?) to upload my sequencing data (4Gb).  Do you have any suggestions to make this faster?

I found a link for installing dropbox on an instance...would you recommend doing something like this?

http://ged.msu.edu/angus/tutorials-2011/installing-dropbox.html

Thanks,

Erin

[Antonio González Peña]

I do not think Dropbox will be faster but you can try it. Something
that I do often is compress everything to reduce the transfer size,
then host it in a web box (this could be Dropbox but I suggest using
other resource), and then use axel. You can install axel in the
instance by:
sudo apt-get install axel
then running:
axel -n 10 -a your_url
This will create 10 connections to the file and download them in
parallel. Axel is a download accelerator:
http://axel.alioth.debian.org/

By the way, sys admins, in general, hate users that do this because
you could be saturating the network where you are hosting the file ...

Now when dealing with really large data files you can always use the
Amazon service for this purpose: http://aws.amazon.com/importexport/

[Erin Nuccio]

Hi Antonio, thanks for the tips!

Sincerely,
Erin

> --
>
>
>

[Erin N]

If anyone is curious about how Dropbox worked out for an EC2 newbie....

I thought this worked pretty well for small and medium sized files (max tested: 200MB).  I had to unlimit the bandwidth of my Dropbox uploads to make this work.  When I installed Dropbox in the instance, the small files would show up almost immediately, and the larger files would take a couple minutes to load into the Dropbox folder.  However, this is FAR better than the upload speeds I was getting through scp (up to an hour for the 200MB files).  Also, it's nice when I need to use the same file in different instances (I don't have to wait for an hour to upload in each one).

Also, I could output files to Dropbox, and they would sync and show up on my computer almost immediately (no need to download at end of session).  This worked well for beta-diversity through plots, not so well for alpha-rarefaction (too many big files, ran out of space).

I will try one of Antonio's suggestions for loading GB of data, since I'm not interested in paying for a Dropbox account.  If anyone has suggestions for a different service, I'm all ears.

Here are the links for unlimiting the bandwith for Dropbox uploads, and installing Dropbox in an instance.  I made an executable shell script so I do the install fairly easily at the beginning of the session.

http://lifehacker.com/5904745/speed-up-dropboxs-file-syncing

http://ged.msu.edu/angus/tutorials-2011/installing-dropbox.html

Cheers,

Erin