Aspera Ascp Download
Avice Michonski
I need to use Aspera, which I have installed and tested with the company demo. The Connect plugin (3.6) fails to work on Windows/Linux with Chrome/Firefox/Edge. I would rather try the ascp command line utility.
All I get is ascp: failed to authenticate, exiting., and no log files. If you are familiar with ascp, could you please post the command line you are using? Did you generate some sort of SSH key for authentification?
As you describe in the linked thread, you tried it on different machines and it worked. Hence, it is specific to the server you're on. Talk to the admins and discuss whether some firewalls are in place or whether ports that aspera requires are not open. If this all does not work then just use regular wget on the ftp link:
As for using wget, unfortunately, it's VERY slow, giving me the max speed of 500 KB/s when ascp is giving me 100 MB/s. So it's just not feasible. In any case, I agree with you about talking to admins. Thanks for your input, and I wish you a Happy New Year!
I had this same issue (and asked myself why anyone would use this over rsync), and then found the answer. It is extremely non-intuitive, but you must download the Aspera Connect browser plugin, and the CLI utility ascp is included in that package. If you still have issues, I have it in my Dropbox.
I don't usually indulge in thread necromancy, but I couldn't help noticing as I read this comment that it's now been 5.4 years since it was made; and we're still stuck with ascp. But hey, the browser plugin installer apparently hasn't been updated since then, so at least we can say some of the pessimism is/was warranted.
To make this a slightly-less-useless comment, I'll note that bare CLIs are available at ososa's link below for all platforms, not just Windows, and the version numbers there are higher than on the browser plugin, if that means anything. It still installs into the boneheaded hardcoded $HOME/.aspera directory, though.
P.S. I tried writing several rex statements to extract the ascp filename, ignore most flags, then one source filename, and finally target user, host and path for the above two statements - and got stuck - my multiple rex statements are stepping on each other and thus do not seem to be the best mechanism. The code below isn't working properly.
The argument to -i may also be different depending on the location of the default key file. The command should not ask you for a password. All the IGSR data is accessible without a password but you do need to give ascp the ssh key to complete the command.
Our aspera browser interace no longer works. If you wish to download files using a web interface we recommend using the Globus interface we present. If you are previously relied on the aspera web interface and wish to discuss the matter please email us at in...@1000genomes.org to discuss your options.
Aspera Connect is software that allows download and upload via a web plugin for popular browsers on machines running Linux, Windows, and Macintosh. The software also includes a command line tool (ascp) that allows scripted data transfer. The software client is free for users exchanging data with NCBI.
Here the my_data.tar.gz file would be uploaded to the my_project folder on the NeMO server. Uploading a directory of files would not require any changes as aspera will recognize a folder is being transferred and will recursively step through ensuring that all files found in the directory are transferred.
e.g. If you want to use aspera to download a non-public data file using datahub (dcc) authentication,provide the ENA datahub username (dcc_*) instead of era-fasp and you will be prompted for the password.(or use environment variable ASPERA_SCP_PASS)
To download things from NCBI a bit faster, you can try aspera connect. This is proprietary, closed-source, software that the NCBI uses for large data transfers, but to run it in batch you need to figure out where to download it from and what to do with it.
I just tried a 10 MB and a 2 GB download from the command line and they were both ok. I did use a slightly different command, with a bandwidth limit: -l 200M.
I found that in some NCBI documentation in the 1000genomes folder on the ftp site- there is an Aspera_Users.README file and an aspera_transfer_guide.pdf there.
By the way, for other Mac users, Aspera put my ascp file in:
/Applications/Aspera Connect.app
/Contents/Resources/
It was actually inside the application file and you need to escape the space with a backslash to get the path to that. The .putty file was in the same directory.
If you type ascp without any argument you will see a small help output. To resume transfers use -k2. Also, the -l argument probably needs to be set to something you network and storage are capable of. A 7200rpm disk will probably maintain 150-200Mbps, so you could try -l200m
Yes, I was aware that typing ascp without any argument would show me all options. However, there is no information in the output to suggest that -k2 can be used to resume a download (any chance John works for Aspera?). In fact, I just tested that option and it didn't work for me at all.
I have edited my initial blog post to show the options for ascp.
In regards to the bandwidth limit, I had also previously tried that but I still got the same connection error. To be clear, my speeds were not even getting close to 200M (max was 35M) so setting that option seems like it would not have any effect. Also, who are you expecting would be able to have bandwidth speeds this high to NCBI?
I do work for Aspera. To get a complete response to what your problem is (there are a lot of potential issues) you should contact sup...@asperasoft.com, open a ticket and let them know you are doing transfers with NCBI. They will ask for your log files, which contain metrics we can look at to find out what the problems are. If you want to isolate log files (normally written to syslog) to send to support, the -L switch can be used to specify a log directory. For example:
$ ascp -i -QT -l 35m -k2 -L /tmp/aspera user@host:ncbifiles /data
The /tmp/aspera directory will need to exist, but after your transfer you will see some log files.
As for the -k2, that option specifies how to deal with partial files. The support team can also look at the logs to see why this did not work.
I have some ideas what is going on, but the support team is very helpful and the log files will help us identify the exact cause.
Morgan,
Your usage missed a fundamental capability in our software. The 'ascp' command line binary MUST have a -Q option in place to use the adaptive rate control. Without it, the software is using a fixed rate mode, without regard for the available bandwidth, and hence the potential for disconnecting your own transfers due to overdrive of connection. This is the sole reason for the disconnects you experienced and the fluctuating speed.
If you use the -Q option, the target transfer rate (-l) does not need to be adjusted at all. This was your primary problem.
I suppose we should make this the default in the 'ascp' command line (it is in the standalone products) but not in the command line.
Morgan, additionally, we are also releasing our 2.6 version of the software with disk-based rate control enabled. This is important for very high speed transfers where the network bottleneck speeds exceeds the disk throughput - basically extends the congestion avoidance of the adaptive rate control through the disk. At the speeds you tested, this is not important, but for other NCBI users it is.
I am taking time to reply on this because it is extremely important that users understand how to properly use the software, to realize its intended performance.
Thanks,
Michelle Munson
President, Aspera, Inc.
Michelle
Hi Michelle,
Thanks for your comments. You are the third person from your company to contact me about how to fix my problem. I am surprised that your support person didn't mentioned that I should use the -Q option?!
You suggest looking at the man page, but there is no man page documentation included with the "Aspera Connect" package. There may be a man page with the "Aspera Client", but this is not available to download by the general public (username & password is requested).
Also, the -K2 option only works if the -l option is used.
Considering the large number of users that will eventually be using the ascp command line program it would be great if many of these options were default or better explained.
As a further comment, there seems to be some limitations with downloading directories with the command line. The files start to download, but then get a permission denied error on hitting the file ".a.swp". That is fine since I don't want that file, but ascp will error and not continue onto the next file.
Also, it seems that I can't use a "*" to only get some of the files in a directory, which I can do with FTP.
Morgan,
Ric Mackie in our support team did in fact tell you to use the -Q flag when downloading to engage adaptive rate control.
Regarding the man page inclusion with 'ascp', we do not include the man page with the Connect client installer because it is intended to be used as a browser plug-in application. You are using the contained 'ascp' binary (which is fine) but that is typically done when a user has installed our desktop client package, Aspera client. This package (rpm or deb) includes the man page.
Regarding the permission denied errors when transferring these files, ascp uses the native file system permissions to determine if it can read or write files. Are you certain that the user account context in which the 'ascp' process is running has access to read and/or write the special files in question (on the source or destination, whichever applies)?
Regarding support of glob matches (*), great point. We are actually adding support for that in our 2.6 release series. I completely agree ascp should support it.
Regarding -k2 depending on the use of the -l option, I haven't noticed that myself, but if -l is not specified at the command line, ascp uses a default target rate of 10 Mbps (which limits the transfer rate to 10 Mbps) on top of the automatically adapted rate -- probably not what you want. You will want to include a "-l" in your command line options that is as high as you would ever want the transfer to be, e.g. -l 200M in your case.
I realize there are a number of command line options, but it is worth learning them to get the results you want. Users of our GUI client products don't have to worry about any of these -- they are built in to the application.
Thanks for all of your feedback.
Michelle