Rate/Connection Limits

[Ratan Sebastian]

Hi,

My name is Ratan Sebastian. I'm a PhD student at Leibniz University Hannover. I'm trying to download the 2019, 2020 and 2021 datasets for a research project that I have going on.

I'm downloading with aria2 using the maximum of 16 connections and downloads in parallel. I've managed to download a fair bit of the dataset but now the downloader fails with connection aborted error messages. Aria can continue downloads from where it last left off but I think it makes a bunch of header requests for all the files that it has downloaded to verify the size. It seems like it fails at this point. For the 2019 set for instance, it has already downloaded 432 files and as it goes through requesting the http URL and being redirected to the https url it starts erroring out after a while before it actually gets to files that it needs to download. I was wondering if there are some rate limits that might be causing this? Are there any connection limits that I need to be aware of?

Thanks,

- Ratan

[Alexander Brinkmann]

Hi Ratan,

Thank you for your mail and for downloading our datasets.
Yes,  we enabled a rate limit of a maximum of 5 concurrent connections for each client.
So switching to 5 downloads at a time will likely solve the problem.

This will slow down your download, but other content like lecture material and lecture videos provided on the machine remains accessible for our students.

Thanks,

Alexander

[Ratan Sebastian]

Good to know. Thanks Alexander.