How can I download the dataset faster?

Seongyun Ko

unread,

Jan 21, 2018, 8:01:42 AM1/21/18

to Web Data Commons

Hi, I am a graduate student

I am looking for very large real graphs such as Web Data Commons - Hyperlink Graph 2012.

I am downloading its Webgraph format version which is about 52GB.

However, it says it would take more than 20 days due to the extremely slow network.

Given the urgent needs for performing experiments with this dataset, I cannot wait that long.

What should I do?

Is there any other mirror link?

Thanks.

Robert Meusel

unread,

Jan 21, 2018, 8:34:47 AM1/21/18

to Web Data Commons

Hi Seongyun,

I have started (for testing purposes) the download for the webgraph file, and I can download with 5mb/s, which means it will be done within 3 hours. I am not sure, why the traffic is that slow. Are you behind a firewall/vpn or similar which blocks the traffic?

Unfortunately, there is no other mirror where you can get the file in addition.

Best,

Robert

Seongyun Ko

unread,

Jan 23, 2018, 8:24:26 PM1/23/18

to Web Data Commons

Thanks!

Fortunately, I managed to download.

Seems the network was in congestion at the moment :D

2018년 1월 21일 일요일 오후 10시 34분 47초 UTC+9, Robert Meusel 님의 말:

Reply all

Reply to author

Forward