Meanpath Jan 2014 Torrent - 1.6TB of crawl data from 115m websites.

[Adam Seabrook]

Hi Data People :)

Yesterday we released a small 1.6 TB crawl of just the front page on 115 million websites that may be useful to those of you who do not need the full Common Crawl index.

You can read more about it here:

http://blog.meanpath.com/meanpath-january-2014-index/

We are huge fans of Common Crawl so want to try and support the mission where we can with occasional releases of crawl data.

Cheers,
Adam Seabrook

CEO

meanpath.com

[Pete Warden]

That's fantastic, thanks Adam!

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To post to this group, send email to common...@googlegroups.com.
Visit this group at http://groups.google.com/group/common-crawl.
For more options, visit https://groups.google.com/groups/opt_out.

--

Check out Jetpac City Guides iPhone app - Just launched!

CTO Jetpac

Follow me on twitter @petewarden

[Lisa Green]

We are super excited about working with meanpath and excited about their donation! Look for a blog post on the Common Crawl website next week.

Lisa

[fightsw...@gmail.com]

Great job,

I did almost similar thing last year - I crawled all frontpages of .com, .net and .org domains, but it took me 2 months to crawl them all :). This is much appreciated

Thanks again,

Michal 

[fightsw...@gmail.com]

OK seems I was too optimistic, the torrent seems to be dead, is there any other way to download source data?

Thanks in advance 

[Adam Seabrook]

We had a few people say that but when we test it from a few remote computers it downloads fine. I did notice that it takes 10-15mins just to start downloading which may be due to the size of the torrent?

[fightsw...@gmail.com]

May I ask which torrent client did you use? I've tested  5, 3 of them (tixati, mtorrent, bittorent) can't even open the torrent file claiming metadata corrupted or too large, other 2 FDM and ancient ABC are able to open it but never find any seeds nor peers regardless how long I keep the app open....
Thanks for help

[Adam Seabrook]

I am using transmission on OSX. I will reply to this thread when we have another download option. Possibly on S3.

--
Cheers,
Adam
http://adamseabrook.com
http://au.linkedin.com/in/adamseabrook

[Adam Gotterer]

Any update on the file location? I've also tried to download the torrent and when I opened it in my torrent app it says the file is too big (25MB).

[Adam Seabrook]

Hi Adam,

We gave up trying to torrent it as it was running into too many weird client issues. A full crawl dump is being synced over to http://archive.meanpath.com which should be complete in 4-5 hours at the current transfer speed. 

[Adam Seabrook]

Upload is now complete:

http://archive.meanpath.com/crawl20140104/

and a more recent one which contains extra fields such as BGP number and the AS Path

http://archive.meanpath.com/crawl20140226/

[Veaceslav Ustiugov]

I want to download a torrent, but the link is not working, how can I download this torrent, please help me.

[Adam Seabrook]

Hi Veaceslav,

We did have the torrent and http live for a few months but the bandwidth costs were becoming excessive so had to move it to a paid download. It does not look like there is anyone seeding the torrent now.

If you still require this data it is available but we charge a small fee to recoup the download cost. Alternatively you can get free access to CommonCrawl which has a much deeper crawl than we provide.

[Lori Bell]

meanpath.com is not available, what happened?