Any other WARC archives like Common Crawl?

[Liz Ron]

I've been using Common Crawl files for my research at the univ and was wondering if there are any other services (commercial) or free like common crawl which have crawl files? (WARC) or any other format?

Thanks,

Liz.

[Ivan Habernal]

Hi Liz,

Have a look at ClueWeb, it's been extensively used in information retrieval research: http://www.lemurproject.org/clueweb12/

Best,

Ivan

[Liz Ron]

Thanks Ivan!

I'm looking for more updated data, like from the recent two~ years. Any hints would be appreciated.

Liz.

[OneSpeedFast]

Hi Liz,

I`d be interested in hearing more about the sort of data you need. I have been releasing tools upon request on w3metrics.com.

Some of the big ones I have so far are:

• Web host distribution - tracking # of domains on a host
• IP distribution - tracking # of domains on an IP
• SEO/HTML related metrics
• Tracking domain squatters
• Mobile vs Desktop sites

And a bunch of other ones - including some filters for people to use in their own spidering projects. If I have data that will work on your project, i`m happy to setup a API or Feed of some sort.

Carlos

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl+unsubscribe@googlegroups.com.
To post to this group, send email to common...@googlegroups.com.
Visit this group at https://groups.google.com/group/common-crawl.
For more options, visit https://groups.google.com/d/optout.