URL Index for a single page

[Aline Bessa]

Hi all,

I wish I could use the Common Crawl Index (https://github.com/trivio/common_crawl_index) to crawl specific URLs, and not entire domains. Do you know if this is possible?

Thanks

[Tom Morris]

On Wed, Feb 25, 2015 at 7:33 PM, Aline Bessa <ali...@gmail.com> wrote:

I wish I could use the Common Crawl Index (https://github.com/trivio/common_crawl_index) to crawl specific URLs, and not entire domains. Do you know if this is possible?

That's supposed to work, modulo the other known problems with the index.  What issues are you having?  What are you trying to search for?

Tom 

[Aline Bessa]

Yeah, I realized it works, but we have to invert the domain. Is there a standard way to do it? Are all URLs inverted in the same way?

Thanks!!

[Stephen Merity]

Hi Aline,

I thought I'd just mention one limitation of the URL index. The URL Index is an amazing resource, created by talented volunteers who are keen on open source and open data, that will work for searching over older crawl archives. Being volunteers, they could only donate so much time to it and it has not been updated to work with recent crawls. We're working on building a new team of volunteers to work on it.

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To post to this group, send email to common...@googlegroups.com.
Visit this group at http://groups.google.com/group/common-crawl.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Stephen Merity

Data Scientist @ Common Crawl