Errors searching for single URL

[Aline Bessa]

Hi all

I realized that Common crawl has this url here at urlsearch.commoncrawl.org: http://urlsearch.commoncrawl.org/page/1346876860611/1346928113106/2383/13960611/1402

but if I look for http://www.aids-sida.org/archivos/directorio_nacional/ags.html there, it returns a 0. Do you folks know why?

Thanks!

[Dominik Stadler]

Hi,

As the crawl is a snapshot of the state at the time it is collected, the
actual contents at the urls can be different over time, e.g. I think here you see in
the crawl how the page looked like in Feb. 2012...

Dominik

[Aline Bessa]

Sorry,  I may have expressed myself inaccurately.  What I mean is that there are pages that I can see in urlsearch.common.org but, when explicitly looking for them INSIDE common crawl, by using the Trivio's interface, I cannot fetch them. The API returns that it wasn't found in the common crawl's index (although it is listed in urlsearch.common.org -- i.e. it IS inside common crawl's index).

Suggestions on how to solve it? More generally: suggestions on looking for specific URLs inside Common Crawl in a way that always works?

Thanks!!

[John Wiseman]

That index is corrupted. See https://github.com/trivio/common_crawl_index/issues/13

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To post to this group, send email to common...@googlegroups.com.
Visit this group at http://groups.google.com/group/common-crawl.
For more options, visit https://groups.google.com/d/optout.

[ikre...@gmail.com]

The new index, available at: http://index.commoncrawl.org should have all the urls for the crawls listed.  There's not yet an index of the old data but that is planned.

[Aline Bessa]

Thanks!!