Access to crawler's results for 2008-2012.

[Dmytro Rakovskyi]

Hello,
How can I access to the crawler's results for 2008-2012?
There are no paths to files on s3 public data set http://commoncrawl.org/the-data/get-started/.
If anybody has the file with paths I will be very thankful.
Best regards,
Rakovskyi Dmytro

[Tom Morris]

On Tue, Mar 29, 2016 at 5:06 AM, Dmytro Rakovskyi <rakov...@gmail.com> wrote:

Hello,
How can I access to the crawler's results for 2008-2012?
There are no paths to files on s3 public data set http://commoncrawl.org/the-data/get-started/.
If anybody has the file with paths I will be very thankful.

There's probably more information in some of the old posts announcing them, but see:

2008/9 s3://aws-publicdatasets/common-crawl/crawl-001/

2009/10 s3://aws-publicdatasets/common-crawl/crawl-002/

2012 s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt

Don't forget that these all pre-date the switch to the WARC format, so you'll need different tooling to deal with the ARC files.

Tom

[Tom Morris]

The blog post describing the 2012 crawl is here:

http://commoncrawl.org/2012/07/2012-crawl-data-now-available/

[Dmytro Rakovskyi]

Thank You but I have already find decision. I have parse all file paths with the help aws cli( aws s3 ls s://path). 

вівторок, 29 березня 2016 р. 19:41:54 UTC+3 користувач Tom Morris написав:

[Tom Morris]

On Wed, Mar 30, 2016 at 10:52 AM, Dmytro Rakovskyi <rakov...@gmail.com> wrote:

Thank You but I have already find decision. I have parse all file paths with the help aws cli( aws s3 ls s://path). 

You shouldn't use that technique for 2012 and later crawls because not all the files you find will be valid. That's why there's a file with the list of valid segments.

Tom

 

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To post to this group, send email to common...@googlegroups.com.
Visit this group at https://groups.google.com/group/common-crawl.
For more options, visit https://groups.google.com/d/optout.

[Dmytro Rakovskyi]

I don't think that it's possible to get paths in other way. And yep I get result only from valid segments(from valid_segments.txt).

середа, 30 березня 2016 р. 17:59:34 UTC+3 користувач Tom Morris написав: