how to read all the data of Common Crawl from AWS with Java?

[Pierre Therrode]

Hello,

how to read all the data of Common Crawl from AWS with Java?
I've allready posted my question on stackoverflow, but I've yet response (http://stackoverflow.com/questions/31287956/how-to-read-all-the-data-of-common-crawl-from-aws-with-java).

If someone has an idea, I'm interested.

Thanks in advance.

[Mat Kelcey]

here's some code that's kinda similar, you might get some ideas from it. 

wrote it awhile go though so it might be a bit stale sorry

mat

https://github.com/matpalm/common-crawl-quick-hacks/blob/master/links_in_metadata/src/com/matpalm/ExtractTldLinks.java

[Robert Meusel]

He, 

we are maining a framework running in AWS which you might want to look at:

http://webdatacommons.org/framework/

Cheers,

Robert

[sohail ahmed]

Here is the java code that I used to get CC Data. Build using https://github.com/commoncrawl/cc-warc-examples

https://github.com/commoncrawl/example-warc-java    

[Pierre Therrode]

Hi,
I coded a little program to remotely read the WAT file. If people have encountered the same problems , here's my github deposit : https://github.com/pi-2r/myReaderS3Bucket

[Pierre Therrode]

Yes,

I've founded this good example, but you need to :

 Configuration conf = getConf();
 //
 Job job = new Job(conf);

to

Configuration conf = new Configuration();
conf.set("fs.s3n.awsAccessKeyId", "your_key");
conf.set("fs.s3n.awsSecretAccessKey", "your_key");
Job job = new Job(conf);

otherwise you will encounter errors connections to AWS .

[Jaffer Wilson]

Hello Pierre,
May I know are you interested in the program for extracting the data or just the extracted data from the commoncrawl corpus?
Kindly, let me know.
Regards,
Jaffer Wilson