Common Crawl Google Group

Valid Segments

robert.meu...@gmail.com (Robert Meusel) — Wed, 22 May 2013 06:19:08 UT

Hi,
I know that this topic has been announced somewhere below but actually
since Q4 last year when there were 56 valid segment IDs in this file the
number of IDs has increase to 177. As we have worked with the "original" 56
segments and want to continue working some questions came into our minds:

Re: Digest for common-crawl@googlegroups.com - 5 Messages in 1 Topic

t...@tomanthony.co.uk (Tom Anthony) — Thu, 16 May 2013 08:02:38 UT

Re: Crawl Depth of Common Crawl

ja...@strangelogic.com (Jason Duke) — Wed, 15 May 2013 17:11:45 UT

Tina,

That's very kind and, of course, I can confirm it will not be distributed
onwards.

regards

Jason

---
Jason Duke

Email: ja...@strangelogic.com
Mob: +44 (0)7595 924 934
Twitter: @JasonD
LinkedIn: [link]

The information contained within this email along with any attachments are

Re: Crawl Depth of Common Crawl

l...@commoncrawl.org (Lisa Green) — Wed, 15 May 2013 17:08:14 UT

Jason - We have a rough draft of a paper summarizing these stats, but it
is not quite ready to publish. I will send you the draft by email and ask
you not to distribute it until we have the final version.

Everyone else - stay tuned, the paper will be posted soon!

Lisa

Re: Crawl Depth of Common Crawl

ja...@strangelogic.com (Jason Duke) — Wed, 15 May 2013 16:28:33 UT

It would be of interest to me :)

Thanks

Jason

---
Jason Duke

Email: ja...@strangelogic.com
Mob: +44 (0)7595 924 934
Twitter: @JasonD
LinkedIn: [link]

The information contained within this email along with any attachments are
confidential, may be legally privileged and/or protected by copyright. If

Re: Crawl Depth of Common Crawl

l...@commoncrawl.org (Lisa Green) — Wed, 15 May 2013 16:26:40 UT

Hi
I am not sure what metrics you would like to know. Sometimes people ask
questions about what percentage of the web we crawl. We crawl billions of
pages, but it is not really reasonable to talk about the percentage of the
web that we crawl because there is no consensus on just how big the web is.

Crawl Depth of Common Crawl

jhn.wood...@gmail.com — Wed, 15 May 2013 12:57:37 UT

I have read that Common Crawl claims that it is not a comprehensive crawl.
I would like to know how deep does their crawl go. Thank you for your
input.

Re: 403 Forbidden when attempting to list s3://aws-publicdatasets/common-crawl/

ahadr...@gmail.com (Ahad Rana) — Tue, 14 May 2013 21:21:30 UT

Hi Andrew,

A lot of stuff in the s3://aws-publicdatasets/common -crawl root is
temporary in nature and not World readable. In the future, we can hopefully
move all of this stuff into a separate directory under the common-crawl
root. Unfortunately, renames are not really feasible due to the size of

403 Forbidden when attempting to list s3://aws-publicdatasets/common-crawl/

amat...@gmail.com (Andrew Mattie) — Tue, 14 May 2013 20:07:04 UT

I'm just getting started w/ the common crawl data set, and I can't seem to
list the common-crawl/ or common-crawl/parse-output directories from an
interactive Pig session on a stock EMR launch.
grunt> ls s3://aws-publicdatasets/common -crawl/

I have no problem listing s3://aws-publicdatasets/common -crawl/crawl-002

Re: question aroundTerms of Use

tfmor...@gmail.com (Tom Morris) — Sun, 12 May 2013 14:20:41 UT

I'm not a lawyer, but

that sounds like a permitted use to me. My interpretation of the
non-sublicensable and respects the sites' ToUs and copyrights is intended
to prevent things like:

- publishing a "Common Crawl for Movies" subset of the crawl (which would
also violate their Common Crawl trademark)

question aroundTerms of Use

dhananjaybak...@gmail.com — Sun, 12 May 2013 05:27:00 UT

Hi,
I am bit confused about terms of using Common Crawl data:
*Terms of Use* Please refer to the Common Crawl Terms of Use <[link]>document
for a detailed, authoritative description of our Terms of Use guidelines,
but, in general, you cannot republish the data retrieved from the crawl

Re: textdata and streaming hadoop jobs

kur...@gmail.com (Kurt Jx) — Mon, 06 May 2013 02:14:38 UT

Hi Mat, Ahad, thnx for the quick reply!
yes! that's looks like just what i need.

Mat, sorry i'm not clear on how to build it. But once it's build I believe
I can just upload it to the EMR job and use it in place of
SequenceFileAsTextInput

Re: textdata and streaming hadoop jobs

matthew.kel...@gmail.com (Mat Kelcey) — Sun, 05 May 2013 16:07:49 UT

Ah, cool! Thanks Ahad!

Re: textdata and streaming hadoop jobs

ahadr...@gmail.com (Ahad Rana) — Sun, 05 May 2013 15:58:43 UT

Hi Mat,

NP. I pushed it into the commoncrawl repo for you a while back:

[link]

Ahad.

Re: textdata and streaming hadoop jobs

matthew.kel...@gmail.com (Mat Kelcey) — Sun, 05 May 2013 15:15:42 UT

yeah this is known pain in the butt when it comes to sequence files and
streaming, they just don't play well together :/

i wrote a custom input format to handle this, but i never followed through
with the PR with ahad to get this into the common crawl code (sorry ahad! i
didn't notice your comment!)