s3://commoncrawl/ access denied

[Borislav Agapiev]

Hello,

I have been using Common Crawl over the years (thank you very much :) ), most recently a few weeks ago. I used successfully s3://aws-publicdatasets/common-crawl/ bucket. I see it is still available but not with all the data (only one arch from 2016), so I assume you are transitioning to s3://commoncrawl/.

My issue is - how to get access permissions for it (s3://commoncrawl/)? I tried many things and could not get it to work, it keeps giving me access denied. Thanks and keep up the great work

Borislav

[Borislav Agapiev]

Never mind, I figured it out, needed to change  my IAM policy to commoncrawl instead of aws-publicdatasets.

[Sebastian Nagel]

Hi Borislav,

no problem. If there are any further problems, please report them.

There should be no difference to use the data on the new bucket: the data is public readable as
before, only the bucket / path prefix has changed.

The transition to the new bucket has been announced here:
https://groups.google.com/forum/#!topic/common-crawl/nKuQK68rebo

Best,
Sebastian

> --
> You received this message because you are subscribed to the Google Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> common-crawl...@googlegroups.com <mailto:common-crawl...@googlegroups.com>.
> To post to this group, send email to common...@googlegroups.com
> <mailto:common...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/common-crawl.
> For more options, visit https://groups.google.com/d/optout.

[Christian Lund]

Hi Borislav,

I'm having trouble copying crawl data from the S3 to an EC2. Could you perhaps elaborate on how you solved it?