Re: [cc] News dataset update frequency

7 views
Skip to first unread message

Sebastian Nagel

unread,
Aug 3, 2022, 3:58:21 PM (3 days ago) Aug 3
to common...@googlegroups.com
Hi Kuanysh,

> there are news datasets available separately

Just to make sure - it's about the news dataset described in [1]
available below s3://commoncrawl/crawl-data/CC-NEWS ?

> However, the updates of the datasets on AWS S3 are only for the first
> 15 days of the months for each release. I would like to know is there
> any reason behind this? And will you do any updates regarding this?

The WARC files of the news dataset are uploaded continuously
over the entire month. Every WARC file name includes the timestamp
when the WARC file was opened for writing, for example:
CC-NEWS-20220731232842-00120.warc.gz
(2022-07-31 23:28:42 UTC)

Best,
Sebastian



[1] https://commoncrawl.org/2016/10/news-dataset-available/

On 8/2/22 10:43, Kuanysh Bakirov wrote:
> Hello,
>
> First of all, I want to thank everyone who is involved in building
> useful open source project like Common Crawl. We have figured out that
> there are news datasets available separately from the web crawler and it
> is perfectly matches our use case. 
>
> However, the updates of the datasets on AWS S3 are only for the first 15
> days of the months for each release. I would like to know is there any
> reason behind this? And will you do any updates regarding this? Thank you!
>
> Best regards,
> Kuanysh
>
> --
> You received this message because you are subscribed to the Google
> Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to common-crawl...@googlegroups.com
> <mailto:common-crawl...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/common-crawl/e7aeef1f-874e-499c-965d-45887fbb1942n%40googlegroups.com
> <https://groups.google.com/d/msgid/common-crawl/e7aeef1f-874e-499c-965d-45887fbb1942n%40googlegroups.com?utm_medium=email&utm_source=footer>.
Reply all
Reply to author
Forward
0 new messages