News crawl daily updates recrawl vs delta?

[Tim Allison]

Is there any documentation on how Common Crawl's news crawl handles changes?  Does a daily snapshot cover literally whatever was crawled, or does it only include a delta from the day before?

Thank you!

Cheers,

        Tim

[Sebastian Nagel]

Hi Tim,

every news article is crawled only once. The URLs come from a recently updated news feeds
or news sitemaps. WARC files are published soon after they're written, there are multiple
WARC files per day. Ideally, there should be no duplicate content at all, in practice there
may be bugs or some sites may publish the same content repeatedly.

Best,
Sebastian

> --
> You received this message because you are subscribed to the Google Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> common-crawl...@googlegroups.com <mailto:common-crawl...@googlegroups.com>.
> To post to this group, send email to common...@googlegroups.com
> <mailto:common...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/common-crawl.
> For more options, visit https://groups.google.com/d/optout.