When is the next CommonCrawl release

[Shyam Maddali]

Hello All,

      I see that the last CC release is from Mid December 2023.

When is the next release slated for ? Is the expectation monthly, quarterly. 

thanks

Shyam

[Greg Lindahl]

Shyam,

Our goal is to release a new crawl every other month. We just finished the fetch phase of the next crawl. Assuming that it passes our data quality checks, we hope to finish postprocessing by the end of this month.

-- greg

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/common-crawl/e17d3323-aab0-4ca1-b693-1f038976cf2dn%40googlegroups.com.

[Shyam Maddali]

Thanks Greg for the quick reply. 

Is there a calendar available ?

[Greg Lindahl]

Shyam,

I'm not clear on what you're asking -- do you want a Google calendar overlay, or a calendar of future crawls (which we can't do accurately) or past crawls (which is kind of available here: https://index.commoncrawl.org/collinfo.json)

-- greg

To view this discussion on the web visit https://groups.google.com/d/msgid/common-crawl/beeea52d-2b6b-4c57-832e-301465bab059n%40googlegroups.com.

[Shyam Maddali]

Thanks Greg, 

I was looking for calendar of future crawls.  I know we cant project the future calendar accurately. 

Any approximate guidance will help. 

Thanks

Shyam

[Greg Lindahl]

Shaym,

We haven't taken more than 3 months to do a crawl in many years, so
that's the best approximate guidance that I have.

One thing that is different from the past is that we're doing a
webgraph each crawl, covering the past 3 crawls. We encourage users to
take advantage of this ranking information.

-- greg