WWW Ranking - Update

[Shawn Lucas]

Hi,

Love the Ranking guide at http://wwwranking.webdatacommons.org/. Given it's quite outdated since 2014, I was wondering two questions:

1) Is the team planning to run a new update in the future?

2) If there is a contact I could reach out to about running my own process to update with latest CC data dumps. 

In regards in #2, would like to understand more about the EC2/EMR configs used prior to estimate costing, etc.

[Sebastian Nagel]

Hi Shawn,

since 2017 Common Crawl releases web graphs and graph-based rankings [1]. Best, have a look at the latest one [2].

The methods are similar than those of the 2012/2014 Web Data Commons graphs, or in other words: the WDC software

was one reference or source of inspiration, used to set up the graph construction and processing. All software is open source

and linked on [1,2].

Best,

Sebastian

[1] https://commoncrawl.org/2017/05/hostgraph-2017-feb-mar-apr-crawls/

[2] https://commoncrawl.org/2022/03/host-and-domain-level-web-graphs-oct-nov-jan-2021-2022/

[Shawn Lucas]

Thanks Sebastian! 

Super helpful, really appreciate it. I'll take a look at both links, play with the data, and write back if any questions.

Cheers,

Shawn

[Chillar Anand]

All the posts related to WWW-Ranking are tagged in https://commoncrawl.org/category/web-graph/

If Categories/Tags are exposed to users in common-crawl blog, it would be easy for users to browse based on them.