Can't find raw data for wwwranking.webdatacommons.org

[Kevin Burton]

I'm trying to find the raw for the data backing this:

http://wwwranking.webdatacommons.org/

I'm really only interested in domain + pagerank... but it would also be nice to have all the URLs and their pagerank.

I can only seem to find the raw webgraph data (nodes+edges) ... 

[Stephen Merity]

Hi Kevin,

That data was created by our friends at the Web Data Commons team. The ranking data you're interested in is released on their Hyperlink Graph 2012 download page under the Ranking Files header.

curl -vs http://data.dws.informatik.uni-mannheim.de/hyperlinkgraph/2012-08/ranking/hostgraph-katz.tsv.gz | zcat | head

wordpress.org 632.7674519980175

youtube.com 586.7626372215032

gmpg.org 481.0557395344608

en.wikipedia.org 434.10923441544645

tumblr.com 314.63741871761187

twitter.com 295.99695933648536

google.com 224.4808039510172

flickr.com 205.62921712369032

rtalabel.org 177.16420282889948

wordpress.com 175.24291503904166

Calculating the pagerank for the domain level graph yourself is also quite feasible with a standard laptop and tools such as FlashGraph or Dato GraphLab Create. Calculating the URL level graph is also possible with both tools though the machines you'll need to use will need to be far more powerful. For GraphLab Create, a single EC2 r3.8xlarge instance takes 10 minutes for a single iteration of PageRank.

If there is sufficient interest, we may be able to release URL level PageRank information in the near future.

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To post to this group, send email to common...@googlegroups.com.
Visit this group at http://groups.google.com/group/common-crawl.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Stephen Merity

Data Scientist @ Common Crawl