Extracting <a hrefs> from the data

[jackyna...@gmail.com]

Hi, I'm slightly confused what data can, and cannot be pulled the from data supplied.

Can you collect links within web pages?

[Sebastian Nagel]

Hi,

> Can you collect links within web pages?

Yes, of course you can. The easiest way is to use the WAT files which
already have the links extracted and wrapped into JSON. That's easier
than to parse the HTML payload in the WARC files. See
http://commoncrawl.org/the-data/get-started/
for a description of the data formats. Sample code how to extract links
at scale you can find on github:
https://github.com/commoncrawl/cc-pyspark

Related resources which may eventually fit your needs are
- the URL index
http://index.commoncrawl.org/
- host- and domain-level web graphs
http://commoncrawl.org/2018/02/webgraphs-nov-dec-2017-jan-2018/

Best,
Sebastian

On 02/07/2018 11:15 PM, jackyna...@gmail.com wrote:
> Hi, I'm slightly confused what data can, and cannot be pulled the from data supplied.
>
> Can you collect links within web pages?
>

> --
> You received this message because you are subscribed to the Google Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> common-crawl...@googlegroups.com <mailto:common-crawl...@googlegroups.com>.
> To post to this group, send email to common...@googlegroups.com
> <mailto:common...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/common-crawl.
> For more options, visit https://groups.google.com/d/optout.