You do not have permission to delete messages in this group
Copy link
Report message
Sign in to report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to Common Crawl
Hi, I'm slightly confused what data can, and cannot be pulled the from data supplied.
Can you collect links within web pages?
Sebastian Nagel
unread,
Feb 8, 2018, 3:59:56 AM2/8/18
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Sign in to report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to common...@googlegroups.com
Hi,
> Can you collect links within web pages?
Yes, of course you can. The easiest way is to use the WAT files which
already have the links extracted and wrapped into JSON. That's easier
than to parse the HTML payload in the WARC files. See
http://commoncrawl.org/the-data/get-started/ for a description of the data formats. Sample code how to extract links
at scale you can find on github:
https://github.com/commoncrawl/cc-pyspark
On 02/07/2018 11:15 PM, jackyna...@gmail.com wrote:
> Hi, I'm slightly confused what data can, and cannot be pulled the from data supplied.
>
> Can you collect links within web pages?
>