Re: [cc] Searching for paths with the Common Crawl Index server (and some unexpected behaviour)

[Tom Morris]

The index server does prefix matches on URLs with inverted host names, so your examples, as transformed, would be:

com.000/foo

com.aaa/foo

com.aardvark/foo

You can do prefix matches on com.aardvark.* or com.aardvark/foo/*, but com.aardvark.*/foo or com.*/foo aren't prefix matches.

Having said that, the columnar index makes it easy to do a table scan matching against whatever pattern you like.

https://commoncrawl.org/2018/03/index-to-warc-files-and-urls-in-columnar-format/

Tom

On Fri, Dec 25, 2020 at 6:48 PM Andy Mackie <andyma...@gmail.com> wrote:

Hi all,

I'm trying to search for all URLs with a certain path, such as '*.com/foo' or even '*/foo'. On executing this query, I'd expect to see results such as:

https://000.com/foo

https://aaa.com/foo

https://aardvark.com/foo

etc.

Testing the Common Crawl Index server, I see that queries can include a path for a specific domain - the following query works:

http://index.commoncrawl.org/CC-MAIN-2020-50-index?url=en.wikipedia.org/wiki/Main_Page&fl=url

When a sub-domain wildcard is introduced, however, the path component of the query is ignored (you receive every URL in the index for all sub-domains):

http://index.commoncrawl.org/CC-MAIN-2020-50-index?url=*.wikipedia.org/wiki/Main_Page&fl=url

Is this expected behavour?

Is there a way that the index server can support paths with sub-domain wildcards, please?

Or is there another way of achieving the same result, please, with existing tools? (I know that I can download the 200GB URL index files and create my own tool, but I'd rather use an existing tool if one exists).

Thanks.

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/common-crawl/b68dd7e5-e9c9-4303-a251-71ad2190d5f4n%40googlegroups.com.