JSON API

[ethanc...@gmail.com]

Can Common Crawl be used as a JSON API for searches?

I'd LOVE a basic endpoint where I can send the search and receive metadata of the search results.

[Sebastian Nagel]

Hi,

there is kind of a JSON API at
http://index.commoncrawl.org/

It allows to search for URLs (domains, etc.) and returns a JSON structure containing
page metadata and the locations of the content in the crawler archives.

Best,
Sebastian

On 11/29/2017 12:25 AM, ethanc...@gmail.com wrote:
> Can Common Crawl be used as a JSON API for searches?
>
> I'd LOVE a basic endpoint where I can send the search and receive metadata of the search results.
>

> --
> You received this message because you are subscribed to the Google Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> common-crawl...@googlegroups.com <mailto:common-crawl...@googlegroups.com>.
> To post to this group, send email to common...@googlegroups.com
> <mailto:common...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/common-crawl.
> For more options, visit https://groups.google.com/d/optout.

[Tom Morris]

If you're looking for a CommonCrawl based search engine which searches the contents of pages rather than just the URLs, that's the sort of thing that Common Search was working towards, but I haven't seen any activity there in a while. 

https://about.commonsearch.org/2016/04/state-of-common-search-april-2016/

Tom

On Thu, Nov 30, 2017 at 2:42 AM, Sebastian Nagel <seba...@commoncrawl.org> wrote:

Hi,

there is kind of a JSON API at
  http://index.commoncrawl.org/

It allows to search for URLs (domains, etc.) and returns a JSON structure containing
page metadata and the locations of the content in the crawler archives.

Best,
Sebastian


On 11/29/2017 12:25 AM, ethanc...@gmail.com wrote:
> Can Common Crawl be used as a JSON API for searches?
>
> I'd LOVE a basic endpoint where I can send the search and receive metadata of the search results.
>
> --
> You received this message because you are subscribed to the Google Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> common-crawl+unsubscribe@googlegroups.com <mailto:common-crawl+unsub...@googlegroups.com>.

> To post to this group, send email to common...@googlegroups.com

> <mailto:common-crawl@googlegroups.com>.

> Visit this group at https://groups.google.com/group/common-crawl.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.

To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl+unsubscribe@googlegroups.com.
To post to this group, send email to common...@googlegroups.com.

[Bakz Awan]

could you detail more about such an endpoint?  

What do you imagine?  Type in a keyword get matching pages/domains, order by domain rank or something?