General Search

[yomoore]

Hi,

I have a qeustion. I want to use the ''web search'' from:

http://code.google.com/apis/ajax/playground/#the_hello_world_of_general_search

I have read somewhere that I'm only allowed 1000 queries a day, as
stated here:

http://code.google.com/apis/soapsearch/api_faq.html#gen12

But at the verry top of that page it says clearly: Google SOAP Search
API (No Longer Available)

So I'm a bit confused: What is ''no longer available'' addressing? Is
it the SOAP or the API?? Because the API is still in use..??

I hope someone can answer my question because I want to integrate the
API with my website and I will have more than 1000 queries a day for
sure.

[Jeremy Geerdes]

On the internet now, you can find documentation for three DIFFERENT Google search APIs. The first API that they released used an approach called SOAP which returned results in an XML format. This is what the documentation you were seeing at http://code.google.com/apis/soapsearch/api_faq.html#gen12 is referring to. The SOAP Search API would return up to 1,000 results for a search term, but it also had a limit of 1,000 queries per day, but it was discontinued in 2008 or 2009.

The second rendition of a Google search API was originally called the AJAX Search API. It did not have a hard limit on the number of requests you could send per day, but it was throttled and restricted in other ways to discourage SEO applications from using the service (e.g., it would only return up to 64 results for a normal query, 100 for Custom Search Engine query). This is what you were seeing demonstrated at http://code.google.com/apis/ajax/playground/#the_hello_world_of_general_search . This API has been expanded for use beyond AJAX, so it became called simply the WebSearch API, and you can find documentation at http://code.google.com/apis/websearch/docs/ . You will notice at the top of that page, though, that this API has been deprecated since November 2010. This means that, while it still works and Google will still maintain it, it is scheduled for shutdown no later than November 2013.

The WebSearch API is being succeeded by the Custom Search API, which is really designed to interface a Google Custom Search Engine (see http://www.google.com/cse for more information) rather than simply search the general web. There are advantages and disadvantages to this API. Two big advantages are that it will return up to 100 results, and those results will generally be pretty good because CSE's are designed to be "curated" - that is, you set it up, pick the sites you want to search, and so you don't get a lot of garbage results. But there are also a couple of big disadvantages to it. For instance, the CS API will not search the general web; it must be coupled with a CSE, effectively rendering it useless for SEO applications and general search functionality. And the big disadvantage is that it is actually limited to only 100 requests per day. You can request more quota than that via the APIs Console, but the criteria on which the increased quota is granted - and the timeframe in which it will be processed - are absolutely unclear. You can sign up for the CS API and find the documentation for it and a number of other Google APIs in the API Console at http://code.google.com/apis/console

Hope this helps clear things up a bit.

Jeremy R. Geerdes

Generally Cool Guy

Des Moines, IA

For more information or a project quote:

jrge...@gmail.com

If you're in the Des Moines, IA, area, check out Debra Heights Wesleyan Church!

--
You received this message because you are subscribed to the Google Groups "Google AJAX APIs" group.
To post to this group, send email to google-ajax...@googlegroups.com.
To unsubscribe from this group, send email to google-ajax-searc...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-ajax-search-api?hl=en.

[dalila benachenhou]

Is Google returning the SOAP Search API to search the general Web?

Dalila

[dalila benachenhou]

From what I understand, Google will not provide an API for general web search.

Dalila

On Mar 8, 2011, at 7:53 AM, Jeremy Geerdes wrote:

[yomoore]

Hi @Jeremy Geerdes,

Thank you so much for your answer.
I'm still not convinced about the unlimited queries. Because on this
link:

http://code.google.com/apis/websearch/docs/

it clearly states: "It will continue to work as per our deprecation
policy, but the number of requests you may make per day will be
limited."

Do you perhaps have any information about this ''limit' and what it
means for a webdeveloper?






On Mar 8, 1:53 pm, Jeremy Geerdes <jrgeer...@gmail.com> wrote:
> On the internet now, you can find documentation for three DIFFERENT Google search APIs. The first API that they released used an approach called SOAP which returned results in an XML format. This is what the documentation you were seeing athttp://code.google.com/apis/soapsearch/api_faq.html#gen12is referring to. The SOAP Search API would return up to 1,000 results for a search term, but it also had a limit of 1,000 queries per day, but it was discontinued in 2008 or 2009.
>
> The second rendition of a Google search API was originally called the AJAX Search API. It did not have a hard limit on the number of requests you could send per day, but it was throttled and restricted in other ways to discourage SEO applications from using the service (e.g., it would only return up to 64 results for a normal query, 100 for Custom Search Engine query). This is what you were seeing demonstrated athttp://code.google.com/apis/ajax/playground/#the_hello_world_of_gener.... This API has been expanded for use beyond AJAX, so it became called simply the WebSearch API, and you can find documentation athttp://code.google.com/apis/websearch/docs/. You will notice at the top of that page, though, that this API has been deprecated since November 2010. This means that, while it still works and Google will still maintain it, it is scheduled for shutdown no later than November 2013.
>
> The WebSearch API is being succeeded by the Custom Search API, which is really designed to interface a Google Custom Search Engine (seehttp://www.google.com/csefor more information) rather than simply search the general web. There are advantages and disadvantages to this API. Two big advantages are that it will return up to 100 results, and those results will generally be pretty good because CSE's are designed to be "curated" - that is, you set it up, pick the sites you want to search, and so you don't get a lot of garbage results. But there are also a couple of big disadvantages to it. For instance, the CS API will not search the general web; it must be coupled with a CSE, effectively rendering it useless for SEO applications and general search functionality. And the big disadvantage is that it is actually limited to only 100 requests per day. You can request more quota than that via the APIs Console, but the criteria on which the increased quota is granted - and the timeframe in which it will be processed - are absolutely unclear. You can sign up for the CS API and find the documentation for it and a number of other Google APIs in the API Console athttp://code.google.com/apis/console

>
> Hope this helps clear things up a bit.
>
> Jeremy R. Geerdes
> Generally Cool Guy
> Des Moines, IA
>
> For more information or a project quote:

> jrgeer...@gmail.com

>
> If you're in the Des Moines, IA, area, check out Debra Heights Wesleyan Church!
>
> On Mar 8, 2011, at 4:28 AM, yomoore wrote:
>
>
>
>
>
>
>
> > Hi,
>
> > I have a qeustion. I want to use the ''web search'' from:
>

> >http://code.google.com/apis/ajax/playground/#the_hello_world_of_gener...

[Jeremy Geerdes]

I should have explained that a little better. The AJAX Search API employs dynamic throttling to try and detect abuse of the system and block it. So if you were to send a massive number of requests in rapid succession, it would cut you off for suspected TOS violation. The throttles have been tightened since the API was deprecated, but there is still no hard limit on the number of requests you can send.

That said, the API's TOS very clearly prohibit the use of automated requests. So you can't set up an application to simply crawl Google's results or even really to poll for site ranking, etc.

Jeremy R. Geerdes
Generally Cool Guy
Des Moines, IA

For more information or a project quote:

jrge...@gmail.com

If you're in the Des Moines, IA, area, check out Debra Heights Wesleyan Church!

[Jeremy Geerdes]

I have seen no announcements to indicate that Google will provide a general web search API in the future. That said, I do not work for Google, and I don't claim for an instant to have any inside information into what the team is working on or the company is thinking. We will all just have to stay tuned for future announcements.

Jeremy R. Geerdes

Generally Cool Guy

Des Moines, IA

For more information or a project quote:

jrge...@gmail.com

If you're in the Des Moines, IA, area, check out Debra Heights Wesleyan Church!

[yomoore]

Hi @Jeremy Geerdes,

Thanks again for a perfect anwser.

On Mar 8, 7:22 pm, Jeremy Geerdes <jrgeer...@gmail.com> wrote:
> I should have explained that a little better. The AJAX Search API employs dynamic throttling to try and detect abuse of the system and block it. So if you were to send a massive number of requests in rapid succession, it would cut you off for suspected TOS violation. The throttles have been tightened since the API was deprecated, but there is still no hard limit on the number of requests you can send.
>
> That said, the API's TOS very clearly prohibit the use of automated requests. So you can't set up an application to simply crawl Google's results or even really to poll for site ranking, etc.
>
> Jeremy R. Geerdes
> Generally Cool Guy
> Des Moines, IA
>
> For more information or a project quote:

> jrgeer...@gmail.com
>
> If you're in the Des Moines, IA, area, check out Debra Heights Wesleyan Church!
>

> On Mar 8, 2011, at 12:18 PM, yomoore wrote:
>
>
>
>
>
>
>
> > Hi @Jeremy Geerdes,
>
> > Thank you so much for your answer.
> > I'm still not convinced about the unlimited queries. Because on this
> > link:
>
> >http://code.google.com/apis/websearch/docs/
>
> > it clearly states: "It will continue to work as per our deprecation
> > policy, but the number of requests you may make per day will be
> > limited."
>
> > Do you perhaps have any information about this ''limit' and what it
> > means for a webdeveloper?
>
> > On Mar 8, 1:53 pm, Jeremy Geerdes <jrgeer...@gmail.com> wrote:

> >> On the internet now, you can find documentation for three DIFFERENT Google search APIs. The first API that they released used an approach called SOAP which returned results in an XML format. This is what the documentation you were seeing athttp://code.google.com/apis/soapsearch/api_faq.html#gen12isreferring to. The SOAP Search API would return up to 1,000 results for a search term, but it also had a limit of 1,000 queries per day, but it was discontinued in 2008 or 2009.

>
> >> The second rendition of a Google search API was originally called the AJAX Search API. It did not have a hard limit on the number of requests you could send per day, but it was throttled and restricted in other ways to discourage SEO applications from using the service (e.g., it would only return up to 64 results for a normal query, 100 for Custom Search Engine query). This is what you were seeing demonstrated athttp://code.google.com/apis/ajax/playground/#the_hello_world_of_gener.... This API has been expanded for use beyond AJAX, so it became called simply the WebSearch API, and you can find documentation athttp://code.google.com/apis/websearch/docs/. You will notice at the top of that page, though, that this API has been deprecated since November 2010. This means that, while it still works and Google will still maintain it, it is scheduled for shutdown no later than November 2013.
>

> >> The WebSearch API is being succeeded by the Custom Search API, which is really designed to interface a Google Custom Search Engine (seehttp://www.google.com/cseformore information) rather than simply search the general web. There are advantages and disadvantages to this API. Two big advantages are that it will return up to 100 results, and those results will generally be pretty good because CSE's are designed to be "curated" - that is, you set it up, pick the sites you want to search, and so you don't get a lot of garbage results. But there are also a couple of big disadvantages to it. For instance, the CS API will not search the general web; it must be coupled with a CSE, effectively rendering it useless for SEO applications and general search functionality. And the big disadvantage is that it is actually limited to only 100 requests per day. You can request more quota than that via the APIs Console, but the criteria on which the increased quota is granted - and the timeframe in which it will be processed - are absolutely unclear. You can sign up for the CS API and find the documentation for it and a number of other Google APIs in the API Console athttp://code.google.com/apis/console

[omr]

Jeremy Geerdes wrote:
> "... the CS API will not search the general web ..."

Actually, a CSE can be set to search the web.
(This option has been available since 2006.)
Example:
http://www.google.com/cse/home?cx=006546854575108700448:t98uaijhwfu

-- omr

[Jeremy Geerdes]

Actually, if you're setting up a new CSE, the form requires you to enter one or more sites to search and does not offer the option that it used to to search the entire web. I thought this was the case. I just confirmed it. Further, while you can set up a CSE with a site (e.g., Google) and then use the control panel to remove that site, the CSE still won't search the web in general.

My guess is that the reference link omr provided is grandfathered in.

Jeremy R. Geerdes
Generally Cool Guy
Des Moines, IA

For more information or a project quote:
jrge...@gmail.com

If you're in the Des Moines, IA, area, check out Debra Heights Wesleyan Church!

[dalila benachenhou]

Omr,

Thank you for the clarification.

Dalila

[omr]

> "... While you can set up a CSE [to search a specified] site
> ... and then use the control panel to remove that site,
> the CSE still won't search the web in general." [?]

It will, if you select the option I mentioned:
On the "Basics" page of the CSE control panel, scroll down to find:

"Preferences":
"How to search included sites":
"Search the entire web but emphasize included sites."

That option sets the CSE to search the web
(even if you delete any specific "included sites").

[Technical note (for completeness):
The "entire web" option changes the CSE's background inclusion "label"
to "BOOST" mode (instead of the default "FILTER" mode). That label's
definition can be found in the CSE XML specification ("context file"),
which can be downloaded via the Advanced page of the control panel.]

The CSE documentation doesn't go into much detail about the "entire
web" search option. Currently the "Getting Started" chapter of the
Developer Guide says:

> "... You can restrict the search to sites [...] that you select, or you can create a junior Google that emphasizes certain sites. ..."

Note that last point: "... You can create a junior Google that
emphasizes certain sites. ..." Clearly, this refers to the "entire
web" option.

But as we've seen, that option apparently works even if you delete
your "included sites" -- in this case the CSE searches the web
*without* any *specified* site 'bias'. In this case, the search
results obtained may be roughly comparable to (though not exactly the
same as) regular web search. ... (Apparently the documentation does
not further discuss this?)

(These are just my observations. I'm not an expert. If anything more
needs to be said here about the use of the "entire web" CSE search
option, then I hope someone from Google can comment. Please also note
my Disclaimer below.)

.

Jeremy Geerdes wrote:
> "... My guess is that the reference link omr provided is grandfathered in."

No. I created that CSE today.

http://www.google.com/cse/home?cx=006546854575108700448:t98uaijhwfu

(BTW, an apology: That CSE was temporarily broken between about 5:40
and 5:45 PM Pacific time today. Sorry. It's fixed and working again
now.)

(That CSE will be deleted soon. It's just a temporary demo for the
purposes of this discussion.)

.

DISCLAIMER:

The foregoing information is posted as-is, with no warranty. I'm not
liable for your use of this information. As always it is your
responsibility to comply with all requirements and terms that apply to
you.

-- omr

[Adam Feldman]

Hi folks,

I know there's been a bit of confusion on this.  It *is* possible to create a CSE that will search the whole web, but it's a little tricky.  We've put together a help center article to describe the process (this is why I didn't respond sooner):

http://www.google.com/support/customsearch/bin/answer.py?hl=en&answer=1210656

I hope this helps - please let us know if you have any further questions.

Regarding the limit of 100 queries per day for the Custom Search API (http://code.google.com/apis/customsearch/v1/overview.html):  Instead of having to purchase Google Site Search for more quota, it's now possible to now "Enable billing" right in the APIs Console (http://code.google.com/apis/console) when you need more quota.  The pricing is in line with Google Site Search, but we hope the purchasing flow is a bit more convenient.  More info on this can be found here:

http://code.google.com/apis/console-help/#Billing

Cheers,

Adam

-- omr

[omr]

Thanks Adam for pointing out that documentation.

BTW, I forgot to mention another known way to obtain "web search"
results from a CSE -- by prefixing an exclamation mark on the CX
value.

For example with the CS API we specify a CX value in the request: *

...cx=012345678901234567890:abcdefghijk...

"Web search" results can be obtained by adding a '!' prefix:

...cx=!012345678901234567890:abcdefghijk...

... or with '!' encoded as %21

...cx=%21012345678901234567890:abcdefghijk...

* (Note: The CX value shown here is obviously a fake example; a real
and valid CX value must be used.)

I don't recall finding any official documentation on that feature --
but IIRC it was first mentioned in early 2007 by a user in an old CS
discussion group. (Users were able to discover the feature because
the '!' CX prefix was visible in URLs produced by the "Web Search"
radio-button option provided on Google-hosted CSE results pages.)

If that feature is supported for general use, but not yet documented
-- would Google consider documenting it?

Thanks

-- omr