search api performance measurements and analysis

[Manfred]

I would like to discuss the performance of the google search api. I've
setup
a simple php-4 proxy, and I'm sending and receiving the soap messages
from there via http directly. After receiving the soap response on the
server, it
will be sent to the client for further processing (xslt). This is not
yet reflected
in the measurements, but may be discussed in a separate thread later
on.

Maybe its only me, but I think a typical response of 3.5 sec for a
query with
10 items returned, or 9 to 15 sec for 30 items returned (remember this
is only
server to server!), seems to be a bit long. So, has anyone experienced
this api with shorter response times, or even worse? Or is Google well
aware of those timings and does consider this (beta) service working as

intended?

To begin with, first question is of course, how can I improve my code
to
boost performance?

Here are the timings (in sec) for 33 successful requests to the api,
each
one returned 10 items (first line, label "request"). So, no retry or
"502"
response go into this. Each request is composed of 3 consecutive
sections of code (labeled here "fsockopen", "firstfgets", and
"otherfgets"),
plus some php code, which contributes to the timings below the msec
only.
min median max %
33 1.986 3.433 9.851 100 request
33 0.113 0.114 0.154 3 fsockopen
33 1.754 3.044 9.62 90 firstfgets
33 0.112 0.115 1.03 7 otherfgets

The main point here: a single instruction, typically taking 3 sec to
complete,
is responsible for 90% of the transaction time!

Code actually used, complete with measurement points:
$etime[] = microtime(); $descr[] = 'request';
$fp = fsockopen('api.google.com', 80, $errno, $errstr, 10);
if (!$fp) {
echo '<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/">',
'<pre>', $errstr ($errno), '</pre></Envelope>';
return true;}
fwrite($fp, $headers);
fwrite($fp, $soap_env);
stream_set_timeout($fp, 20);
$i = 1;
$etime[] = microtime(); $descr[] = 'fsockopen';

$response = fgets($fp);
$etime[] = microtime(); $descr[] = 'firstfgets';

$status = stream_get_meta_data($fp);
while(!feof($fp) && !$status['timed_out']) {
$response .= fgets($fp);
$status = stream_get_meta_data($fp);
$i = $i + 1;}
$etime[] = microtime(); $descr[] = 'otherfgets';

Regards, Manfred

[mcc...@yahoo.com]

I have also experienced extremely slow response times from the API.
There are also numerous 502 errors. It appears that Google is not
giving many resources to their API servers these days (the API is not
making them any money). You may want to use the Yahoo or MSN APIs
which give faster response times.

Regards,
Frank

[Manfred]

mcc...@yahoo.com wrote:
> You may want to use the Yahoo or MSN APIs
> which give faster response times.

Yes, but they don't index filetype .xml, which I need.

Regards, Manfred

[Manfred]

Manfred wrote:
> To begin with, first question is of course, how can I improve my code
> to boost performance?

It turned out, that while using HTTP/1.1, its much faster when the
header "Connection: close" is omitted, even for a single POST!
But be aware, without this header feof will simply time out. So
I replaced feof by reading the response with chunk-size (for SOAP),
and Content-length (for HTML, when encountering the famous 502).

A big Thanks goes to the API folks, for improving the service
dramatically, so both together brought my response time well
_below_ 1 sec.

Regards, Manfred

[cda...@gmail.com]

Manfred, could you elaborate on how you managed to get your response
times to be faster? Did you work with the API team to get this
working?

I use NuSOAP in PHP, and last year and earlier, spelling suggestion
requests from the API server would take an average of 0.25 seconds.
Nowadays, I consider myself lucky if I get a response in under 3
seconds.

[Manfred]

cda...@gmail.com wrote:
> Manfred, could you elaborate on how you managed to get your response
> times to be faster?

Just check for the "Connection: close" header and avoid it. But I don't
know, if NuSOAP allows you to get control at the http level (to use
http/1.1 with a persistent connection).

> Did you work with the API team to get this working?

No, they did their work independently, and it looks good so far!
Let us know, whether your response times have changed now.
Also, did your measurements include retries from 502?

Regards, Manfred

[cda...@gmail.com]

Manfred,

I put together a little test script to gage the speed of the API
service to help me in making changes to the NuSOAP library, and
suddenly the API service is running MUCH faster today!

Have you noticed any change? I'm not getting errors, response times
are generally under 0.2 seconds... it's just like the old days again!

Maybe I'm just getting lucky with my tests...

[Manfred]

Here are some timings again:
fsockopen: 0.15 seconds
first request:
get-head: 0.55
get-msg: 0.11
for subsequent requests:
get-head: 0.39
get-msg: 0.11
each with 10 items and 200 OK response
The api performance appears to be stable over time.
I've not seen a 502 for the last 2 weeks.

Regards, Manfred