OVER_QUERY_LIMIT For Elevation Service

[JavaJive]

I have written a page that finds the best local TV transmitters (this
uses Google Maps v2, I am currently trying to adapt it to v3) ...
http://www.macfh.co.uk/JavaJive/AudioVisualTV/TerrestrialTV/TerrestrialCalculator.html

It works as follows
1: Create a list of all transmitters within 100km sorted by distance.
2: Choose up to the five nearest relays (sub-transmitters) and the
five nearest main transmitters.
3: For each look up the transmission powers from my server and
attenuate them for distance.
4: Discard any whose signals are too weak to reach the site.
5: For the remaining, get the path elevation.
6: Attenuate the signal powers further according to the terrain.
7: List any remaining above the minimum required level.

The currently published version uses a UK-only private elevation
service, but I'm trying to get the v3 compatible version to use the
Google Elevation Service, using getElevationAlongPath().

As can be seen from the above procedure, the absolute maximum number
of paths that will be requested in quick succession is 10, though
usually it will be about half that. Nevertheless, despite this small
number, I'm getting OVER_QUERY_LIMIT responses.

I wasn't able to find any documentation on the actual limits to using
this service. What exactly are they, and what can be done to overcome
this problem?

[Rossko]

> I wasn't able to find any documentation on the actual limits to using
> this service.

We can hazard a guess that they are similar to geocoding limits.
Another guess would be that you have broken the rate limit, rather
than max counts.
http://groups.google.com/group/google-maps-js-api-v3/browse_thread/thread/a9bab8ca2e4d63d/b3f01f010f07f558

[Nathan Raley]

Taken from google's documentation on it at: http://code.google.com/apis/maps/documentation/elevation/

Use of the Google Elevation API is subject to a query limit of 25,000 elevation location requests per day. Note that on any given query, you may request elevation data for multiple locations, which count as separate queries. Up to 512 locations may be queried within a single request. This limit is enforced to prevent abuse and/or repurposing of the Elevation API, and this limit may be changed in the future without notice. Additionally, we enforce a request rate limit to prevent abuse of the service. If you exceed the 24-hour limit or otherwise abuse the service, the Elevation API may stop working for you temporarily. If you continue to exceed this limit, your access to the Elevation API may be blocked.

The Elevation API returns data for single point queries of the highest accuracy possible. Batch queries involving multiple locations may return data with less accuracy, especially if the locations are spread apart, as some smoothing of data occurs.

Note: the Elevation API may only be used in conjunction with displaying results on a Google map; using elevation data without displaying a map for which elevation data was requested is prohibited. For complete details on allowed usage, consult the Maps API Terms of Service License Restrictions.

Then the post Rossko posted gives you more detailed explanations about the usage limiations.

--
You received this message because you are subscribed to the Google Groups "Google Maps JavaScript API v3" group.
To post to this group, send email to google-map...@googlegroups.com.
To unsubscribe from this group, send email to google-maps-js-a...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-maps-js-api-v3?hl=en.

[JavaJive]

To judge from that, I certainly haven't exceeded the number of queries
per day (2,500) and am unlikely while testing to exceed the maximum
number of points per day (25,000), so it looks like the rate, but that
does not appear to be documented anywhere. What can I do about this?

On Jul 2, 2:55 pm, Rossko <ros...@culzean.clara.co.uk> wrote:
> > I wasn't able to find any documentation on the actual limits to using
> > this service.
>
> We can hazard a guess that they are similar to geocoding limits.
> Another guess would be that you have broken the rate limit, rather
> than max counts.

> http://groups.google.com/group/google-maps-js-api-v3/browse_thread/th...

[Nathan Raley]

I never saw mention of what the specific rate is; however, they do mention that there is one and that they have prevention mechanisms to keep people from scapping their databases.  The person in the post that Rossko used was rate limiting his requests to limit how fast he was tossing out requests.  You could easily implement something along these lines with some clever timer usage.  If you need some help with timers let me know.

[JavaJive]

Thanks, but timers are no problem.

However, I don't want to go blundering around in the dark trying to
second guess Google's limits, and maybe exceeding them in obscure
situations. I'd rather have something that I can guarantee to stay
within limits wihich are known and understood by both parties.

[Nathan Raley]

Best thing I can suggest is to read more into what Ben was suggesting with checking the data that you are requesting and everything, and then throttling your requests down to a level that appears to be acceptable by Google.

[Rossko]

> I'd rather have something that I can guarantee to stay
> within limits wihich are known and understood by both parties.

The classic example of smart throttling code -
http://econym.org.uk/gmap/geomulti.htm
this is geocoding for v2, but the principles are equally applicable
here.

[JavaJive]

Thanks both for your help.

As far as I can see, what I'm trying to do is not against Google's
T&C, and in normal use is not likely to occur more than about 25 - 50
times a day, far less currently. I don't see why there should be a
difficulty with it.

In testing from my home postcode, 7 queries are sent to the Elevation
Service, the last 2 of which were being rejected. Therefore,
suspecting the maximum number that could be outstanding at any one
time was 5, I created a counter to wait for the number pending to fall
below 5 before sending more, but the two that had to wait were still
rejected with the OVER_QUERY_LIMIT response.

This is what I mean by floundering around in the dark.

Can Google please give us some precise T&C with numerical usage limits
and rates for the successful use of this service?

[Nathan Raley]

Well, Ben had pointed out that there are means in which to protect against bad requests being sent over and over requesting the same data.  That could also be a possible problem, that was why I recommended reading more into the details of what Ben and them were trying to discuss in the original source that Rossko had sent you.  It could be something wrong in the request you are sending or you could be sending too many at once and had reached the counter.

That post that Rossko had linked in the later message regarding the throttling provides a perfect example of how you can throttle your requests and I would highly recommend trying to implement something along those lines to help you trouble shoot this even more.

[William]

On Jul 3, 3:01 am, JavaJive <c.e.m.harri...@macfh.co.uk> wrote:

> In testing from my home postcode, 7 queries are sent to the Elevation
> Service, the last 2 of which were being rejected.  

how many locations were in each of the queries? I guess each query
was a straight line path, defined by start point, end point, and
number of samples?

[JavaJive]

On Jul 2, 6:14 pm, Nathan Raley <nlrale...@gmail.com> wrote:
> Well, Ben had pointed out that there are means in which to protect against
> bad requests being sent over and over requesting the same data.

Requests made individually using the same initial data by the same
function work for both transmitters the requests of which are being
rejected in batch mode, so there is nothing wrong with the requests
that are being sent.

> you could be sending too many at once and had reached the
> counter.

I think we've established that this is the case. However, unitl I
know the exact parameters behind the rejections, I can't circumvent
them with any assurance of my app always working, or as nearly so as
one can ever get on the web, while remaining within Google's T&C.

> That post that Rossko had linked in the later message regarding the
> throttling provides a perfect example of how you can throttle your requests
> and I would highly recommend trying to implement something along those lines
> to help you trouble shoot this even more.

I would have thought that waiting for the earlier results to return,
as the above test did, would have been the best possible way of doing
it, because that prevents the server(s) being overloaded, however fast
or slow it/they and the internet are working. But apparently not.

[Rossko]

> Can Google please give us some precise T&C with numerical usage limits
> and rates for the successful use of this service?

It's likely to vary from time to time, by server load.
Just implement the dynamic throttle handling and stop worrying about
it.

[JavaJive]

Each query is from a UK transmitter location to the same location set
by the user, wherever in the UK that happens to be. I divide the
angular path length by the angular resolution of the SRTM data, and
use the minimum between that figure or 190, which is the maximum that
I can fit into a URL to the Google Chart API to display the path
profile. In practice that means that nearly all queries will be for
190 samples. To achieve a reasonable number of daily hits in the
eventual live app, I may cut that down if I have to, but that's what
it's doing at the moment during testing.

[Nathan Raley]

Rossko pointed you to the site:

http://econym.org.uk/gmap/geomulti.htm

In this post, if you view the example, he provides you with a means of which to throttle your requests, complete the entire query, and have it working 100% of the time no matter how long the responses take.  I suggest reading into the example he provided and taking a look into that b/c that is exactly what you are looking for.

[JavaJive]

I've read that, and it takes much longer than a mere 100ms for the
requests to return. Therefore, if implementing that delay between
requests works, waiting for the server to return the first batch
before sending any more should be comfortably more than adequate,
shouldn't it?

I've tried reducing the number of pending requests to 4, but one still
gets rejected. I dare say 3 might do it. But will it work tomorrow,
let alone next year?

This is why I want hard facts, not suggestions made in the light of no
greater knowledge than I myself have! The links provided have at
least told me the daily quotas, which I couldn't find before, so
thanks for those, but I need to understand *exactly* why I'm getting
this message.

[Nathan Raley]

Actually the example in that post basically waits for a response from the current query before sending the next.  If it gets the error code it slightly increments the delay and then requests that information again until it succeeds.  Thus if you implement the example in that cited source yours should be working fine and you won't have to worry about it not working tomorrow or in the future.

[Rossko]

> This is why I want hard facts

There probably aren't any hard facts available.
In the past, Google have been evasive/flexible (depending on your
degree of paranoia) about rate issues on the other rate-limited
services.
There is no reason for them to publish their anti-abuse
implementation in detail ; there is no reason why the parameters it
operates under should stay constant from minute to minute.
There are recognised methods to deal with the situation, and
throttle request rates to good advantage. They've been successfully
used for years.

Deal with it, or ask for your money back.

[William]

On Jul 3, 4:25 am, JavaJive <c.e.m.harri...@macfh.co.uk> wrote:
>
> In practice that means that nearly all queries will be for
> 190 samples.  
>

I'll do some testing to see whether there is a limit of 512 concurrent
location requests, in addition to the limit of 512 locations per
request, which would suggest that 3 simultaneous requests of 190=570
locations might trigger the rate limit for subsequent requests.

[William]

the limit appears to be no more than 1024 concurrent location
requests.

for example:

with number of samples=128, the limit is 8 paths
http://www.william-map.com/20100703/1/128.htm

with number of samples=190, the limit is 5 paths
http://www.william-map.com/20100703/1/190.htm

with number of samples=512, the limit is 2 paths
http://www.william-map.com/20100703/1/512.htm

so in general if number of samples is n, then limit = floor(1024/n)

samples, limit (<=1024), next (>1024)
125, 8=1000, 9=1125
128, 8=1024, 9=1152
145, 7=1015, 8=1160
150, 6=900, 7=1050
190, 5=950, 6=1140
200, 5=1000, 6=1200
250, 4=1000, 5=1250
256, 4=1024, 5=1280
300, 3=900, 4=1200
341, 3=1023, 4=1364
342, 2=684, 3=1026
350, 2=700, 3=1050
400, 2=800, 3=1200
450, 2=900, 3=1350
500, 2=1000, 3=1500
512, 2=1024, 3=1536

...

[JavaJive]

That's a neat investigation, which appears to make sense, but
unfortunately I have also been doing some investigating, and I'm not
sure our results are mutually compatible!

I think I'm going to have to explain the way the data is fetched ...

There is a routine which I shall here GetCheck(stage), not its real
name, but perhaps more descriptive. It initialises data for other
stages (stage = 0), fetches and check transmission tables (stage = 1),
and fetches and checks terrain (stage = 2), obviously the latter being
the stage we're interested in. If GetCheck(stage) finds that data for
the current stage is still outstanding, the last thing it does is to
check that a timeout value has not been exceeded, and if not set a
500ms timer to call itself again.

The process is kick-started by the user choosing the relevant option.
A list of the nearest transmitters is created, and GetCheck(0) and
GetCheck(1) are called. In due course, all the transmission data is
returned, or so we hope, the signals are analysed, and GetCheck(0) and
GetCheck(2) are called for the remaining transmitters, causing, on the
current live page, all the terrain requests to be fired off together.
There are usually between about 2-3 and 7-8.

To fetch from the Google Elevation Service. I have modified
GetCheck(2) to fire up to GESLimit requests together, and only when
the number of outstanding requests falls below this level, through
data being returned, will it fire off more, always keeping the number
of outstanding requests at or below GESLimit.

So by modifying GESLimit I can easily experiment to see what works.

The answer is that only GESLimit = 1 works. 2-4 produces
OVER_QUERY_LIMIT for a single transmitter, 5 errors for two
transmitters. There didn't seem much point in going any higher.

I like, and thank you for, what you've done, and it seems to make
sense, but how to reconcile it with the above I have absolutely no
idea!

This is why I think Google should explain themselves. We could p*ss
around a long time deriving such results, thinking that we've
understood everything, only to find that when someone else tries
something different, the results are incompatible with our own. Such
a situation doesn't lend confidence that over the widely differing
conditions of the UK TV Transmitter network, my page will always work.

On Jul 3, 8:18 am, William <william.g...@gmail.com> wrote:
> the limit appears to be no more than 1024 concurrent location
> requests.
>
> for example:
>

> with number of samples=128, the limit is 8 pathshttp://www.william-map.com/20100703/1/128.htm
>
> with number of samples=190, the limit is 5 pathshttp://www.william-map.com/20100703/1/190.htm
>
> with number of samples=512, the limit is 2 pathshttp://www.william-map.com/20100703/1/512.htm

[Rossko]

> Such
> a situation doesn't lend confidence that over the widely differing
> conditions of the UK TV Transmitter network, my page will always work.

If that confidence were truly what you wanted, you only have to keep
to the published 512-per-request (no problem, you've selected ~190 for
other reasons) and implement a dynamic throttle. That will deal alike
with whatever fixed, dynamic or periodically variable rate limitation
strategies that Google might use today - or tomorrow. (Even if they
were compelled to tell us today's strategy, it still may not apply
tomorrow, which is what you seem to be worrying about.)

The geocoding example of the throttling technique is just that, an
example, and given the greater latency of path elevation requests, it
would certainly be smart to alter the size of the delay chunks just as
you suggested. Maybe have an initial 500mS and bump in 250mS steps.

From your description of your code, it seems all you need to actually
add is a retry-after-delay upon receiving a OVER_QUERY_LIMIT, and bump
the delay.

[JavaJive]

Correction, I missed an error for GESLimit = 4. So the results were:

1: Good
2-3: 1 x OVER_QUERY_LIMIT error
4-5: 2 x OVER_QUERY_LIMIT errors

In all cases, as one would expect, the errors came on the request(s)
sent last.

[William]

On Jul 4, 8:38 am, JavaJive <c.e.m.harri...@macfh.co.uk> wrote:
> Correction, I missed an error for GESLimit = 4.  So the results were:
>
> 1:      Good
> 2-3:    1 x OVER_QUERY_LIMIT error
> 4-5:    2 x OVER_QUERY_LIMIT errors
>
> In all cases, as one would expect, the errors came on the request(s)
> sent last.

that's interesting. I'll see if I can replicate these results.

One thing I've noticed using Firebug is that the OVER_QUERY_LIMIT
seems to be implemented on the client side, because the requests when
over the limit aren't sent to the server and the OVER_QUERY_LIMIT
error is very quick (50ms - 100 ms). Also caching is being used so
it's important to clear the cache when repeating these kind of tests.

[William]

the issue here is whether google's implementation is working correctly
or not. There might be a bug in their implementation which is causing
the problems that JavaJive is reporting in this thread.

The mechanism for this service is implemented in javascript, and I can
see the parameters, and how the mechanism works, but I don't yet
understand how it is supposed to reset the throttle once 1024
locations have been processed. There is a reset function but I can't
yet see where it is called.

[William]

On Jul 3, 3:01 am, JavaJive <c.e.m.harri...@macfh.co.uk> wrote:
>

> Can Google please give us some precise T&C with numerical usage limits
> and rates for the successful use of this service?
>

from looking at the source it appears the limit is 1024 locations,
beyond which the rate is 128 additional locations per second.

[Rossko]

> the issue here is whether google's implementation is working correctly
> or not.  There might be a bug in their implementation which is causing
> the problems that JavaJive is reporting in this thread.

Fair enough, I missed something ; the only problem I'd seen was the
limiting.

[JavaJive]

Yes. After I finished posting last night, I briefly tried the same
test in Opera v9.64, and was surprised to find that the results were
different! There, GESLimit = 5 worked fine!

Consequently, having known of Firebug changing behaviour before, I
tried disabling it, using my own Debug.js which simply adds a textarea
to receive console.log messages, but the behaviour in FF3 was just the
same, 2 errors for GESLimit = 5.

I shall obviously have to waste time more precious time investigating
this even more closely :-(

[William]

On Jul 4, 9:41 pm, JavaJive <c.e.m.harri...@macfh.co.uk> wrote:
> Yes.  After I finished posting last night, I briefly tried the same
> test in Opera v9.64, and was surprised to find that the results were
> different!  There, GESLimit = 5 worked fine!
>
> Consequently, having known of Firebug changing behaviour before, I
> tried disabling it, using my own Debug.js which simply adds a textarea
> to receive console.log messages, but the behaviour in FF3 was just the
> same, 2 errors for GESLimit = 5.
>

Regarding Firebug, I found there was a problem with timer accuracy on
Firefox using window.setTimeout.

Also there's a problem with timer resolution on Windows, with around
15 ms between ticks.

Taking the parameters of 1024 locations, with additional locations at
128 per second, means the following delay between requests:

L = total locations (eg 190x10=1900)
N = number of paths (eg 10)

var interval = Math.ceil(1000*(S - 1024) / (128*(N-1)));

I added 50ms to overcome any rounding issues with the timer
resolution.

see the following example:
http://www.william-map.com/20100703/1/rate.htm

...

[JavaJive]

Apologies for not replying sooner, it's not that I don't appreciate
your input, far from it, but I was temporarily required to give all my
attention to something else ...

On Jul 5, 2:34 am, William <william.g...@gmail.com> wrote:
> On Jul 4, 9:41 pm, JavaJive <c.e.m.harri...@macfh.co.uk> wrote:
>
> > Yes. After I finished posting last night, I briefly tried the same
> > test in Opera v9.64, and was surprised to find that the results were
> > different! There, GESLimit = 5 worked fine!
>
> > Consequently, having known of Firebug changing behaviour before, I
> > tried disabling it, using my own Debug.js which simply adds a textarea
> > to receive console.log messages, but the behaviour in FF3 was just the
> > same, 2 errors for GESLimit = 5.
>
> Regarding Firebug, I found there was a problem with timer accuracy on
> Firefox using window.setTimeout.

Isn't that just another manifestation of the following, compounded by
the extra overhead of the processing that a debugger such as FB must
do?

> Also there's a problem with timer resolution on Windows, with around
> 15 ms between ticks.

Yes, by default the hardware timer only fires at approximately that
sort of interval, but that is easily sufficient resolution here.

> Taking the parameters of 1024 locations, with additional locations at
> 128 per second, means the following delay between requests:
>
> L = total locations (eg 190x10=1900)
> N = number of paths (eg 10)
>
> var interval = Math.ceil(1000*(S - 1024) / (128*(N-1)));
>
> I added 50ms to overcome any rounding issues with the timer
> resolution.
>
> see the following example:http://www.william-map.com/20100703/1/rate.htm

Thanks for your work and an interesting demo, which in time may well
be useful, but I've not got that far yet. I'm still struggling to
produce a system that recovers gracefully in different browsers if
errors are generated.

As previously explained, my system fetches up to GESLimit profiles at
once. For the results below where I am trying to create a reliable
system of recovery if an error is caused, GESLimit = 5, which
according to your calculations, should be small enough to avoid the
OVER_QUERY_LIMIT error. Sometimes it does, sometimes it does not,
sometimes it generates another error entirely ...

To take the example of my home postcode already mentioned, the terrain
profiles for 7 transmitters must be fetched. In order of likely
signal strength after stage 1, and therefore of fetching, they are:

0: Hannington
1: Crystal Palace
2: Oxford
3: Rowridge
4: Midhurst
5: Hemdean
6: Henley-On-Thames

The profiles are requested as follows ...

console.log( req.index + ": " + (req.userObject ?
req.userObject.name : "") + ", [" + LLs[0] + "," + LLs[1] + "]" + ",
" + samples );
GoogleES.getElevationAlongPath( {path:LLs, samples:samples}, new
Function("path","status", "reqTerrainCB("+ index +
",path,status);") );

... where index is the position in the queued requests array of the
particular request object whose results are being returned, which in
turn contains a reference to the transmitter to which it belongs, and
where ...

function reqTerrainCB( index, path, status )
{
console.log( "getElevationAlongPath(): " + index + ", " +
(__rqs[index] && __rqs[index].userObject ?
__rqs[index].userObject.name : "undefined") + ", " + status );
if( __rqs[index] )
switch( status )
{
case "OK": __rqs[index].wrapCallBack( path );
break;
default: __rqs[index].destroy();
break;
}
}

As you can see, if the status returned is not OK, then the request
destroys itself, and this in turn causes a new request for that
transmitter to be sent to the Elevation Service, next time the
outstanding results are checked.

The results for Opera 9.64 are the most bloody-minded ...
0: Hannington, <parameter dump>
1: Crystal Palace, <parameter dump>
2: Oxford, <parameter dump>
3: Rowridge, <parameter dump>
4: Midhurst, <parameter dump>
getElevationAlongPath(): 0, Hannington, OK
getElevationAlongPath(): 1, Crystal Palace, OK
getElevationAlongPath(): 2, Oxford, OK
getElevationAlongPath(): 3, Rowridge, OK
5: Hemdean, <parameter dump>
6: Henley-On-Thames, <parameter dump>
getElevationAlongPath(): 4, Midhurst, UNKNOWN_ERROR
0: Midhurst, <parameter dump>
getElevationAlongPath(): 5, Hemdean, OK
getElevationAlongPath(): 6, Henley-On-Thames, OK
getElevationAlongPath(): 4, undefined, OK
getElevationAlongPath(): 0, Midhurst, UNKNOWN_ERROR
0: Midhurst, <parameter dump>
getElevationAlongPath(): 0, Midhurst, OK
getElevationAlongPath(): 0, undefined, UNKNOWN_ERROR

As you can see, the requests are all sent to the server, but sometimes
the last one or two returns UNKNOWN_ERROR. This seems to be more
likely if the requests are sent immediately after the Google code has
loaded. On the test page I use, the relevant option is only enabled,
via the code load callback, once Google Maps API v3 has loaded. The
sooner I click 'Find likeliest' after it has been enabled, the less
likely it seems to be to complete successfully.

If UNKNOWN_ERROR does happen, temporary chaos takes over, in that:
A: A subsequent re-sending of the request is almost guaranteed to
fail with the same error
B: Request(s) that initially return UNKNOWN_ERROR later appear to
return OK, but by then the record of which transmitter the results
refer to have been lost when the request object destroyed itself after
the initial error.

So what happens if we introduce another case statement in the
handler ...
case "UNKNOWN_ERROR": break;
... so that the system waits for the ultimate return of the errored
results with OK? Surely that will save unnecessary requests being
resent after the first error, and will be more efficient? Not so,
although in the results below this does indeed happen with Midhurst,
sometimes, as with Rowridge, the OK is not returned after the initial
error, and, now that there's no mechanism to re-request that profile
for this particular circumstance, the routine times out waiting for
failed results which never materialise ...

0: Hannington, <parameter dump>
1: Crystal Palace, <parameter dump>
2: Oxford, <parameter dump>
3: Rowridge, <parameter dump>
4: Midhurst, <parameter dump>
getElevationAlongPath(): 0, Hannington, OK
0: Hemdean, <parameter dump>
getElevationAlongPath(): 1, Crystal Palace, OK
1: Henley-On-Thames, <parameter dump>
getElevationAlongPath(): 2, Oxford, OK
getElevationAlongPath(): 3, Rowridge, UNKNOWN_ERROR
getElevationAlongPath(): 4, Midhurst, UNKNOWN_ERROR
getElevationAlongPath(): 4, Midhurst, OK
getElevationAlongPath(): 0, Hemdean, OK
getElevationAlongPath(): 1, Henley-On-Thames, OK

It's a mess. As I have been saying all along, we are floundering
around in the dark here. Google should publish a mechanism that is
guaranteed to work for making successive requests, so that we can just
follow it. We shouldn't be having to second guess or reverse engineer
their code in order to get it to work.

[Rossko]

>         GoogleES.getElevationAlongPath( {path:LLs, samples:samples}, new
> Function("path","status", "reqTerrainCB("+ index +
> ",path,status);") );

I'm not convinced this will always give the result you expect ; I
wonder if it might even be browser dependant. When will the "new
Function(..)" body string get evaluated, what will be the value of
'index' at that time? I don't know enough about it myself, but not
convinced that closure will be obtained. Your Opera results seem to
illustrate that?

[JavaJive]

The new function will be created at the time the call to GES is made,
so that it has a callback to call. The Opera results given show quite
clearly that each call has the expected index value.

[William]

On Jul 13, 1:50 am, JavaJive <c.e.m.harri...@macfh.co.uk> wrote:
> For the results below where I am trying to create a reliable

> system of recovery if an error is caused ...

I agree this is the most important part of the system, to have
assurance it's going to work in all conditions, and so far there's
been OVER_QUERY_LIMIT and UNKNOWN_ERROR, which require different
handling. I will try to replicate your results with the
UNKNOWN_ERROR: