Thanks George, I read that, that's great. I may have to go for a support level in the near future.
For anyone who's interested, I was able to fix my problems with the Search API as follows:
My code enqueues a huge amount of tasks very rapidly, which modify datastore objects. In the _post_put handlers, these also update search indices.
I was having problems that the datastore updates were working, but the rate was too high for the search indices; I was getting timeouts, quota limitations, bad things :-)
I've changed my _post_put handlers to do the updates to search indices in separate tasks, on a queue I've configured just for this. It's rate limited to below the maximum rate for the search api (ie: maximum search throughput is 15000/minute, or 250/s, so I've set it lower than this).
That's completely fixed the problem. And because I'm not hitting quota limits, the real throughput is much higher than it was; I don't see this queue hitting the limit and lagging behind the rest of the job.
Now I get to sleep comfortably at night :-) Thanks for the help.