Search limit

[Emlyn]

There's a documented rate limitation on the search api (which I am hitting :-( ), of 

• 15,000 Documents added/deleted per minute

https://cloud.google.com/appengine/quotas#search

Is this per index, or for my entire app?

--

Emlyn

https://medium.com/the-infinite-machine - A publication about Google App Engine
sutllang.com - My language sUTL
https://plus.google.com/u/0/100281903174934656260 - Google+

[George (Cloud Platform Support)]

Hello Emlyn, 

The 15000 documents per minute quota is meant for the whole app. You can see in the document you link to that the action is named "Adding documents to Indexes", so generally, not one index or per index. 

In the Java "Quotas" document the corresponding entry is "Maximum documents added or deleted 15,000 per minute". No mentioning of index(es) is made, value being defined per app. Similar statements can be read in the corresponding Python and Go documents. 

[Emlyn]

Thanks George. Not quite the news I wanted, but better to know.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/bec25533-ce04-4ae6-8f3c-d622a546d9bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George (Cloud Platform Support)]

Hi Emlyn, 

News is not entirely bad: "Note: Although these limits are enforced by the minute, the Cloud Platform Console displays the daily totals for each. Customers with Silver, Gold, or Platinum support can request higher throughput limits by contacting their support representative." This is to be read just below the quotas paragraph we speak about. In short, you may consider asking for a quota increase. 

Information on how to reduce your app's needs for paid resources, and so keep within quotas, you may consult the "Managing App Resources" documentation page. 

[Emlyn]

Thanks George, I read that, that's great. I may have to go for a support level in the near future.

For anyone who's interested, I was able to fix my problems with the Search API as follows:

My code enqueues a huge amount of tasks very rapidly, which modify datastore objects. In the _post_put handlers, these also update search indices.

I was having problems that the datastore updates were working, but the rate was too high for the search indices; I was getting timeouts, quota limitations, bad things :-)

I've changed my _post_put handlers to do the updates to search indices in separate tasks, on a queue I've configured just for this. It's rate limited to below the maximum rate for the search api (ie: maximum search throughput is 15000/minute, or 250/s, so I've set it lower than this).

That's completely fixed the problem. And because I'm not hitting quota limits, the real throughput is much higher than it was; I don't see this queue hitting the limit and lagging behind the rest of the job.

Now I get to sleep comfortably at night :-) Thanks for the help.

 

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/623ea57b-1c0c-4454-a854-5cb5fa84bd8b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nickolas Daskalou]

That's a nice way of working around the rate limits Emlyn.

In your _post_put function you can also compare a hash of the last document you put in the Search API index for the Datastore entity, and only send to the Search API if the current document's hash is different.

To make it fast and cost-effective, you can use Memcache to store the hash of the most recent document sent to the Search API for a particular Datastore entity.

This method is most useful if your app does a lot of updates to Datastore entities which would not actually change the Search API document at all (e.g. when updating Datastore entity properties which are not indexed in the Search API).

Nick

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBy7WjB-Qw8N0mkdR0CEP%2BqEf%2Byz%2BxwKCn2GwT1EfGsuA%40mail.gmail.com.

[Emlyn]

I do this is a slightly more costly way Nick; I generate my new document, read any existing document from the index, compare, only put on a change. I like your memcache idea, I could possibly cache the last written version. 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAOj3zuCVz3iR9HDYZdAHHF_LygKGa7HwtZZiKe-Swb6mVpL9sA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.