Best Practices for Search API?

[Emlyn]

I've just noticed that the old 10GB limit on search indices can be raised on request to 200GB max, on an index by index basis.

I've got a use case where I'd ideally like search indices to be unbounded, but 200GB will go a long way. Is there any penalty associated with this, though? Do the indices perform worse as they get larger? Does anyone know why there is a limit at all?

--

Emlyn

http://point7.wordpress.com - My blog
https://plus.google.com/u/0/100281903174934656260 - Google+

[Jeff Schnitzer]

The search index is incredibly efficient. I had some data I was indexing in the datastore, and the index was consuming 100GB+. When I moved it to the Search API, the index consumed a few GB. Afterwards I felt silly for asking for the quota raise in advance.

YMMV, of course.

Not a direct answer to your question but something to keep in mind. I was surprised by how small the index is.

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAMp1VPBcfZZUT14s3v7mxCoPy33Yryv3T2OaxPg_kvX4-aUm3w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Nickolas Daskalou]

Jeff, were you storing docs in the Search API somewhat intelligently, e.g. using short field names, "0" instead of "false" for boolean values etc., or you just sent documents to the Search API without much thought and it was still efficiently storing them?

Nick

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_fsAVXTSjQt1qYGNMCJQ%40mail.gmail.com.

[Emlyn]

I've noticed the same thing Jeff, it's really brilliant. But what I'd like to put into it is a pretty dense time series of real time events; it's really difficult to put a sensible upper bound on how big that might get.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uhbRNHBDpj5%2BspLM0rp_-o5Ga_fsAVXTSjQt1qYGNMCJQ%40mail.gmail.com.

[Jeff Schnitzer]

No not at all; quite the opposite, really. I was storing word fragments for typeahead, eg “foobar” became [“f”, “fo”, “foo”, “foob”, “fooba”, “foobar”]. It was very expensive to index in the datastore and very cheap to index in the search api.

The search API is pretty cool. I hope Google makes an official service out of it someday.

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAOj3zuDkQLAE%2BV5H1WWXxu4hj-MWNpddNiShC7R8YkWE5fug9Q%40mail.gmail.com.

[Emlyn]

"The search API is pretty cool. I hope Google makes an official service out of it someday."

I second this! I live in fear of deprecation. How could we possibly replace it?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CADK-0uhhw4f%2BOwpJ%2Bws2S0ngLeJFWi-Li%3D%3DuOLd0hRODsUA7PQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--