Managing Auto Generated Indexes

[Chris Burnham]

I'm doing an advanced search like the one detailed in https://cloud.google.com/appengine/articles/indexselection

I'm increasing the number of filters from 2 to 4, with 4 sort orders. If I create a perfect index on this search, I'll need 64 indexes in my index.yaml. I followed the advice in the Advanced Search article and created basic indexes for each filter and sort order resulting in 20 indexes. I tested that these indexes satisfy my search by running my development server with dev_appserver.py --require_indexes

If I go back to my regular development workflow and run just the dev_appserver.py, accessing my search page creates the perfect indexes. I don't want to give up the ability to generate indexes for other aspects of development by always running with --require_indexes. I also don't want the dev server to create these extra indexes either. Is there a way to have the server only create new indexes if it would otherwise raise a NeedIndexError?

[Layo]

Hi Chris,

 As announced before community support is moving to Stack Overflow. I will advise you to post this specific question on Stack Overflow. Simply post your questions on Stack Overflow with the tag "google-app-engine", and a member of our support team or a member of our global developer community will respond shortly after. Look here at the current App Engine questions.

Happy coding!

[Alex Martelli]

On Thursday, February 5, 2015 at 11:46:58 PM UTC-8, Layo wrote:

Hi Chris,

 As announced before community support is moving to Stack Overflow. I will advise you to post this specific question on Stack Overflow. Simply post your questions on Stack Overflow with the tag "google-app-engine", and a member of our support team or a member of our global developer community will respond shortly after. Look here at the current App Engine questions

Right, except we can't guaranteed the "shortly" (we try our best but there's no SLA for unpaid support:-) -- and as a side note (not germane to Chris's Q) serverfault is better than stackoverflow if a Q is all about sys admin, not development (I know we've been saying stackoverflow forever -- but we need to fix our mistake and say "stackoverflow for development, serverfault for sysadmin" and having very recently joined cloud tech support from a totally different job at Google I'm striving to make that happen, which is why I miss no opportunity to make myself obnoxious about the issue:-).

Anyway, I did spot Chris's Q on SO and answered it -- unfortunately in the negative, i.e, can't do that right now (plus a little advice on opening a feature request, surviving until it's met, and avoiding the strong, understandable temptation to hack on the [open source after all!-)] SDK to adapt it to Chris's very reasonable desiderata).

Thanks,

Alex

 

[Kaan Soral]

I didn't dive into what exactly you are doing, however I want to teach a method that reduces the number of indexes, while keeping the added indexes the same -> precomputing properties

Instead of running a query for a=1, b=2, c=3 - you can query abc=1|2|3 instead by creating a pre-computed property called abc - you can also query for a=1 by querying abc=1||, or query for a=1 c=2 with abc=1||2

You can adjust the format to suit N properties the way you like

When my indexes get close to the 200 limit, I use this method to reduce the number of indexes while keeping the complexity/cost the same (so basically it's just to tackle the 200 limit)

-----

It's also possible to not add exact indexes for each query, and let the system compute things on query, which is logical for non-frequent queries, as complete indexing is very costly (also as far as I remember, the upper limit for indexes of a single entity was something like 5000)

The simplest way to do this is to use --require_indexes and let the sdk suggest the minimal index, while having a general idea of how indexing/queries work

-----

I've just read your question better, my information seems a bit unrelated for the exact question you asked, yet it's very logical to use --require_indexes and handle the index-logic on development, otherwise you have to digest each index usage after the development is near-completion and do it at once, it's better to do it on-usage/development

-----

Just disregard my answer if you were looking for a more-spot-on sdk feature/answer