datastore queries run locally but fail on GAE due missing indexes

[Mihai B]

I have several queries that run just fine locally but they fail on GAE with error `API error 4 (datastore_v3: NEED_INDEX): no matching index found. recommended index is:`. Is it a known bug/behaviour?  I understand that some queries require indexes but I don't have any index file (manually created or auto-generated by the sdk) locally so I'm expecting the tests to fail if there is any error before deployment.

Mihai.

[Strom]

Would help to see an example query.

[Esse]

I've been there. Actually, I had memory overflow without indexes on one query.

It seems that you really need them for some queries. The deploy occurs without raising errors.

If dev_appserver autogenerate indexes, then it will never raise errors on missing indexes (just thinking, I have not noticed the exact behavior for a long time now). I sometimes have dev_appserver not saving index.yaml. Actually, for a long time, I never saw an autogenerated index. At some update, it has begun adding those indexes, but inside modules. Wrong, wrong, datastore is only one for the project. Well, I always had to write indexes manually.

The catch is, dev_appserver doesn't complain enough about missing indexes. It just doesn't return the items. But the cloud may raise bad errors, or not complain and just crash.

Check the dev_appserver admin for indexes in use. Try to close dev_appserver with ctrl+c and see if it ends "saving indexes". See if the index.yaml file shows up.

If you have modules, you may not have been updating the indexes too.

For me, I always write queries and indexes at the same time, and on deploy, if are new indexes, always run a manual index update.

Hope this sheds some light.

[Mihai B]

 If the tests pass locally but fail in production the local datastore emulation is really close to useless.    If someone decides to fix this it would be great to have the indexes injectable in the context (though the aetest package) so that we can test the packages individually(i.e. not in a monolithic application).

[Ronoaldo José de Lana Pereira]

I usually disable index auto generation when launching dev_appserver.py. This helps me keep dry with indexes that I really need during development.

The documented behavior is that when you launch devserver it can generate indexes for you. If you have an empty index.yaml without the comments that are generated, it will not manage indexes for you. I like to have all of them explicitly appear in my files.

--require_indexes=yes|no

Disables automatic generation of entries in the index.yaml file. Instead, when the application makes a query that requires that its index be defined in the file and the index definition is not found, an exception will be raised, similar to what would happen when running on App Engine. The default value isno.

As explained here: https://cloud.google.com/appengine/docs/python/tools/local-devserver-command

The local unit tests are another think. They assume you are running a temporary webserver instance to allow you to write headless unit tests with your code against the API calls, not against a particular app config. There is the value I used this tool for in the past four years.

Best,

On 26 de jun de 2016 12:42 PM, "'Mihai B' via google-appengine-go" <google-ap...@googlegroups.com> wrote:

 If the tests pass locally but fail in production the local datastore emulation is really close to useless.    If someone decides to fix this it would be great to have the indexes injectable in the context (though the aetest package) so that we can test the packages individually(i.e. not in a monolithic application).

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.