Re-indexing existing data instability

[PK]

This is an issue I hit a while ago when I was actively changing indexes. I had not needed to change my indexes for a long time but yesterday I had to make some changes. Once again, I spent several hours in the following loop:

1. Upload new index definitions

2. Indexing of some new complicated indexes fail after some time

3. Vacuum the failed indexes

4. when vacuum succeeds GoTo 1 if it fails Goto 3

I finally managed to get all of them working after several attempts. As step 4 suggests, even “removing an index" failed at times and had to be retried. I have some complicated indexes but not too much data in the big scheme of things. 

What has been your experience with re-indexing existing data? Is there any tip/workaround to avoid this?

Thanks
PK

p...@gae123.com

PS If Google wants my app’s id to take a look please contact me to send it to you in private

PS2 I have also asked this here: http://stackoverflow.com/questions/34208667/how-to-avoid-several-failed-attempts-when-re-indexing-existing-datastore-data

Relevant issues I suggest you star:

10419: Index Usage Statistics

6133: Improved index management

[Nick (Cloud Platform Support)]

As discussed in the linked StackOverflow thread, overly-complex indexes on overly-complex entities can strain against certain limits. You should read about Datastore Indexes in the documentation for a discussion of the limits and potential pitfalls of complex indexes.

I've never encountered failures when re-indexing or vacuuming indexes, although this can surely happen. In a distributed computing environment, failures of certain operations are certainly possible. It's difficult to tell without looking deeper into it why your index operations failed. You should file a Public Issue Tracker issue if you notice something and are concerned about it. Information which would be useful would be logs from your end, the app id, the Entity kinds affected, and the index file.