Datastore Indexes stuck. vacuum_indexes didn't fix it

[Gwyn Howell]

I have datastore indexes stuck in "Building" status. I have tried deleting my index.yaml and running vacuum_indexes. That deleted all the other indexes, but not the 5 stuck in "Building" status. How do I fix this? (Googlers, my App ID is phoenixhealth-dms!)

Thanks

[Takashi Matsuo]

Hi Gwyn,

Sorry for the inconvenience. We're looking into it.

On Sun, Jun 10, 2012 at 4:46 PM, Gwyn Howell <gwyn....@appogee.co.uk> wrote:

I have datastore indexes stuck in "Building" status. I have tried deleting my index.yaml and running vacuum_indexes. That deleted all the other indexes, but not the 5 stuck in "Building" status. How do I fix this? (Googlers, my App ID is phoenixhealth-dms!)

Thanks

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/aQW5zXh1HIsJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Takashi Matsuo | Developer Advocate | tma...@google.com

[Takashi Matsuo]

Okay, the indexes should be serving now. Please enjoy!

-- Takashi

[Udi h Bauman]

We're having the same problem - indices are stuck "Building" & vacuum didn't help. 

Anything else we can do?

The app id is: apps-poc (HRD).

Thanks,

Udi

[Takashi Matsuo]

Hi Udi,

I'm looking into it.

-- Takashi

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/GP4yuYP_otsJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Takashi Matsuo

[Takashi Matsuo]

Hi Udi,

Now you're good to go.

Sorry for the inconvenience.

-- Takashi

--
Takashi Matsuo

[Takashi Matsuo]

Hi Oleg,

I looked at your app and noticed there were 220 custom indexes on a single kind.

It's way too much. I'm going to bandaid your app, so can you read the following doc for a time being?

https://developers.google.com/appengine/articles/indexselection

-- Takashi

On Tue, Aug 21, 2012 at 4:50 PM, Oleg Bashtenko <o...@cloudwk.com> wrote:

I have the same problem. I've got 224 (!) indices stuck in "Building..." state and unable to do anything to them (vacuum_indexes doesn't help). My app ID is "dev-tcs" (HRD).

Actually, most of those indices were created by mistake, and now I can't do anything (including creation of correct indices) because of "number of indexes" quota.

Thanks in advance,

Oleg

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bkzhxyldeFwJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Takashi Matsuo | Developers Advocate | tma...@google.com

[Prakhil Samar]

Hey i'm also facing the same problem on most of my appengine applications.

i have created a post too  https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/-VU4XPc7TGM 

Please address it..

[Oleg Bashtenko]

Thank you very much for that information!

It seems that your improved Query Planner will GREATLY help me with my indexing problem. I have a situation very similar to the one described in that doc (custom search on any combination of properties), and now it looks like that I'll not have to reject some of my app features just to fit the indexing quota. Same thing about exploding search problem with multivalued properties.

So now I'm going to rewrite my index definitions according to that example to use new query planner capabilities.

What about deleting those nasty 220 indices?

[Oleg Bashtenko]

Got another problem.

There is a "Ticket" kind in my datastore. I want to run a search query on a several fields, e.g.: { active, technicianKey, vip, dueDate } and order results by {techPriorityLevel} field (ascending).

According to article you recommended, it's not necessary for me to define a composite index for all those fields since I already have indexes for all pairs "search field, order field", like

<datastore-index kind="Ticket" ancestor="false" source="manual">
       <property name="vip" direction="asc"/>
       <property name="techPriorityLevel" direction="asc"/>
</datastore-index>

and so on.

However, instead of performing a zigzag merge join, datastore throws a DatastoreNeedIndexException stating that I should explicitly define a composite index like

<datastore-index kind="Ticket" ancestor="false" source="manual">
        <property name="active" direction="asc"/>
        <property name="technicianKey" direction="asc"/>
        <property name="vip" direction="asc"/>
        <property name="dueDate" direction="asc"/>
        <property name="techPriorityLevel" direction="asc"/>
</datastore-index>

Currently experiencing this behavior on development server (AppEngine SDK 1.7.0, Eclipse 3.7.1, using Google plugin for Eclipse).

Is there something I'm missing or doing wrong? Maybe this will work correctly only with real datastore, not local development server? (Still can't check that out because of those 220 stuck indices in datastore.)

Thanks in advance,

Oleg

[Alfred Fuller]

Are you using the low level API? Can you give me the .toString() of the query?

Thanks in advance,

Oleg

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/JjnskSfa_4YJ.

[Oleg Bashtenko]

No, I'm working with objectify-appengine. May it cause such problem? There are only equality filters in my query, and one ordering field. Everything works OK if I define such composite index on all fields used in the query, but the point is to avoid that since there are too many possible combinations of fields (user can select different search filters).

[Jeff Schnitzer]

Objectify more-or-less passes data through to the low-level query API
as-is. If you run .toString() on the Objectify query, you'll get some
text that describes the query - it should be as useful as .toString()
on the LL Query for the purposes of debugging (ie posting here).

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/sGmgOK0R9VoJ.

[Oleg Bashtenko]

Oops. Thank you. Looks like that I forgot to remove one more inequality constraint. This query:

com.googlecode.objectify.impl.QueryImpl{kind=Ticket,ancestor=,filter=activeEQUALtrue,filter=technicianKeyEQUALUser("tech"),filter=vipEQUALtrue,sort=techPriorityLevelASCENDING,limit=10}

works OK, but this one

com.googlecode.objectify.impl.QueryImpl{kind=Ticket,ancestor=,filter=activeEQUALtrue,filter=dueDateLESS_THAN_OR_EQUALTue Aug 21 17:15:44 UTC 2012,filter=dueDateNOT_EQUALnull,filter=technicianKeyEQUALUser("tech"),filter=vipEQUALtrue,sort=dueDateASCENDING,sort=techPriorityLevelASCENDING,limit=10}

fails.

As far as I understand, I have to define additional indices with {dueDate} field since I want to set an inequality filter on it, right? This should double the number of indices I have to define, but still fits the quota.

Thank you for your help. Now the only my problem left is vacuuming those 220 stuck indices in datastore.

[Takashi Matsuo]

Hi Oleg,

Sorry it took long, but I've just put your indexes into 'Error' state.

I think you should be able to vacuum them now.

Enjoy :)

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/dwxb_Sk3DogJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Oleg Bashtenko]

Bingo! That works now.

Thanks guys for your help and useful information. My app is working perfectly now, and I can run advanced search queries on any combination of fields with only 66 indexes on that entity kind (this is acceptable for my purposes).

Good luck to all your team!

Best regards,

Oleg