bulkloader: No descending index on __key__, performing serial download

[bsb]

I'm struggling a bit with this. I'm trying to bulk dump an entity in
my datastore. I get the following info:

[INFO ] Model_mobileapp: No descending index on __key__, performing
serial download

Now, I added the following index:
- kind: Model_mobileapp
properties:
- name: __key__
direction: desc
- name: appId

The index built fine and is serving. However, I still don't get
parallel fetching. Why is that? Does that have to do with me using
AppEnginePatch for Django, which makes a bit of a mess out of the
datastore? The class to this entity is actually called MobileApp in
module Model (ie. Model.MobileApp). However, doing a fetch on that
entity name works fine:

q = db.GqlQuery("SELECT * FROM Model_mobileapp").fetch(10)

gives results. Any suggestions on how this can be fixed? GAE-style
indexes are still a bit of a mystery to me, I must admit...

TIA,
Ben

[Nick Johnson (Google)]

Hi Ben,

On Sun, Oct 18, 2009 at 10:14 PM, bsb <b...@pearcomp.com> wrote:

I'm struggling a bit with this. I'm trying to bulk dump an entity in
my datastore. I get the following info:

[INFO    ] Model_mobileapp: No descending index on __key__, performing
serial download

Now, I added the following index:
- kind: Model_mobileapp
 properties:
 - name: __key__
   direction: desc
 - name: appId

The problem here is the additional field. The index has to exactly match the query; if the query doesn't have an 'appId' filter or sort, this index cannot be used.

-Nick Johnson

 

The index built fine and is serving. However, I still don't get
parallel fetching. Why is that? Does that have to do with me using
AppEnginePatch for Django, which makes a bit of a mess out of the
datastore? The class to this entity is actually called MobileApp in
module Model (ie. Model.MobileApp). However, doing a fetch on that
entity name works fine:

q = db.GqlQuery("SELECT * FROM Model_mobileapp").fetch(10)

gives results. Any suggestions on how this can be fixed? GAE-style
indexes are still a bit of a mystery to me, I must admit...

TIA,
Ben

--
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047

[bsb]

Ah, thanks for the clarification! This is indeed very different from
e.g. indexing in mysql. I'm used to build composite indexes, assuming
that the first indexed column will be used in queries for that column
only, too.

May I suggest that somewhere down the line, extending the
documentation page for indexes would be really helpful. I believe a
"beginner's guide to GAE indexes" would be a great thing to have: How
are indexes used, defined, uploaded, deleted. The current document has
this info, but some of it is difficult to understand as the BigTable
datastore is so fundamentally different from classical databases. At
least that's my feeling.

Thanks again,
Ben

On Oct 19, 3:02 pm, "Nick Johnson (Google)" <nick.john...@google.com>
wrote: