No descending index on __key__, performing serial download

[Ubaldo Huerta]

While using the bulkloader --dump I get the "No descending index on
__key__, performing serial download" warning of sorts. The dump
occurs, and reasonably "fast". However, I thought I didn't have to
explicitly create this index on __key__, that this index was
implicitly created like an index on any property. I guess I'm wrong?
Should I create this index?

I hesitant to create indexes in production for experimentation because
I've had my share of issues with indexes not building, vacuuming
indexes acting up, etc, in the past.

[Ikai L (Google)]

You'll have to create a descending index. We're tracking some inconsistencies with bulk built custom indexes at the moment, so if you see weirdness (e.g. value appears in ascending order but not descending order) please report it: http://code.google.com/p/googleappengine/issues/detail?id=2481

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Ikai Lan
Developer Programs Engineer, Google App Engine

[Dennis Lo]

Hi Ikai,

I hope you can help me out even though this is such an old thread.

I've been looking around the internet and I can't seem to figure out why my appcfg.py download_data (backup to my csv) is performing so slowly.

I see the "No descending index on __key__, performing serial download" and the dot's indicating progress just keep going. It
is downloading quite slowly. I'm looking at my mac's "Activity Monitor" and it's download it at around 23kb/sec. By the way, I'm from Australia, so maybe that might be one cause.

To give you some context, my GAE app is in Java and the kind I'm downloading has about 25,0000 entries.

I'm also following your blog entry: http://ikaisays.com/2010/06/10/using-the-bulkloader-with-java-app-engine/
the closet but I'm also read this to help me get a better understand of how to edit bulkloader.yaml files: http://code.google.com/appengine/docs/python/tools/uploadingdata.html

My questions are:
1) How to I speed up this backup and eventually the restore process?

2) How do I make a "descending index" on __Key__, I've already got some custom indices inside my datastore-indexes-auto.xml and datastore-indexes.xml but nothing for the key.

In fact I'm not even sure how to make a desc index on the key property inside my datastore-indexes.xml with desc direction. Would it be like (assuming the name of the key is id):

    <datastore-index kind="Game" ancestor="false" source="manual">
        <property name="id" direction="desc"/>
    </datastore-index>

Thanks

[Dennis Lo]

Just an update: The download for this kind with 25,000 entries took 390 seconds (about 6.50minutes).
Is that normal?

I've got 5 more kinds around that size. So essentially, the appcfg.py download_data would take 32.5 minutes to back everything up.

I've also checked the datastore and its just 9% of the 1GB capacity. That's not lot of data, but it just feels so slow the backup for such little data.

Regards,
Dennis