|Using bulkloader to restore entities created using objectify||Dennis Lo||5/15/11 8:46 AM|
I'm using objectify 3.0 and the bulkloader to backup and restore entities that I've created.
To test I'm using the "Game" kind and outputting it to a csv file called game.csv.
However, when I upload the data and try to retrieve data I get the following error:
Uncaught exception from servlet java.lang.IllegalStateException: Loaded Entity has name but com.example.app.model.Game has no String @Id at com.googlecode.objectify.impl.ConcreteEntityMetadata.setKey(ConcreteEntityMetadata.java:343) at com.googlecode.objectify.impl.ConcreteEntityMetadata.toObject(ConcreteEntityMetadata.java:210) at com.googlecode.objectify.impl.QueryImpl$ToObjectIterator.translate(QueryImpl.java:640) at com.googlecode.objectify.impl.QueryImpl$ToObjectIterator.translate(QueryImpl.java:629) at com.googlecode.objectify.util.TranslatingIterator.next(TranslatingIterator.java:35) at com.googlecode.objectify.impl.QueryImpl.list(QueryImpl.java:470)This is the process I go through:
1. Download the game kind to game.csv using:
|Re: [objectify-appengine] Using bulkloader to restore entities created using objectify||Jeff Schnitzer||5/15/11 11:01 AM|
The error message says that the data in the datastore as a String name
key, but your Game entity has a numeric Long @Id. I don't really know
the syntax for the bulk loader, but the most suspect line is this one:
> export_transform: transform.key_id_or_name_as_string
It looks like you are converting all numeric ids to strings here,
(background info: entities in the datastore can have either a numeric
|Re: [objectify-appengine] Using bulkloader to restore entities created using objectify||Dennis Lo||5/16/11 4:52 PM|
I did some experiements and I believe I have the solution.
I actually got the idea from a stackoverflow post I asked: http://stackoverflow.com/questions/6008929/using-java-google-app-engine-bulkloader-to-download-entire-datastore-to-one-csv-f
The fix is to avoid using the --config_file and bulkloader.yaml.
I used the following to download every kind to a single csv file:
I used the following to upload the single csv file back to the datastore:
They are the same commands but just download_data and upload_data swapped around.
The idea is just let appcfg download and upload all entities (not being kind specific) with using any export or import transformations.
|Re: [objectify-appengine] Using bulkloader to restore entities created using objectify||Dennis Lo||5/16/11 5:00 PM|
Sorry, the last sentence:
"The idea is just let appcfg download and upload all entities (not being kind specific) WITHOUT using any export or import transformations from your bulkloader.yaml config file."
|Re: [objectify-appengine] Using bulkloader to restore entities created using objectify||Jeff Schnitzer||5/16/11 5:55 PM|
That seems safest.
One thing to watch out for, reported here in the past: The bulk
|Re: [objectify-appengine] Using bulkloader to restore entities created using objectify||Dennis Lo||5/16/11 6:11 PM|
I'm unsure what you mean by "same indexed" state?
How do I "re-put()" them all my entities (again, I'm thinking about
|Re: [objectify-appengine] Using bulkloader to restore entities created using objectify||Jeff Schnitzer||5/17/11 10:59 AM|
Every individual property of every single entity has a state of
indexed-or-not. It's the difference, at the low-level api, of calling
Entity.setProperty() or Entity.setUnindexedProperty(). In addition to
whether or not the single-property index is created, this state
defines whether or not the entity participates in a multi-property
index that includes that property.
The problem is that this is bit of state is write-only. There is no
The only solution to this is to define your schema (and its indexing
Another solution is to use Objectify to re-put() every single entity.
Obviously, going through and re-put()ing all your entities may have a
This is a major deficiency in the bulk load/restore process. There
|Re: Using bulkloader to restore entities created using objectify||Riley Eynon-Lynch||5/18/11 7:35 AM|
Regarding unindexed bulkuploader... are you sure that it doesn't
assume that every property SHOULD be indexed? I've used the
bulkuploader to restore ~1M entities (cost 6 cpu hours) and my queries
work fine afterwards.
> On Mon, May 16, 2011 at 6:11 PM, Dennis Lo <lo.den...@gmail.com> wrote:
> > On 5/17/11, Jeff Schnitzer <j...@infohazard.org> wrote:
|Re: [objectify-appengine] Re: Using bulkloader to restore entities created using objectify||Jeff Schnitzer||5/18/11 11:19 AM|
Not sure at all. I just know that one person complained about it on
this list a few months ago.
Assuming every property should be indexed brings up an entirely