Copying data to new app for HR datastore, process seems broken, please help!
28 views
Skip to first unread message
tempy
Jun 15, 2011, 11:21:24 AM6/15/11
to Google App Engine
I'm trying to migrate to the HR datastore and have started the
process, but with only about 15 MB worth of data it has already gone
through 18 hours of CPU and does not seem to be anywhere near
stopping. This doesn't smell right so I stopped it by pausing the
default queue.
The status does not change from "Active (0 steps completed, 12
active)". So, with the default queue paused, I'm not quite sure what
to do. If I start it again it will soon chew through my remaining
quota and bring my app down, but as long as its paused my app isn't
working right anyway. If I purge the queue, are there any steps I
need to take to restart the process? Or maybe its supposed to take
this much time?
AppID: openkleio
process, but with only about 15 MB worth of data it has already gone
through 18 hours of CPU and does not seem to be anywhere near
stopping. This doesn't smell right so I stopped it by pausing the
default queue.
The status does not change from "Active (0 steps completed, 12
active)". So, with the default queue paused, I'm not quite sure what
to do. If I start it again it will soon chew through my remaining
quota and bring my app down, but as long as its paused my app isn't
working right anyway. If I purge the queue, are there any steps I
need to take to restart the process? Or maybe its supposed to take
this much time?
AppID: openkleio
tempy
Jun 15, 2011, 11:55:34 AM6/15/11
to Google App Engine
I don't know if this helps any, but I can see that for every entity
type, the last shard seems stuck.
So, if entity X has 1 shard assigned to it, then the details will say
that shard 0 is running, the last work item is unknown, and 1 second
has elapsed.
If entity Y has 26 shards assigned to it, then 0-24 will report
success and 25 will be running with 1 second elapsed and an unknown
current item. Its like this for every entity kind.
type, the last shard seems stuck.
So, if entity X has 1 shard assigned to it, then the details will say
that shard 0 is running, the last work item is unknown, and 1 second
has elapsed.
If entity Y has 26 shards assigned to it, then 0-24 will report
success and 25 will be running with 1 second elapsed and an unknown
current item. Its like this for every entity kind.
tempy
Jun 15, 2011, 1:47:46 PM6/15/11
to Google App Engine
Word to the wise: The data-transfer utility does not like orphan
objects.
objects.
Robert Kluin
Jun 17, 2011, 5:24:24 AM6/17/11
to google-a...@googlegroups.com
Could you elaborate on this?
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>
tempy
Jun 17, 2011, 10:04:01 AM6/17/11
to Google App Engine
Well, I let it stay in the "stuck" state that I describe above for a
good hour, watching it slowly eat my CPU, then I finally purged my
queue, removed all the tracker objects from the datastore, totally
wiped the datastore in the target app (which had had some fraction of
the data transferred to it), and tried again. Same result. Then I
set about randomly changing things to see if it would behave any
differently. Eventually I wrote a task that scanned through all
entities of a given kind, about half of which I knew were orphans.
The task simply looked up the entity's key's parent key, and if the
parent key didn't correspond to a datastore object then the task
deleted the entity. After this, when I tried the transfer again it
completed in a few minutes.
good hour, watching it slowly eat my CPU, then I finally purged my
queue, removed all the tracker objects from the datastore, totally
wiped the datastore in the target app (which had had some fraction of
the data transferred to it), and tried again. Same result. Then I
set about randomly changing things to see if it would behave any
differently. Eventually I wrote a task that scanned through all
entities of a given kind, about half of which I knew were orphans.
The task simply looked up the entity's key's parent key, and if the
parent key didn't correspond to a datastore object then the task
deleted the entity. After this, when I tried the transfer again it
completed in a few minutes.
Joshua Smith
Jun 23, 2011, 1:07:18 PM6/23/11
to google-a...@googlegroups.com
Along these lines, if I have an object that has a reference property to another object, and that object was deleted, but the original reference wasn't cleared, is that going to cause problems when I migrate to HR? Or does the data transfer utility just duplicate the keys without worrying about whether they refer to objects that actually exist?
-Joshua
Reply all
Reply to author
Forward
0 new messages