HRD migration tool hangs for hours - how to find a root cause?

[Alexander Trakhimenok]

We are migrating our app to an HRD version and the migration tool process hangs on 2nd step "Copy" at 99% for 10+ hours.

We've restarted migration but it did not help - still hangs at 99% and the "approximately remaining" time is steadily and slowly growing.

I've been googling around and as I understand it can be because of indexes issue (5K index entry limit per entity) or some other reason that prevent an entity to be stored to the new HRD database. Actually removing some indexes helped us to move from 98.50% to 99% and  not  any further.

But we can not find any error/status logs that would point us on what exactly entity kind/id causing the issue.

On the Migration Tool screen it says:
=====
If there is a problem, you can pause the migration. While paused, all migration mapreduces will be paused and but no other actions will be taken until an administrator either resumes the migration or reverts the process. To pause the migration, please click the button below.
=====

But how do we know if there is a problem and what exactly the problem is? It would be cool to have a "last or list if error messages & entity keys for the migration process.

Any ideas how to identify and solve the issue? We have major release pending due to limitations of M/S database and would greatly appreciate any help.

I've raised a production issue for this as well http://code.google.com/p/googleappengine/issues/detail?id=7787

but may be someone from community can help us to resolve it before Google Engineers notice the ticket.

Alexander Trakhimenok

Lead developer on http://www.myclasses.org/ project powered by GAE (AppEngine + Python + Django).

[Waleed Abdulla]

On my initial copy I remember it staying at 99% for a long time, but it eventually finished. I was copying a lot of data so the full copy took 3.5 days. In the sync phase, I remember it staying at 99% for almost half the time of every sync and sometimes more. My first sync took 12 hours, and the second 6 hours. On the last sync it stayed on 99% too long so I stopped it and jumped to the last stage (read-only mode) and that made it much faster. 

Waleed

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/AYfYbuyKGsYJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.