HRD migration tool now available!

[Stacy (Google)]

We’ve just released a new self service HRD migration tool.  This tool copies all of your data to HRD minimizing read-only times by using a multi-phase approach.  Please see the documentation for details on how to use this.  This tool is still considered experimental, please report problems you experience here in the forum or to the external issue tracker.We are still working on solutions for users of blobstore.
Using HRD makes your app eligible for our SLA, and will lead to much better availability and more consistent performance.  We think every app will benefit from the switch to HRD.

Stacy

[Kenneth]

Do you have any sample code for downloading and uploading blobs?  Or am I just being lazy?

[johnP]

Is it possible to have an option to perform a test migration before
committing to the actual migration? I suspect that my app will
require quite a bit of tweaking (changing queries to be ancestor
queries; updating string-encoded keys; etc.) before going live under
HRD. It would be great to do a test migration, test/update code until
it's reliable on HRD (freely, without worrying about damaging user
data), and only then do an actual migration.

johnP

[Stacy Kerkela]

You can do the initial copy, and as many syncs as you would like - 

then test as much as you want - and then sync again before flipping the alias.

I don't think we've provided a way to delete all the data, and start the copy from scratch.

-- Stacy

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Jim]

I'd like to verify this:

I've taken one app through the migration process following all the steps in the instructions.   

It looks like there's one more step: delete the original app.   That way I get an application slot back and there's no confusion about which apps are active.   

The instructions for deleting an app say it takes 72 hours and the name can't be re-used.

My preference would be to "OK, you're migrated, now you can rename foo-hrd to foo".   

Any chance of that happening?

Thanks for providing the tool.

[Joshua Smith]

Guess I know what I'll be doing on my day off tomorrow.

In case anyone didn't notice this in the docs:

Currently, the HRD Migration Tool does not support copying data from Blobstore. If you use Blobstore, you need to manually copy that data over.

Ugh.  That's a pretty huge hole in the tool.

Perhaps I know what I'll be doing all weekend…

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/RzmZ--blDgIJ.

[Darien Caldwell]

I'm still not clear about the domain through Google Apps that points
to the application. Does this tool switch the domain over too? Or does
that have to be done manually? And if manually, how do you do it?

[Joshua Smith]

OK, so the migration from MS to HR was really fast, and I ran into no problems.  Woo hoo!

I didn't do the alias step, because I now need to figure out how to move my old blobs into the new app, and I suspect that if I did the alias that would get really, really hard. Right?

So instead, I changed the mapping of my www.myapp.com to point to the HR version using google apps domain management.  Seems to have worked.  Woo hoo again!

How do I get my post-migration email budget back up to 2000?  App ID of the new app is towngovernment-hr

Now to move the blobs...

[Joshua Smith]

Another question. I have this code in my app, so that when someone comes in with an old key, my code will quietly convert it to the new app-id:

from google.appengine.api.app_identity import get_application_id

appname = get_application_id()

@monkeypatch_method(webapp.Request)

def get_db(self, key_attr):

  k = self.get(key_attr)

  if not k:

    return None

  key = db.Key(k)

  if key.app() == appname:

    return db.get(key)

  logging.info("Fixing up key from old app %s (%s=%s)" % ( key.app() , key_attr , k ))

  fixed_key = db.Key.from_path(*key.to_path())

  return db.get(fixed_key)

So what's happening is it is "fixing up" every key, because get_application_id() doesn't have "s~" on the front.

What should I call to get the app id used in Keys? (The one with the s~ in front?)

(BTW: This approach turned out to be a real time-saver for migrating my blobs. I could pass the old version keys from the new HR version, and it fixed them up into its own namespace without my having to even think about it.)

-Joshua

[Alfred Fuller]

full_app_id = os.getenv('APPLICATION_ID')

[Joshua Smith]

Cool. And that will be OK to use in Python 2.7 (I recall reading something about not using environment variables there)?

[Alfred Fuller]

I believe it should work. Tou should probably file a feature request to add a function for this purpose to that app_identity api.

[Joshua Smith]

Done.  Issue 6327

[Brian Quinlan]

On Mon, Nov 14, 2011 at 8:13 AM, Joshua Smith <Joshua...@charter.net> wrote:
> Cool. And that will be OK to use in Python 2.7 (I recall reading something
> about not using environment variables there)?

Environment variables are thread-local in Python 2.7 so this pattern
will work there as well.

Cheers,
Brian

[Dennis]

Will this tool move from "experimental" to "production"?   Any ETA for the production version?

It seems that the migration is a 1-way process and cannot be undone.

Thus, I would like a solid migration process before using it.

[MikeR]

I migrated my app 'bla' to 'bla-hrd' which worked fine.
Trying to copy the blobstore failed however. I have 98 blobs in 'bla'
and 1 test blob in 'bla-hrd'. Querying both 'bla' and 'bla-hrd' my
code always returns 1 blob. It used to return 0 blobinfo when the test
blob in 'bla-hrd' wasn't created yet. It seems as if all blobstore
access is routed to the new hrd application, making my old blobs
unreachable. I also tried using the BlobInfoService (can't remember
exact name) with same results.

Have a look at the code below and let me know if I missed something or
if this is a bug in the migration tool:

http://www.copypastecode.com/128643/

[Simon Knott]

I didn't think blobstore migration was supported until a few months into next year - I'll try and dig out which thread specified the timescales.

Until it's supported, you have to manually migrate all of your blobstore contents.

Cheers,

Simon

[MikeR]

Hi Simon,

Thanks for the reply; I'm not expecting an automatic migration, I'm
just not able to do the manual migration (using the remote_api).
Hoping someone can take a look at my code and point out an obvious
mistake...

Mike

[Alfred Fuller]

Did you alias your old app id to your new app id? Cause if that is the case, then remote_api is likely talking to the new app (and I don't think it is possible to make it talk to the old app).

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[MikeR]

> Did you alias your old app id to your new app id? Cause if that is the
> case, then remote_api is likely talking to the new app (and I don't think
> it is possible to make it talk to the old app).

can't remember actively doing that (except maybe as a checkbox in the
migration process).
Could you tell me how aliasing is done?

If I can't talk to my old app via remote_api, any suggestions how to
copy the old blobs to the new hrd app ?

thx
Mike