Migrating to HRD from Master/Slave

[luka]

We are considering to migrate our master/slave application to HRD
one.

We have a few questions regarding this process:

1. Today we have other collaborating site which uses our application
URL "lugotest.appspot.com" (we used it instead of the domain name for
SSL reasons).
Will we be able to preserve the application URL? or will we have to
change it to new application URL name.

Is Google app engine support SSL for non "appspot" domains?

2. Is the only way to migrate data is to use python instructions?


3. Is there a way to purchase support for this operation to ensure the
success of the migration?

[luka]

Regarding the SSL I saw that I can define a reverse proxy to support
it.

The real thing that I am concerned is the application URL, can I keep
it while migrating to the high replication datastore?

Thanks,
Uri

[Robert Kluin]

Hi,
See responses inline.

On Tue, Jun 21, 2011 at 11:31, luka <ulu...@gmail.com> wrote:
> We are considering to migrate our master/slave application to HRD
> one.
>
> We have a few questions regarding this process:
>
> 1. Today we have other collaborating site which uses our application
> URL  "lugotest.appspot.com" (we used it instead of the domain name for
> SSL reasons).
> Will we be able to preserve the application URL? or will we have to
> change it to new application URL name.

You can request an alias from the old appid to the new by submitting a
billing issue.

>
> Is Google app engine support SSL for non "appspot" domains?

It is on the roadmap, unfortunately it has been there a very long time
and there is still no ETA.

>
> 2. Is the only way to migrate data is to use python instructions?

You could use the built in datastore admin, or write something
yourself. I've actually done several migrations using my own tools to
directly move the data.

I also think with the new files API you could put something much more
efficient together by dumping your data to large blobs, moving the
blobs to the new app, then having something process the blob into
datastore entities.

>
>
> 3. Is there a way to purchase support for this operation to ensure the
> success of the migration?
>
>
>

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

[vlad]

Robert, why did you need to develop your own data migration tools? What is wrong with appcfg.py [upload_data/download_data] ? Admittedly I ran bulkloader only on small datasets but it was pretty quick and straight forward.

[Robert Kluin]

Hey Vlad,
One reason I wrote my own tools was because I was also doing schema
changes, that was a good opportunity to do so. I also did not want to
transfer all of my entities, so I used my app's standard APIs so that
all of my stats / reporting entities would just be regenerated
instead. I also didn't want to waste resources downloading then
re-uploading all of my data, rather I transferred them directly from
one app to another. If I were doing another large migration I would
probably dump to the blobstore, transfer the blob, then reconstitute
my entities from the blob.

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/SOIcCHwjsnMJ.

[Jeff Schnitzer]

One other issue seems to be that the bulk down/uploader doesn't have access to the indexed state of individual properties (there is no API to obtain this information).  So you might get the right data structure on restore, but all the wrong indexes (typically "none" or "all").  The only way around this right now seems to be a) feed a python schema to the bulkloader (annoying if you're in java, and broken if you have partial indexes), write your own tools, or just re-put all your entities using index-aware tooling post-load.

Jeff

[Greg]

The built-in datastore admin tool is excellent - see
http://neogregious.blogspot.com/2011/04/migrating-app-to-high-replication.html
for a discussion of data transfer options.

It won't do schema changes or data subsets, but knowing what I know
now I'd change the schema (see
http://groups.google.com/group/google-appengine/browse_thread/thread/9ae4c6fc3ca5acc0/520c294137bc8dbf)
and delete the unwanted entities in place, then use the datastore
admin tool. This keeps your keys, IDs and indexes correct.

Cheers
Greg.

[Robert Kluin]

Yeah, so far I've went with option 3 because one set of transfers
involved apps with several hundred million entities. Would have been
a real bummer to have the datastore admin crap-put midstream with
that. And downloading then reuploading that.... well, you decide.
The other *much* more envolved migration was a small but very complex
dataset and I wanted to make some very large schema changes. The
migration basically gave a fresh-start, I just had to write a old to
new conversion script.