Sydney Data Center and migration of app engine application

[Mark Hetherington]

Hi,

With the commissioning of the Sydney data centre, there are no doubt many Australia based customers wanting to migrate to the new data centre. I imagine the same was true when other locations opened up worldwide.

We have existing app engine, cloud sql, cloud storage and data store assets we would like to migrate. Has anyone undertaken this process? Our understanding is that we can create a new application in the new zone, and do a backup and restore. However there are a few hurdles. Some of the issues we see are:

• The application name currently in use won't be available for use in the new zone, thus requiring a rename of the application
• The process of cutting over between zones may require us to update a number of DNS A records managed by third parties. 
• The CNAME to ghs.googlehosted.com must cut over to the new application at some point, but I'm not sure how this is managed. Presumably when verifying the domain into the new application google starts serving a new result when ghs.googlehosted.com is queried?

Are there any migration tools available? Does google offer any assistance in this regard? According to a news article at https://www.crn.com.au/news/google-cloud-finally-comes-to-sydney-today-465735, "

Google Cloud platform managing director APJ Rick Harshman told CRN that customers wanting to migrate to local availability zones can contact one of its partners, or Google's customer engineering team to help with the migration.", but I'm not sure how to get in touch with the "customer engineering team".

Cheers,

Mark

[Stefano Ciccarelli]

Hi,

we are EU based and early App Engine users, we have the same kind of problem because we'd like to move to EU.

I'm watching for answers.

Bye!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/92e5a3ca-c3a8-4507-8f2f-d6961aad239b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Stefano Ciccarelli 
GAE Application Division 
/ Director 
stefano.c...@mmbsoftware.it 

M.M.B. s.r.l. 
via Granarolo, 177/7 - 48018 Faenza (RA) - Italy 
tel. +39.0546.637711 - fax +39.0546.46077 
www.mmbsoftware.it - in...@mmbsoftware.it

Le informazioni contenute in questa comunicazione sono riservate e destinate esclusivamente alla/e persona/e o all'ente sopra indicati. E' vietato ai soggetti diversi dai destinatari qualsiasi uso, copia, diffusione di quanto in esso contenuto sia ai sensi dell'art. 616 c.p., sia ai sensi del DL n. 196/03. Se questa comunicazione Vi e' pervenuta per errore, Vi preghiamo di rispondere a questa e-mail e successivamente cancellarla dal Vostro sistema.

[Anastasios Hatzis]

We moved our app from US to EU last year, that is GAE Python Standard with Cloud Storage and Cloud Datastore.

• Yes, you will need a new project (with new app name, datastore, storage), the region/zone selection might be hidden under "advanced setup", if I remember correctly
• Your assumption about CNAME is correct, and the switch happened within seconds. However, for both, A and CNAME records I suggest to temporarily decrease the DNS TTL values before you actually migrate, so clients will resolve the names faster to your new app

I suggest to make a list of all services needed in your project. Then check if the new region/zone has different limits or quotas (I'm not sure if this is still an issue today). You might also want to check if the pricing in the new region/zone is different.

Since we were able to take the app offline for a few hours, backup & restore with the Datastore Admin tool was sufficient. Keep in mind though that indexing might take significantly more time than the actual import of the data. If your app has huge data size, you can try to perform two back & restores... maybe the (second) indexing is faster when most of the data is already existing. If you cannot turn off the app altogether, maybe it is possible to disable writes to the old datastore until the migration is completed.

For Cloud Storage, the Console includes a Transfer tool between buckets (or other services). We didn't use it, so I can't tell how much time is needed. However, it offers features to perform the transfer multiple times.

One issue we almost missed: if your app uses Task Queues, you need to decide how to handle them. For example, if you have non-idempotent tasks, you probably don't want the same task to run multiple times (in the old and the new app). In our case, we paused all such queues before running the datastore backup (including the hidden kinds that also include the tasks).

We also performed a side-by-side comparison of both projects in Google Cloud Console, especially with regards to API settings and service accounts.

And a final remark (#CaptainObvious): we used the very same code-base between the old app and the new app for the entirety of the migration.

Wish you the best with your migrations!

Ani

Mit freundlichen Grüßen / Kind regards

i. A.
Anastasios Hatzis

Fon: +49 8374 930813
Fax: +49 8374 930810
Mobil: +49 1520 8592878

2017-08-29 12:45 GMT+02:00 Stefano Ciccarelli <stefano.c...@mmbsoftware.it>:

Hi,

we are EU based and early App Engine users, we have the same kind of problem because we'd like to move to EU.

I'm watching for answers.

Bye!

Il giorno mar 29 ago 2017 alle ore 10:34 Mark Hetherington <ma...@keodesign.com.au> ha scritto:

Hi,

With the commissioning of the Sydney data centre, there are no doubt many Australia based customers wanting to migrate to the new data centre. I imagine the same was true when other locations opened up worldwide.

We have existing app engine, cloud sql, cloud storage and data store assets we would like to migrate. Has anyone undertaken this process? Our understanding is that we can create a new application in the new zone, and do a backup and restore. However there are a few hurdles. Some of the issues we see are:

• The application name currently in use won't be available for use in the new zone, thus requiring a rename of the application
• The process of cutting over between zones may require us to update a number of DNS A records managed by third parties. 
• The CNAME to ghs.googlehosted.com must cut over to the new application at some point, but I'm not sure how this is managed. Presumably when verifying the domain into the new application google starts serving a new result when ghs.googlehosted.com is queried?

Are there any migration tools available? Does google offer any assistance in this regard? According to a news article at https://www.crn.com.au/news/google-cloud-finally-comes-to-sydney-today-465735, "

Google Cloud platform managing director APJ Rick Harshman told CRN that customers wanting to migrate to local availability zones can contact one of its partners, or Google's customer engineering team to help with the migration.", but I'm not sure how to get in touch with the "customer engineering team".

Cheers,

Mark

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/92e5a3ca-c3a8-4507-8f2f-d6961aad239b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Stefano Ciccarelli 
GAE Application Division 
/ Director 

stefano.ciccarelli@mmbsoftware.it 

M.M.B. s.r.l. 
via Granarolo, 177/7 - 48018 Faenza (RA) - Italy 
tel. +39.0546.637711 - fax +39.0546.46077 

www.mmbsoftware.it - info@mmbsoftware.it

Le informazioni contenute in questa comunicazione sono riservate e destinate esclusivamente alla/e persona/e o all'ente sopra indicati. E' vietato ai soggetti diversi dai destinatari qualsiasi uso, copia, diffusione di quanto in esso contenuto sia ai sensi dell'art. 616 c.p., sia ai sensi del DL n. 196/03. Se questa comunicazione Vi e' pervenuta per errore, Vi preghiamo di rispondere a questa e-mail e successivamente cancellarla dal Vostro sistema.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAArRkeW%3DkWz2p6UpCmGJXPwPA_H9n5Tx7oit8j9jtaKmwyd6rQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

HATZIS Edelstahlbearbeitung GmbH
Hojen 2
87490 Haldenwang (Allgäu)
Germany

Handelsregister Kempten (Allgäu): HRB 4204
Geschäftsführer: Paulos Hatzis, Charalampos Hatzis
Umsatzsteuer-Identifikationsnummer: DE 128791802

http://www.hatzis.de/

[Yannick (Cloud Platform Support)]

That is a great explanation Anastasios! As they point out migrating your app to a new region is a manual process where you have to move each service individually. If you have a support asset then you can contact support for advice related to your specific circumstances, but they will not perform the migration for you.

This being said if you have any questions regarding how to migrate certain parts of your project this is a great place to ask them!

On Tuesday, August 29, 2017 at 7:53:08 AM UTC-4, Anastasios Hatzis wrote:

We moved our app from US to EU last year, that is GAE Python Standard with Cloud Storage and Cloud Datastore.

• Yes, you will need a new project (with new app name, datastore, storage), the region/zone selection might be hidden under "advanced setup", if I remember correctly
• Your assumption about CNAME is correct, and the switch happened within seconds. However, for both, A and CNAME records I suggest to temporarily decrease the DNS TTL values before you actually migrate, so clients will resolve the names faster to your new app

I suggest to make a list of all services needed in your project. Then check if the new region/zone has different limits or quotas (I'm not sure if this is still an issue today). You might also want to check if the pricing in the new region/zone is different.

Since we were able to take the app offline for a few hours, backup & restore with the Datastore Admin tool was sufficient. Keep in mind though that indexing might take significantly more time than the actual import of the data. If your app has huge data size, you can try to perform two back & restores... maybe the (second) indexing is faster when most of the data is already existing. If you cannot turn off the app altogether, maybe it is possible to disable writes to the old datastore until the migration is completed.

For Cloud Storage, the Console includes a Transfer tool between buckets (or other services). We didn't use it, so I can't tell how much time is needed. However, it offers features to perform the transfer multiple times.

One issue we almost missed: if your app uses Task Queues, you need to decide how to handle them. For example, if you have non-idempotent tasks, you probably don't want the same task to run multiple times (in the old and the new app). In our case, we paused all such queues before running the datastore backup (including the hidden kinds that also include the tasks).

We also performed a side-by-side comparison of both projects in Google Cloud Console, especially with regards to API settings and service accounts.

And a final remark (#CaptainObvious): we used the very same code-base between the old app and the new app for the entirety of the migration.

Wish you the best with your migrations!

Ani

Mit freundlichen Grüßen / Kind regards

i. A.
Anastasios Hatzis

Fon: +49 8374 930813
Fax: +49 8374 930810
Mobil: +49 1520 8592878

[Paul Ceccato]

That article says that Google App Engine is not available in the Sydney region and will come later...

[Yannick (Cloud Platform Support)]

App Engine is currently available in Sydney. You can check product availability for yourself using this chart.

[Paul Ceccato]

So it is. I managed to recreate my app in the sydney region and migrate the data from a datastore backup.

I don't know if its my imagination, but the app seems snappier with the reduced latency

Paul

[Paul Ceccato]

well, I managed to migrate my app to the australian zone... it's a very manual process involving recreating the app in the new zone and restoring the datastore from a backup. However, it didn't quite work 100%

When I remapped my custom domain to my new app, without warning somehow my ssl certificate vanished from the appengine console.

Now, when I try to upload the same certificate to my new app it fails with an extremely unhelpful error message (see below)

hopefully it will starting working eventually... I have been getting that error for the last 12hours 

in the meantime my application configuration has been changed to only allow insecure connections...

Error

Sorry, there’s a problem. If you entered information, check it and try again. Otherwise, the problem might clear up on its own, so check back later.

Tracking Number: 460337055749622297

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/mdsYsfNk50o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.

To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/e4b6c960-15b1-46e3-8204-15d18d2afe15%40googlegroups.com.

[Yannick (Cloud Platform Support)]

Hello Paul, I understand you're not able to remove the certificate from your old application? If this is still a problem for you please create a new private issue on the Issue Tracker using this link with all relevant details as well as the affected project IDs and a HAR Capture of the interactions leading to this error message. 

[Paul Ceccato]

no, the certificate seemed to remove itself when I remapped the custom domain. I tested the app was working at the time but the next morning it wasn’t serving as the app was set to secure: always

Then I had a problem uploading the new certificate, because I accidentally used the wrong private key file, however, the error message that is returned is very vague and not particularly useful.

Anyway, its all resolved now, thanks!

Paul

To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/1c8e1f36-b19e-4bfc-b1d4-8f78e2db5763%40googlegroups.com.