appcfg to gcloud app deploy migration guide?

[Brian]

I just installed/updated my local gcloud SDK after a Ubuntu 14.04 to 16.04 upgrade. Apparently, appcfg is no longer available by default. I'd like to start using the preferred gcloud commands to manage my projects but I can't find an "appcfg" to "gcloud app deploy" migration guide anywhere. On Stack Overflow a while back, Zachary mentioned a migration guide was planned.

From what I've read, the changes look good, but there seems to be several gotchas that could really mess-up my active projects, so I would like to review the migration guide in fine detail to avoid any problems.

Please let me know where I can find the migration guide so I can keep my projects moving forward. (At this point, even a rough draft would be very useful!)

(I would have commented on the original Stack Overflow thread, but as a newbie there I'm not allowed because I don't have a high enough reputation yet.)

[Adam (Cloud Platform Support)]

Pre-release documentation can't be made available to the public unfortunately. My recommendation is to read the current available App Engine and gcloud docs and test. I can list a few main differences here though:

1) New versions are promoted to receive all traffic by default. I'm of the opinion that this is bad, and hopefully the default will change, but for now you should disable this with the following command:

gcloud config set app/promote_by_default false

2) In your app.yaml and service .yaml files, the 'application: ' and 'version: ' fields are not used. You should set the project to deploy to with 'gcloud config set project', and the version is automatically generated, or set with the '--version=' flag to 'gcloud app deploy'.

3) You can use 'gcloud app deploy <deployable.yaml>' for any .yaml, such as cron.yaml, dispatch.yaml and indexes.yaml. No need for separate commands like update_indexes and update_dispatch.

4) For Java apps, everything is done using the new Gcloud Maven Plugin which handles translating the .xml config files to .yaml files for deployment. It's similar to how the App Engine Maven Plugin worked on the standalone SDK, but the plugin goals are different.

5) There is no vacuum_indexes command, instead you would use 'gcloud preview datastore cleanup-indexes'.

[Brian]

Thanks, Adam -- that's a big help!

I'm making the changes to my workflow based on your suggestions and the related documentation, however the process has raised a few questions:

1) I haven't noticed dispatch.yaml mentioned in the new gcloud command documentation. Should this be included with the other yaml files for my default app? (app.yaml dos.yaml index.yaml) If I need to update just a service, (service1.yaml) should I include the default dispatch.yaml too?

2) Since the dispatch.yaml file applies to my entire project (both default app and services) if I update it alone with the gcloud command, is the --version flag ignored, or would it know to apply it just to my default project app, and not the individual version ids of the services?

3) Since the "version" id field is no longer used in the .yaml files, it appears that I can not update both my app and services in a single command AND include unique version ids for each with a single --version flag. In this case is it best practice to update the app and services individually so I can specify unique version ids for each?

4) While the "version" id field is no longer used in the .yaml files, the documentation suggests otherwise. On https://cloud.google.com/appengine/docs/python/tools/uploadinganapp under "Updating Indexes" it states, "Define a different version ID in your app's app.yaml file." Is this accurate or not?

Any refinement you can offer on these issues would be very helpful.

Brian

[Attila-Mihaly Balazs]

Great discussion, definitely something I'll keep my eye on since dev_appserver doesn't seem to be maintained any-more (for example the version check doesn't show that "1.9.49" is available).

Couple of follow-up questions:

- is there a built-in solution for deploying multiple modules in parallel? (we have 8 modules and it would be nice to speed up deployment)
- could we get a clear documentation on how routing is affected by dispatch.yml? in particular:
  - does having dispatch.yml mean that individual services can no longer be accesses via the "servicename-dot-appname.appspot.com" syntax?
  - does the answer to the above change depending on if the dispatch.yml ends in a "*/*" url or not?
  - do requests from PUSH taskqueues go trough the dispatcher? do requests from cron?

Thank you,
Attila

[Adam (Cloud Platform Support)]

To answer Brian's questions:

1) The gcloud app deploy documentation mentions DEPLOYABLES as such:

        A DEPLOYABLE can be a service's .yaml file or a configuration's .yaml file.

It might help to update the documentation to be more specific about what is meant by 'configuration's .yaml file', but this does mean any .yaml file listed in the 'Configuration files' section of the documentation such as 'queues.yaml, dispatch.yaml, dos.yaml, cron.yaml and index.yaml'.

2) The dispatch rules are applied project level, so the --version flag is ignored. This is the same behavior as the standalone SDK.

3) The --version= flag specifies the version for this deployment. If you are targeting different versions for different services, you'll need to use separate commands.

4) This part of the docs is now incorrect, and should be edited. Following the link to 'app.yaml' shows that the 'version: ' field is not present, and if you test 'gcloud app deploy app.yaml' with either 'version: ' or 'project: ' present, you'll get an error.

For Attila:

1) I've tested parallel deployments for the same version by backgrounding them in a terminal and it appears to work, however I can't guarantee there won't be any issues as its not supported. Eg.

$ gcloud app deploy -q -v myversion app.yaml & \
gcloud app deploy -q -v myversion module1.yaml & \
gcloud app deploy -q -v myversion module2.yaml

The '-q' flag is necessary to avoid the prompt. Otherwise, there's no builtin solution, and you might consider filing a feature request.

2) Routing with dispatch.yaml is and has always been covered by the documentation, specifically 'How requests are routed' and the 'dispatch.yaml' reference. Using dispatch.yaml does not remove subdomain routing, but you can use it to route by domain. The exception is on the development server, where there exists no subdomain to route on (you could add one in /etc/hosts, but this would be another feature request to support it).

        Note: Dispatch rules apply to the URLs in a cron file.

        Note that dispatch rules also apply to URLs that are used in cron configuration or in task queue configuration.

      The server does not support dispatch.yaml file rules that include hostnames (for example, url: "customer1.myapp.com/*"). 

[George Bittmann]

On Saturday, January 7, 2017 at 2:54:10 AM UTC-5, Attila-Mihaly Balazs wrote:

Great discussion, definitely something I'll keep my eye on since dev_appserver doesn't seem to be maintained any-more (for example the version check doesn't show that "1.9.49" is available).

dev_appserver.py is still referenced in the documentation for local development.

https://cloud.google.com/appengine/docs/python/tools/local-devserver-command 

Are you talking about appcfg instead?

[Attila-Mihaly Balazs]

No, I'm talking about dev_appserver which says on startup:

INFO     2017-01-11 07:24:37,002 sdk_update_checker.py:229] Checking for updates to the SDK.
INFO     2017-01-11 07:24:37,492 sdk_update_checker.py:273] This SDK release is newer than the advertised release.

This seems to indicate to me that current / legacy dev_appserver and the infrastructure around it (like the URL it pings to check for the latest released version) are no longer being kept up-to-date.

Attila

[George Bittmann]

By default it seems to skip the update check so I had to add this flag, but here are my results.

Running:

dev_appserver.py /wamp/www/myprojecthere --skip_sdk_update_check=false

I just got this:

INFO     2017-01-11 10:44:53,634 sdk_update_checker.py:231] Checking for updates to the SDK.

INFO     2017-01-11 10:44:54,176 sdk_update_checker.py:259] The SDK is up to date.

[Kartik Domadiya]

How do we specify the environment variables while deploying using gcloud app deploy command ?