Your migration stories

[Charlie Engelke]

Have you recently migrated from App Engine for Python 2.7 to App Engine for Python 3.7? Are you working on migrating now, or planning to soon? Or maybe you are using App Engine for Python 2.7 and aren't planning such a migration.

In each of those cases, I'd like to hear how things went or are going for you, pain points you encounter, things that went easily, and anything else you have to share. I'm working on creating some new code samples to help with this, and your stories will be a big help.

Thanks,

Charlie

[Sebastien Dupere]

Hi Charlie,

We did not have much pain except the change in the code (like prints and str and bytes being more exclusive for example). This had nothing to do with AppEngine. 

** If anything moving to the latest python made sure we were using the latest google cloud libraries and avoid more issues.

* Are you using AppEngine Standard or AppEngine Flexible?

** This might impact the group's feedback.

We used AppEngine Flexible.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/cd38289b-8848-416f-a369-ccea982b4bee%40googlegroups.com.

--

Thank you,

Sebastien Dupere

[David Trupiano]

Hi Charlie,

We've almost completed our migration off of AppEngine python 2.7.  We ended up just jumping over to Kubernetes rather than the python 3 version of AppEngine.  The biggest reason for this being that we ended up needing to manage the replacements for many of the builtin services from 2.7 (ie Elasticsearch to replace the search API, Redis to replace Memcache etc...) so it was just easier to put everything into a Kubernetes cluster (just one thing to manage, most of these services have operators available, some of the services will be co-located,  etc...). The Kubernetes move will also remove our dependence on any particular vendor which is nice to have (especially given this recent experience with AppEngine).

The biggest pain points for us were not really updating any code but rather around how to replace the builtin services from 2.7 and how those changes ended up impacting our dev workflow.  Before it was really simple: just run the emulated environment locally and start coding away.  Now, the dev workflow is a lot more complex.  We are doing a lot with docker and minikube to make everything work locally before we deploy.  I would also say that a big reason that our code updates were fairly easy was the extensive tests that we had built out - huge lifesaver for detecting issues during the transition.

Hope this helps

Dave

[Charlie Engelke]

I'd still like to here anyone's stories involving migrating their App Engine apps from Python 2.7 to Python 3.7. Have you migrated yet? Planning to do so soon? Not going to move to the new platform?

And any particular pain points? Things that would have helped? 

Thanks,

Charlie

[Joshua Smith]

I have several mission critical services running in P2.7. About 13,000 lines of python code.

To get a feel for what P3.7 would be like, I used the new environment to create a couple new services. It was not fun. Everything about the new environment is 10x more complicated, working with a local database is well neigh impossible, and the new code patterns are different enough from the old code patterns, that I’m looking at a line-by-line rewrite.

For example, to delete something from google.appengine.ext.db

model.delete()

To delete something from google.cloud.ndb

model.key.delete()

It’s a small change, but since Python is not compiled, nothing is going to help me find that other than rigorous search and runtime errors.

All told, I think migrating all my services from P2.7 to P3.7 is going to take at least a couple person-months of labor. It’s a nightmare.

Google completely botched this. They should have created polyfills so we can just deal with the few language change issues, but not have to switch to an entirely different set of services.

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/850cd6fa-77a4-436d-8d7e-40eab67f84eb%40googlegroups.com.

[Charlie Engelke]

This is very helpful feedback. Anybody else have comments for us?

[Ryan B]

hi charlie, thanks for asking!

my experience was similar to joshua's, if not quite so extreme. i've migrated seven or so apps now, the largest weighing in at 26kloc. the big differences in API availability and compatibility were the most painful parts, followed by changes and difficulties with local development, as joshua mentioned.

here are a handful of threads and issues with details:

• local dev, deployment:
  
• https://groups.google.com/d/topic/google-appengine/BJDE8y2KISM/discussion (fast startup or static assets, choose one?)
  
• https://issuetracker.google.com/issues/144150446 (dev_appserver patch to fix this)
  
• https://issuetracker.google.com/issues/146780347 (Cloud Build uses stale GitHub clones)
  
• logging:
  
• https://groups.google.com/d/topic/google-appengine/xTDZ-da27ow/discussion (difficulties reading GAE app logs from the logging API)
  
• https://stackoverflow.com/questions/58753378/how-to-deserialize-app-engine-application-logs-from-stackdriver-logging-api
• https://github.com/googleapis/google-cloud-python/issues/2572
• https://github.com/snarfed/bridgy/issues/940 (my app's feature that uses logging API is now way slower)
  
• https://groups.google.com/forum/#!topic/google-stackdriver-discussion/k96EQHCacKM (can't request increase to logs API read quota)
  
• https://stackoverflow.com/questions/59398479/why-is-my-app-engine-app-hitting-the-stackdriver-logging-256kb-limit
• https://github.com/snarfed/bridgy/issues/904 (originally ingested way more logs than i expected)
  
• ndb, misc growing pains:
  
• https://github.com/googleapis/python-ndb/issues/263
  
• https://github.com/googleapis/python-ndb/issues/238
• https://github.com/googleapis/python-ndb/issues/277
• https://github.com/googleapis/python-ndb/issues/376
• https://github.com/googleapis/python-ndb/issues/267
• cloud tasks missing local emulator, transactional tasks
• https://cloud.google.com/tasks/docs/migrating#features_in_task_queues_not_yet_available_via
• migration increased our frontend instance usage noticeably, had to rearchitect and extract out task processing into a separate background service to get costs back down: https://github.com/snarfed/bridgy/issues/578
  (not entirely sure this was GAE's fault)
• shared memcache got lost entirely, so i had to switch to a runtime in-memory cache, which isn't as helpful
  
• https://github.com/GoogleCloudPlatform/webapp2/issues/151 (webapp2 is unmaintained, and its port to Python 3 is a bit rough)
  
• https://groups.google.com/d/topic/google-appengine/WHo99Gz1S3g/discussion (difficulty finding replacement for app_identity module)
  

[Ryan B]

i was also extremely fortunate that pretty much all of these apps had solid unit test coverage, which automatically found the vast majority of code that needed updating. if i'd had to resort to "rigorous search and runtime errors," as joshua mentioned, i'd probably be a lot more frustrated.

[Alex Martelli]

Just as a personal note (I'm a Googler but also help maintain App Engine non-profit sites in my spare time etc), I enthusiastically agree that the migration suddenly validated my own long-standing obsession with more and more (automated, esp. unit) testing -- suddenly, that boring test activity became a life-saver, catching most glitches well before moving into staging, much less prod.

Unsurprisingly the main exceptions (which we had to catch over time) were about code hidden in Jinjia templates, which I had never thought of subjecting to the same ambitious, aspirational standards of coverage I hold Python code in .py files to.

There ARE ways to unit test code hidden in Jinja, and I plan to research and apply them, but the error I made as a tech lead in not insisting on such testing earlier is yet another confirmation of how precious unit tests ARE!-)

Alex

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5f63c9c5-72f3-40f5-a0b5-cd92979cda2d%40googlegroups.com.

[Ryan B]

hey alex, long time no see! yup, tests were a life saver for me in these migrations. probably also the single best technique i know of for software productivity and quality over time in general.

re testing jinja templates, i've gradually leaned toward writing more and more web app unit tests at the HTTP request level, integration test style, which makes testing templates and HTML output easier. set up the context, make the request, check the HTML output. example:

https://github.com/snarfed/bridgy/blob/dab476ec1f7cfd261341f381de645edaf3d97599/tests/test_app.py#L375-L381

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

[Charlie Engelke]

This is extremely helpful. Thanks for posting such detail. I hope we can use it to help people migrate more easily.

[alexander noteboom]

We are in the middle of the migration.  It is a horrible experience.

We tried to break it into small steps. 

Still in python2, we first replaced memcache with redis and  replaced search with elasticsearch.  These steps were do-able.

Next we tried to make the python2 code compatible with python3 using futurize.  Had a lot of problems with future_builtins and unicode, so we futurized except those.

After that we tried to migrate to Cloud Tasks, but that failed horrible because of the lack of transactional tasks. Our application heavily relies on that.

We thought up a solution, but that required the migration to Cloud ndb. So we have to combine these two steps.

That is where we currently are. Fighting with Cloud Tasks, Cloud NDB, namespaces, and getting it running using python3 on a local machine.

The most annoying thing is that there is no local Task Emulator available, and no Datastore Emulator viewer available. 

[Charlie Engelke]

Thanks for this info. Sorry it's so difficult, and I'll look to see what we might have to help. One thing I worked on was NDB migration. There's a sample at https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/migration/ndb/overview.

Basically, you use from google.cloud import ndb instead of the old import, create a global client with something like "client = ndb.Client()", and then wrap your old NDB operations in a context:

    with client.context():
        do_old_ndb_operation

This works fine under Python 2.7 and Python 3. There seems to be a performance hit when you use this library under Python 2.7, though it might not matter unless you are database-bound. I've been doing benchmarks, and the performance seems to return to normal when you use this in Python 3.

If you are already using a Redis cache, it's easy to add that to the new NDB by adding the parameter global_cache=redis_cache_client to the client.context calls. Using that cache may not make much difference either way, though; it depends on the pattern of database access specific to your usage.

Charlie

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/BXHOXut7yGA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/2de767ae-ee69-4e3c-9db4-2c1a1897c045o%40googlegroups.com.

--

Charles Engelke

GCP AppDev DPE