Updates from the Google App Engine team (Fall 2021)

743 views
Skip to first unread message

wesley chun

unread,
Nov 4, 2021, 10:47:52 AM11/4/21
to google-a...@googlegroups.com

App Engine and Google Cloud serverless users:


The App Engine team at Google Cloud had a flurry of announcements and product updates over the past few months. In case you missed them, we've summarized them here with relevant links.



While we are working hard to improve your App Engine experience, recognize App Engine is no longer the only serverless option available from Google Cloud:

 

  • If you don't have an entire app, have single-function utilities or microservices, Cloud Functions is a great alternative; it also supports event-driven workloads

  • For those leveraging containerization as part of their app modernization or software development workflows or wish to avoid some of the constraints from App Engine or Cloud Functions, consider Cloud Run.


Both Cloud Functions and Cloud Run have similar execution profiles, are pay-per-use, deploy quickly, and autoscale as needed, just like what you're used to from App Engine. Providing a more complete serverless product suite that meets all your use cases and workloads is one of the goals of the Google Cloud serverless team.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"A computer never does what you want... only what you tell it."
    wesley chun :: @wescpy :: Software Architect & Engineer
    Developer Advocate at Google by day; at night: Core Python

Joshua Smith

unread,
Nov 4, 2021, 11:38:52 AM11/4/21
to google-a...@googlegroups.com
Thanks for sending out this update. I did, indeed, miss most of this news.

This item, in particular, is awesome:
This sounds like it will make it so much easier to migrate to Python 3.7.

One thing I've noticed is that my older apps on 2.7 seem to handle peak scaling a lot better than my newer apps on 3.7. For example, if I have a web page that hits a 2.7 app with 100 REST calls at startup (bad design, but it happens), the old app serves them all eventually. But if I do the same thing in a 3.7 app, it's likely to choke and fail a bunch of those requests with these:


The specific pattern is that after the first couple requests, it spins up a new instance. That takes 5 seconds to serve its first request (simple app, so I guess that's just GAE overhead). Then it spins up a couple more. Then I start getting those errors on some of the requests, because 15s have passed. In this capture the blue ones are new instances spinning up, and the orange ones are timeout errors (note the 15s time on those).


I've been designing around the issue by making sure my web apps go easy on the server on startup. But it is yet another concern about migrating.

Is there a way to tune the auto-scaling for 3.7 to behave like the 2.7? Here's a similar set of requests to my 2.7 app, which only spins up one extra instance, and never throws timeout errors, ever:


-Joshua


David (Cloud Platform Support)

unread,
Nov 4, 2021, 4:56:43 PM11/4/21
to Google App Engine

Hello,

Other than applying changes within your own application so it will go easy at startup, you can try using warmup requests along with tweaking the min_idle_instances element in your app.yaml, in order to reduce request and response latency during the time when your app's code is being loaded to a newly created instance.

There’s not an official way of forcing the scaling to behave like it used to be on python 2.7. However, this type of feedback can be passed to the App Engine engineering team in the form of a feature request. You can create such a feature request here. The App Engine engineering team would then evaluate it and decide whether it could be implemented or not.

Nicolas Fonrose (Teevity)

unread,
Nov 5, 2021, 5:50:42 AM11/5/21
to Google App Engine
Hello David,

>There’s not an official way of forcing the scaling to behave like it used to be on python 2.7.
Is there an official description, somewhere, of "how the scaling has changed" or "how it has been implemented differently" ?
Or this change just a side effect of technical choices that were made in the new runtimes?

Thanks,

George (Cloud Platform Support)

unread,
Nov 5, 2021, 2:50:52 PM11/5/21
to Google App Engine
There is no official description, somewhere, of "how the scaling has changed" or "how it has been implemented differently", as there is no change in scaling behavior, and no different implementation. This is an assumption while we investigate the situation described by Joshua above. I could not find a similar issue in the Public Issue Tracker, so the difference in scaling between Python 2 and 3 is likely a one-time occurrence, and caused by idiosyncratic settings in the project, or application-specific factors. Such issue are difficult to approach in a publicly-accessible thread as this one, as investigation needs particulars such as project ID, and access to other private data. To have this issue properly addressed, it's better to open a support case or an issue in the Public Issue Tracker. 

Jason Collins

unread,
Nov 5, 2021, 3:17:15 PM11/5/21
to Google App Engine
It sounds like there might be some differences due to concurrency between 2.7 and 3.7/8/9. There are some notes on how to tweak the number of worker processes started for Python 3 here: https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices

Joshua Smith

unread,
Nov 5, 2021, 3:29:43 PM11/5/21
to google-a...@googlegroups.com
I think it's telling me that since my P3.7 app has a F4_1G instance, I should be configuring it to run gunicorn with 8 workers. I don't see any information about the default number of workers. I guess I can do an experiment to see what explicitly setting that does...

-- 
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/150109bb-18a4-40b1-9efb-8b86bc60eceen%40googlegroups.com.

Jason Collins

unread,
Nov 5, 2021, 4:04:03 PM11/5/21
to Google App Engine
We'll work to open source the launcher source. In the meantime, the default logic looks like this:

workerCount := 8
if memoryMb <= 128 {
workerCount = 1
} else if memoryMb <= 256 {
workerCount = 2
} else if memoryMb <= 512 {
workerCount = 4
}

The number of workers to use is highly dependent on your workload, so these are only reasonable general starting points.


Joshua Smith

unread,
Nov 5, 2021, 4:48:31 PM11/5/21
to google-a...@googlegroups.com
I figured out from the logs (since it prints a message for each worker it starts) that 3.7 is starting 8 workers by default. So adding the entrypoint with defaults has no effect.

However, I did some experiments and I think I can characterize the difference between 2.7 and 3.7 when it comes to scaling.

When first landing on my app's page it shows a list of thumbnails. These come from the datastore, with each thumbnail making a HTTP request (GET /image?which=1, GET /image?which=2, etc.). If we:

1. Query for all of these at once so the server gets hammered with simultaneous requests:
- 2.7 starts one extra instance, and while that's happening the images are all served by the instance that already existed; the new instance never actually serves any of those image requests.
- 3.7 starts a lot of instances; it dumps a bunch of requests to each of them (one per worker thread, I presume), all of which are going to take 5 seconds because that's how long it takes each instance to start.

The net result is the user of the 2.7 app sees all the images load right away, while the 3.7 app gets them over the course of 15 seconds or so, and sometimes there are timeout errors.

2. Query for these with a delay that's long enough to avoid overlapping requests:
- Neither starts extra instances.

The user sees all the images load pretty quick, but not quite as quick as 2.7 with the first approach.

My conclusion is that the change is that the old scheduler wouldn't use an instance until it was running, whereas the new scheduler queues up requests for instances that are still starting up. The old scheduler doesn't start a new instance until the one it started most recently is running, whereas the new scheduler starts a new instance every time the request queue fills up for existing instances.

If you can serve a burst of requests in the time it take an instance to start, the old scheduler will get all those served. But the new scheduler will choke and perform very badly.

I'm not saying that the old scheduler was better. It just handled this particular case (cold start with a firehose) a lot better. Once the two systems are running at scale, I wouldn't expect there to be much difference.

-Joshua

Jason Collins

unread,
Nov 5, 2021, 5:09:23 PM11/5/21
to Google App Engine
The scheduler is the same.

This might have to do with `threadsafe` in app.yaml, which is complicated and confusing. I'd suggest removing `threadsafe` if it's present, then try setting the max_concurrent_requests to something higher (like 2x your number of workers): https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests


You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/pEi3hxX5YpE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5B04AE8B-06A4-43C3-9922-D9859BB4610E%40gmail.com.

Joshua Smith

unread,
Nov 5, 2021, 6:36:39 PM11/5/21
to google-a...@googlegroups.com
I don't have threadsafe in my 3.7 app's app.yaml. Just runtime and instance class. (I do have it in my 2.7 app, of course.)

I'll try playing with max concurrent threads, but if I bump it from 10 (the default) to 16, that really just changes it from 6 request to start an instance, to 9. Hard to see how that would make any difference.

As I said, the 2.7 only starts one more instance when I hammer it, and it doesn't end up giving it any work. That's very different behavior from the 3.7 app which starts a bunch of instances and spreads the work to all of them (resulting in all the requests taking at least 5 seconds to fulfill, since the instance takes 5s to start up). This is clearly visible in the logs I included in the first message.

You're positive the scheduler is the same?

-Joshua

Jason Collins

unread,
Nov 5, 2021, 6:40:13 PM11/5/21
to Google App Engine
Yes, it is the same scheduler (this is all on GAE Standard, right?) Of course, that scheduler has lots of knobs.

If you can build a repro, can you work with support? If anything, the concurrency of GAE Python 3 should be better than GAE Python 2.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/18074444-B887-45B4-9186-12620383E631%40gmail.com.

Joshua Smith

unread,
Nov 6, 2021, 10:12:37 AM11/6/21
to google-a...@googlegroups.com
I'll put it on my to-do list to make a mock application that just does this hammer-on-startup thing. That seems pretty easy. (And it'll be interesting to see if a trivial mock app takes 5s to start up, or if there's some import I'm doing that's weighing things down.)

-Joshua

Will Reiher

unread,
Nov 8, 2021, 12:38:40 PM11/8/21
to Google App Engine
I'm slowly moving my way towards a migration path to the new Python 3 runtime. 2022 is the year. I've been a AppEngine customer since 2009 - so I've already seen my share of smaller migrations as the oldest of the services died off. 

I currently have a question about Blobstore. As I'm planning to decouple services in preparation for the first of many migrations I'm wondering about all my old blobs. All my new blobstore files, as of summer of last year, are in the default cloud storage bucket and I'm keeping track of the "gs_object_name" so that they'll continue to be easy to work with as I migrate which is nice.

The real question is after I move my apps to the Python3 runtime how will I access my legacy blobs? I see that cloud NDB "supports" blobkeys and I currently I'd use "blobstore_handlers.BlobstoreDownloadHandler" and "send_blob" to allow my users to download their files. I literally have a decade of stored blobkeys that I would like to be able to service after I move to the new runtime. I see that "send_bob" puts the blobkey in a header which I assume is picked up by the platform to deliver the file to the end user. Please point me in the right direction if you can!

Amit Sinha

unread,
Nov 8, 2021, 4:55:41 PM11/8/21
to Google App Engine

Hi,

I believe you are referring to this for python 2. While taking a look into this , I have found two stack overflow posts recommending to use Cloud Storage here and here while using python 3. You can check if any of those fits into your use case.

Will Reiher

unread,
Nov 8, 2021, 7:53:27 PM11/8/21
to Google App Engine
Hi,

Thanks for the follow up. The articles detail the process fine if the underlying blobs are "on" cloud storage. In my case I'm talking about legacy blobs stored outside of the my visible buckets. They're not represented in cloud storage and the python 3 runtime does not seem to have a way to access them from what would be the remaining blobkey references. From my understanding my legacy blobs would not have a corresponding "gs_object_name" that would facilitate me accessing them. That is also very much moot on the new runtime since it doesn't provide the BlobInfo api where I would be able to access the gs_object_name property.

I have a feeling the new "appengine-python-standard" library will provide a partial answer as it looks like it will expose some of the old Blobstore APIs that I would need to continue to access legacy blobs that are not listed in an assessable bucket. 

The new library still seems to be missing the BlobstoreDownloadHandler that would allow the end user to be redirected to download the file. This basically set a header and the platform would deliver the blob to the client.

Kaan

unread,
Nov 9, 2021, 1:47:34 AM11/9/21
to Google App Engine
Honestly this is all very exciting, and like you've also pinpointed the main issue, we were (I was!) blind to the most important of changes, there's so much segmentation now that what's actually important goes missing in all the noise


Honestly I haven't tried the new to me dev app server yet, but I'm finally in theory ready to go from Python2 to Python3! But as Will Reiher pointed out, for a more complex project, there are still icky details like the blobstore, images api etc. - so for "legacy legacy" projects, I'd not attempt it and sunset them instead

And a plug: Python in my opinion is dying, while JS / NodeJS is thriving, consider bringing emulators into the JS ecosystem too and also please invest some effort into code/api design, it seems to me that someone is just translating libraries from Python to JS and releasing it as it is

Walt Pang

unread,
Nov 9, 2021, 1:02:56 PM11/9/21
to google-a...@googlegroups.com
In data science industry most people are still using python as development tool.

Kaan <kaan...@gmail.com>于2021年11月9日 周二下午2:47写道：
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

Alex Martelli

unread,
Nov 9, 2021, 1:36:20 PM11/9/21
to google-a...@googlegroups.com
On Mon, Nov 8, 2021 at 10:47 PM Kaan <kaan...@gmail.com> wrote:
   ...
And a plug: Python in my opinion is dying, while JS / NodeJS is thriving

According to TIOBE, in October Python was language #3 in terms of popularity (behind Java and C); in November, it vaulted to first place, "end[ing] C and Java's 20-year reign atop the TIOBE index" as TechRepublic titled their article -- "just one more feather in Python's ever-plumier hat", as their subtitle has it, and "Python's latest win is just another in its long line of toppling long-standing language leaders", as they write in the body of the article (Python is also the most popular programming language according to the IEEE study, the most "wanted" language according to StackOverflow's survey, the "AI lingua franca" according to datanami, and so forth). Javascript is #7 on TIOBE's list and #5 on the IEEE's, by the way.

Alex

Kaan

unread,
Nov 10, 2021, 6:34:36 AM11/10/21
to Google App Engine
Numbers aren't supporting me as well :D https://trends.google.com/trends/explore?date=all&q=python,javascript - somehow JS is dying while Python is thriving

But my point is, I loved App Engine because it was agile and practical, with JS and the new unleashed services, the experience is just incredible, apart from the regular flexibility, no more python platform-specific bugs and quirks to constantly chase, you have npm and your package.json and it just works anywhere!

However the new services aren't designed and documented as well, a concrete example is NodeJS datastore and the "Already Exists" exception, in the docs that I've read a while ago, there was no exception handling, it's not that hard to manually reproduce this exception and handle it, but still, it would be nice to have all the exception codes in one place, as the new JS library isn't really transactional like ndb was, it doesn't auto retry etc. - so most of the implementation is up to the user now - and after experiencing the Python2/NDB era and that masterpiece level design, the new JS counterparts just make you want to cry

Matt Larkin

unread,
Nov 10, 2021, 6:59:08 AM11/10/21
to Google App Engine
>The new library still seems to be missing the BlobstoreDownloadHandler that would allow the end user to be redirected to download the file. This basically set a header and the platform would deliver the blob to the client.

Yes, the Public Preview of the App Engine bundled services currently does not have the handlers implementation (for Blobstore, Mail, or Deferred).  Those handlers relied on the webapp framework, so we had to update them to allow use with idiomatic web frameworks (e.g., Flask, Django, etc.).  We are actually getting ready to launch the Blobstore & Mail handlers into the Private Preview (deferred is already available), so if you are interested in testing, please fill out this registration.  Once we have validated the updated design, we will be launching support for the handlers into Public Preview, so the more testing/feedback we get in Private Preview, the faster that can happen.

Sajal Gupta

unread,
Nov 30, 2021, 9:05:28 AM11/30/21
to Google App Engine
Hi,

I was trying to run the now bundled memcache service on localhost (python3/flask) but got an error: Attempted RPC call without active security ticket.

Could you shed some light on this and if localhost is not supported as of now and if so, by when will this support be available.

Thanks

Sajal

Matt Larkin

unread,
Nov 30, 2021, 11:21:42 AM11/30/21
to google-a...@googlegroups.com
Hi Sajal,
Are you using dev_appserver to run locally, or are you using some other type of toolset (e.g., the Flask dev server)?

Best,
Matt

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/pEi3hxX5YpE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/8519af42-dafe-4b44-968b-60c71917422cn%40googlegroups.com.

Sajal Gupta

unread,
Dec 1, 2021, 1:13:09 AM12/1/21
to Google App Engine

I am using Flask's server as follows:

if __name__ == '__main__':
# # This is used when running locally only. When deploying to Google App
# # Engine, a webserver process such as Gunicorn will serve the app. You
# # can configure startup instructions by adding `entrypoint` to app.yaml.
 app.run(host='127.0.0.1', port=8080, debug=True)

And in Pycharm, have set up environment parameter as follows:
FLASK_ENV=development

Also tried by not having this variable but end result is still the same.

dev_appserver doesn't seem to work anyway - gives an assertion error (expects python 2, I believe) while my project is Python 3 - in fact trying to migrate from 2 to 3.

Happy to share any additional information which can help debug this.

Regards

Sajal Gupta

unread,
Dec 1, 2021, 4:38:21 AM12/1/21
to Google App Engine
Am able to make this work now using dev_appserver. It doesn't work with flask server (at least I cold not make it work using flask server as above).

If that is indeed the case, may be the GAE getting started guide for python 3 page needs to change to reflect this fact that memcache etc. will work only thru dev_appserver - at least as of now.

Regards

Matt Larkin

unread,
Dec 1, 2021, 12:35:37 PM12/1/21
to google-a...@googlegroups.com
Thanks.  I'm following-up with the engineering team to determine if this is expected (meaning the docs need to be updated) or a bug.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/8abd1488-60b8-443d-81ff-dcd037428b05n%40googlegroups.com.

Matt Larkin

unread,
Dec 1, 2021, 12:42:16 PM12/1/21
to google-a...@googlegroups.com
>dev_appserver doesn't seem to work anyway - gives an assertion error (expects python 2, I believe) while my project is Python 3 - in fact trying to migrate from 2 to 3.
Sorry, missed this.  Without any more details, my guess is you need to provide the `--runtime_python_path` flag (docs).  If you do not, dev_appserver looks for a Py2 interpreter in order to run (that's the default to remain consistent with legacy behavior).  If that doesn't work, please provide more details (e.g. full error message).

Sajal Gupta

unread,
Dec 1, 2021, 11:59:43 PM12/1/21
to google-a...@googlegroups.com
Hi Matt,

Yes, earlier I was trying to make dev_appserver work (along with this flag) from within PyCharm. That did not work if the base python is set up to be Python 3 in PyCharm project.

It worked through command line.

It also started to work when I changed the base python to python 2 in pycharm project setup.

Now the only thing that does not work is PyCharm debugger along with dev_appserver with this runtime_python_path flag. I am assuming it is due to some conflict between python 2 and python 3 libs somehow getting mixed up.

Even if I run dev_appserver thru console/command line and then try to attach PyCharm debugger with this task, it doesn’t work.

It will help if there is a way to make it work and also is well documented.

Regards

Sajal

wesley chun

unread,
Jan 8, 2022, 3:36:58 PM1/8/22
to google-a...@googlegroups.com
Reviving this for a bit: @Joshua: are you able to try out Python 3.8 (or 3.9) vs. 3.7 on App Engine? Several performance enhancements went into 3.8, and I'm curious whether that helps with what you've been experiencing. Also any updates from your side?

Cheers,
--Wesley

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"A computer never does what you want... only what you tell it."
    wesley chun :: @wescpy :: Software Architect & Engineer
    Developer Advocate at Google by day; at night: Core Python


To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/D1F2E824-AE53-45B0-BD6E-2B582309F6A3%40gmail.com.
Reply all
Reply to author
Forward
0 new messages