Python 3 local development: fast startup or static assets, choose one?

[Ryan B]

Hi all! My Python 3 topic for the day is local development. How are you all doing it? Any tips?

The two main ways seem to be a third party server, eg Flask or Gunicorn, or dev_appserver.py. I have both working, but each one has a fatal flaw that makes it basically unusable for me so far.

When I run my app with eg gunicorn -b :8080 app:application - the way Google recommends - it works, but doesn't serve any assets from static file handlers in my app.yaml. This is widely reported on StackOverflow and elsewhere. Responses are often some variation of, "why do you need static assets for dynamic requests?" The obvious reason is that pretty much all webapps use images, CSS, and/or JS, among other static files. They're generally not usable without those assets.

When I run my app with  dev_appserver.py, it creates a new virtualenv and installs all of my dependencies from requirements.txt. This is nice, but also extremely slow for any nontrivial app. My current medium sized app has 155 packages, and even when they're all cached locally, installing them takes around 60s. I often switch between different apps and other stacks, so I can't often keep dev_appserver running for hours or days on end to avoid this. (I've filed this feature request for dev_appserver to reuse an existing virtualenv.)

Let me know if you have any ideas!

[Andrew Gorcester]

Hi Ryan,

I would recommend trying to solve the first problem and serving static files locally.

At first glance I think the most straightforward ways to do that would be to either:

- Use the Flask development server instead of gunicorn and leverage Flask's integrated /static directory https://flask.palletsprojects.com/en/1.1.x/tutorial/static/

- If you prefer to use gunicorn or you need more serving options, add WhiteNoise middleware to your app; ideally, have your app detect whether it's running in production or in development mode and conditionally add WhiteNoise in development only: http://whitenoise.evans.io/en/stable/

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/eff3cf33-3a7b-4cc4-8b8c-2a46b2c56c45%40googlegroups.com.

[Ryan Barrett]

thanks for the reply!

On Fri, Nov 8, 2019 at 2:40 PM 'Andrew Gorcester' via Google App Engine <google-a...@googlegroups.com> wrote:

I would recommend trying to solve the first problem and serving static files locally.

understood, but using a different static file server is not great. it makes local development diverge significantly from production. local no longer uses or tests app.yaml's static file handlers, but prod does, so bugs will eventually pop up that only happen in production, and are thus harder to test for and debug.

for that reason, i'm actually going the other way and using dev_appserver right now, soi still have some confidence that my app will behave more or less the same in production that it does locally. the minute long delay on startup is definitely painful though.

You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/BJDE8y2KISM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CANtYgF%2BPRh3vU%3Dpp3708sRAFE8%2BYxC%3Dm%2BmC8oGVqHyLw3yKBhw%40mail.gmail.com.

--

https://snarfed.org/

[Andrew Gorcester]

That does make sense. If you could bridge the gap by using a single static file directory in app.yaml served with a single static_dir handler, that might work too. But I know some apps have very complex app.yaml defined static files, so I understand why you're looking at the dev_appserver.py option instead.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CA%2BcaGh-8rd4NVj2HmcS0t0PRAAJ%2BiWo3huzjXcbJSDxyJtsAPw%40mail.gmail.com.

[Ryan B]

The latest in the saga of using dev_appserver.py locally is that Python 3 ndb (aka google-cloud-ndb) doesn't support it. I've filed this bug report. Whee!

On Friday, November 8, 2019 at 3:35:40 PM UTC-8, Andrew Gorcester wrote:

That does make sense. If you could bridge the gap by using a single static file directory in app.yaml served with a single static_dir handler, that might work too. But I know some apps have very complex app.yaml defined static files, so I understand why you're looking at the dev_appserver.py option instead.

On Fri, Nov 8, 2019 at 3:33 PM Ryan Barrett <ry...@barrett.name> wrote:

thanks for the reply!

On Fri, Nov 8, 2019 at 2:40 PM 'Andrew Gorcester' via Google App Engine <google-a...@googlegroups.com> wrote:

I would recommend trying to solve the first problem and serving static files locally.

understood, but using a different static file server is not great. it makes local development diverge significantly from production. local no longer uses or tests app.yaml's static file handlers, but prod does, so bugs will eventually pop up that only happen in production, and are thus harder to test for and debug.

for that reason, i'm actually going the other way and using dev_appserver right now, soi still have some confidence that my app will behave more or less the same in production that it does locally. the minute long delay on startup is definitely painful though.

At first glance I think the most straightforward ways to do that would be to either:

- Use the Flask development server instead of gunicorn and leverage Flask's integrated /static directory https://flask.palletsprojects.com/en/1.1.x/tutorial/static/

- If you prefer to use gunicorn or you need more serving options, add WhiteNoise middleware to your app; ideally, have your app detect whether it's running in production or in development mode and conditionally add WhiteNoise in development only: http://whitenoise.evans.io/en/stable/

On Fri, Nov 8, 2019 at 2:21 PM Ryan B <goo...@ryanb.org> wrote:

Hi all! My Python 3 topic for the day is local development. How are you all doing it? Any tips?

The two main ways seem to be a third party server, eg Flask or Gunicorn, or dev_appserver.py. I have both working, but each one has a fatal flaw that makes it basically unusable for me so far.

When I run my app with eg gunicorn -b :8080 app:application - the way Google recommends - it works, but doesn't serve any assets from static file handlers in my app.yaml. This is widely reported on StackOverflow and elsewhere. Responses are often some variation of, "why do you need static assets for dynamic requests?" The obvious reason is that pretty much all webapps use images, CSS, and/or JS, among other static files. They're generally not usable without those assets.

When I run my app with  dev_appserver.py, it creates a new virtualenv and installs all of my dependencies from requirements.txt. This is nice, but also extremely slow for any nontrivial app. My current medium sized app has 155 packages, and even when they're all cached locally, installing them takes around 60s. I often switch between different apps and other stacks, so I can't often keep dev_appserver running for hours or days on end to avoid this. (I've filed this feature request for dev_appserver to reuse an existing virtualenv.)

Let me know if you have any ideas!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/eff3cf33-3a7b-4cc4-8b8c-2a46b2c56c45%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/BJDE8y2KISM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-a...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CANtYgF%2BPRh3vU%3Dpp3708sRAFE8%2BYxC%3Dm%2BmC8oGVqHyLw3yKBhw%40mail.gmail.com.

--

https://snarfed.org/

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

[Ryan B]

the patch below makes dev_appserver use an existing virtualenv instead of creating a new one from scratch each time. it applies cleanly against google cloud SDK 272.0.0, app-engine-python component 1.9.87. i'd happily add a command line flag and send a PR, but afaik the google cloud SDK isn't open source.

put it in a file named dev_appserver.virtualenv.diff, then apply it with:

patch /opt/homebrew-cask/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/instance_factory.py ~/dev_appserver.virtualenv.diff

that path is for Mac OS, for a google cloud SDK originally installed via homebrew. you may need to update it for your own environment. you'll also need to change 'local3' below in the patch to your own virtualenv's path, relative to the app root directory.

diff instance_factory.py.orig instance_factory.py
105,106c105
<     if os.path.exists(venv_dir):
<       shutil.rmtree(venv_dir)
--
>     pass
120,121c119,124
<     self._CleanUpVenv(self._venv_dir)
<     self._venv_dir = tempfile.mkdtemp()
--
>     self._venv_dir = os.path.join(os.path.dirname(self._module_configuration.config_path), 'local3')
>     self.venv_env_vars = {
>         'VIRTUAL_ENV': self._venv_dir,
>         'PATH': ':'.join([os.path.join(self._venv_dir, 'bin'), os.environ['PATH']])
>     }
>     return

i'd hoped to write a wrapper script with a monkey patch instead, so i wouldn't have to reapply the patch every time i upgrade gcloud, but dev_appserver goes through a couple layers of import bootstraps at startup, and it wasn't easy to inject the monkey patch at just the right time. ah well. fingers crossed for the feature request to add this officially.

[Remko Tronçon]

Hi Ryan,

 

understood, but using a different static file server is not great. it makes local development diverge significantly from production. local no longer uses or tests app.yaml's static file handlers, but prod does, so bugs will eventually pop up that only happen in production, and are thus harder to test for and debug.

The way I solve this is to have a little piece of middleware (only in dev mode) that loads app.yaml and serves the static_* handlers statically. The basic logic (I might not be handling all cover cases, but it works for me) is pretty simple, and the entire middleware class takes about 40 lines of code.

thanks,

Remko

[Ruby Tokantekai]

How can i download spreadsheet apis to my group.

[Katayoon (Cloud Platform Support)]

Hi Ruby,

It seems your question is not related to this thread and Google App Engine, and is about Google Sheets API. If so, I recommend to reach out to the Cloud Sheets API support. 

[Ryan B]

Resurrecting this year and a half old thread to plug https://github.com/XeoN-GHMB/app_server , a little dev_appserver replacement that wraps your existing local app server and adds static file serving that supports app.yaml static_* handlers. Sounds similar to the 40 line middleware you mentioned.

I was finally forced to migrate away from dev_appserver due to the continued degrading of the Python 2 ecosystem. Specifically, when I migrated to macOS Monterey, dev_appserver failed to start and complained, Cannot use the Cloud Datastore Emulator because the packaged grpcio is incompatible to this system. Please install grpcio using pip. I tried to install grpcio for Python 2, but Montery doesn't include Python 2's pip, and easy_install couldn't install it. app_server to the rescue!

[Ryan B]

I revisited this again recently and wrote a drop-in Flask extension that serves app.yaml static file and directory handlers: flask-gae-static. Feedback is welcome!

[George (Cloud Platform Support)]

It does look as if this is a coding issue, so it may be worth mentioning: this is a tracker for App Engine specific issues. When it comes strictly to programming and coding in Python, you’ll be at an advantage to rather post your questions on Stackoverflow, to gain this way access to a large number of experts; it is meant for providing you help with coding.