Upgrade multiple apps

[Gertjan Klein]

Hi,

I have a VPS that runs four WSGI applications of varying importance.
These run on Debian Buster, with the standard Debian versions of the
various components: Python 2.7.16, Apache/2.4.38 (Debian), mod_wsgi
4.6.5-1. The sites use Django and Flask.

Python 2.7 is nearing EOL, and I want to migrate the applications to
Python 3.7 (the standard Buster Python 3 version). I am seeking advice
on how to approach this.

I could upgrade all applications on my development environment, and then
do the server in one go. However, I would prefer to get rid of the
interdependency of these applications: they currently all need to use
the same mod_wsgi, and hence Python.

Two approaches I can think of to do that are Docker and, perhaps,
mod_wsgi express. I'd like to keep things as light-weight as possible
(these are tiny sites, with only a few users).

What would be the best way to go here?

Regards,
Gertjan.

[Graham Dumpleton]

Using mod_wsgi-express behind the original Apache acting as a proxy is one way. Just be warned that mod_wsgi-express is mostly intended for running a single WSGI application. That said, it should use less memory than your main Apache as the configuration it uses is designed to reduce memory over default Apache setup.

There is discussion about how to use mod_wsgi-express behind Apache as a proxy in:

* http://blog.dscpl.com.au/2015/06/proxying-to-python-web-application.html

* http://blog.dscpl.com.au/2015/07/redirection-problems-when-proxying-to.html

These talk about mod_wsgi-express running in a container, but you don't need to and discussion is still relevant when you don't. You would instead use the `mod_wsgi-express setup-server` command to generate configuration up front and then integrate invocation of 'apachectl' it generates into your system startup system.

Graham

[Gertjan Klein]

Gertjan:

>> Python 2.7 is nearing EOL, and I want to migrate the applications to
>> Python 3.7 (the standard Buster Python 3 version). I am seeking advice
>> on how to approach this.

[...]

Graham:

> Using mod_wsgi-express behind the original Apache acting as a proxy is
> one way. Just be warned that mod_wsgi-express is mostly intended for
> running a single WSGI application. That said, it should use less memory
> than your main Apache as the configuration it uses is designed to reduce
> memory over default Apache setup.

If I'm not mistaken, mod_wsgi_express runs in a virtualenv? My apps do
as well, so that would work. I would like to use a separate
mod_wsgi_express for each application, though, so memory savings I don't
see (but are not the goal). Thanks for the links (snipped), it looks
very useful!

Graham:

> These talk about mod_wsgi-express running in a container, but you don't
> need to and discussion is still relevant when you don't. You would
> instead use the `mod_wsgi-express setup-server` command to generate
> configuration up front and then integrate invocation of 'apachectl' it
> generates into your system startup system.

I am not opposed to using containers. I noticed the containers you
provide stop at Python 3.5. Is there any particular reason for that?

Thanks for your input, regards,
Gertjan.

[Graham Dumpleton]

> On 6 Jun 2019, at 4:20 am, Gertjan Klein <gkl...@xs4all.nl> wrote:
>
> Gertjan:
>>> Python 2.7 is nearing EOL, and I want to migrate the applications to Python 3.7 (the standard Buster Python 3 version). I am seeking advice on how to approach this.
> [...]
>
> Graham:
>> Using mod_wsgi-express behind the original Apache acting as a proxy is one way. Just be warned that mod_wsgi-express is mostly intended for running a single WSGI application. That said, it should use less memory than your main Apache as the configuration it uses is designed to reduce memory over default Apache setup.
>
> If I'm not mistaken, mod_wsgi_express runs in a virtualenv?

If you are using virtual environments, yes, it would be installed into that. Each virtual environment would have its own copy if you had more than one, allowing different Python versions to be used. A separate instance of it would be run for each application in respective virtual environments.

> My apps do as well, so that would work. I would like to use a separate mod_wsgi_express for each application, though, so memory savings I don't see (but are not the goal). Thanks for the links (snipped), it looks very useful!

When you use a single Apache instance to host multiple WSGI applications, you would either being using separate daemon process groups for each, or separate sub interpreters of a single process. In either case, Python code and modules in larger part is not shared. Each application also still has its own separate memory footprint.

That combined with a typical Apache installation not being setup very well for Python applications, means that total memory usage may not be much different, especially if using frameworks like Django. In fact, the better configuration of mod_wsgi-express may make it better overall if you weren't using daemon mode, as you can tune further each mod_wsgi-express instance to needs of a specific application. This means if you have one application that needs a multi process configuration, you aren't then bloating out size used by others unnecessarily. Memory use would be least of my concerns.

>
> Graham:
>> These talk about mod_wsgi-express running in a container, but you don't need to and discussion is still relevant when you don't. You would instead use the `mod_wsgi-express setup-server` command to generate configuration up front and then integrate invocation of 'apachectl' it generates into your system startup system.
>
> I am not opposed to using containers. I noticed the containers you provide stop at Python 3.5. Is there any particular reason for that?

There wasn't any real interest in the community for having curated container images for Python web application hosting that used best practices. Everyone wanted to build their own from scratch. After all, docker was a shiny new tool and was silly to think you could deprive people of their opportunity to waste lots of time learning it, making lots of mistakes, but still come up with only a half baked solution that sort of worked. ;-)

>
> Thanks for your input, regards,
> Gertjan.
>

> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
> To post to this group, send email to mod...@googlegroups.com.
> Visit this group at https://groups.google.com/group/modwsgi.
> To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/qd9158%245hqt%241%40blaine.gmane.org.
> For more options, visit https://groups.google.com/d/optout.

[Gertjan Klein]

Gertjan:>> If I'm not mistaken, mod_wsgi_express runs in a virtualenv?

Graham:

> If you are using virtual environments, yes, it would be installed into that. Each virtual environment would have its own copy if you had more than one, allowing different Python versions to be used. A separate instance of it would be run for each application in respective virtual environments.

Apologies, I see now that I did not mention this in my original post.
Yes, I use virtual environments for all applications. (Two use Django,
two use Flask.) Do you see any disadvantages in such a setup?

Graham:> That combined with a typical Apache installation not being

setup very well for Python applications, means that total memory usage
may not be much different, especially if using frameworks like Django.
In fact, the better configuration of mod_wsgi-express may make it better
overall if you weren't using daemon mode, as you can tune further each
mod_wsgi-express instance to needs of a specific application. This means
if you have one application that needs a multi process configuration,
you aren't then bloating out size used by others unnecessarily. Memory
use would be least of my concerns.

I'm not sure what you're saying here. The setup I thought you were
suggesting was one mod_wsgi (express) for each application, all of them
behind the "system" Apache functioning as a proxy. I don't know if I am
currently us "daemon mode", Apache setup is a black art to me. I have
currently made no special provisions for the applications (that I can
think of), other than specify WSGIScriptAlias. I did do (I see now) some
configuration in wsgi.conf: WSGIDaemonProcess and WSGIProcessGroup.

I don't expect tuning to be necessary, but it would be nice, if I felt
the urge to try some, that the results be contained to one site, and not
influence all of them like now. It seems the mod_wsgi per virtualenv is
ideal for that.

Graham:

> There wasn't any real interest in the community for having curated container images for Python web application hosting that used best practices. Everyone wanted to build their own from scratch. After all, docker was a shiny new tool and was silly to think you could deprive people of their opportunity to waste lots of time learning it, making lots of mistakes, but still come up with only a half baked solution that sort of worked. ;-)

I've had too many occasions of not understanding Apache and/or mod_wsgi
configuration not to appreciate having that done right for me. :-) It is
unfortunate that the images didn't get more interest.

I would like to move to the latest available Python version. Docker was
certainly an option for me, but it seems that it would require perhaps
even more tinkering than virtualenv+mod_wsgi-express. So currently that
is my favorite option. (I won't have time to actually try something
until the weekend.)

Thanks again, regards,
Gertjan.

[Graham Dumpleton]

> On 7 Jun 2019, at 5:16 am, Gertjan Klein <gkl...@xs4all.nl> wrote:
>
> Gertjan:>> If I'm not mistaken, mod_wsgi_express runs in a virtualenv?
>
> Graham:
>> If you are using virtual environments, yes, it would be installed into that. Each virtual environment would have its own copy if you had more than one, allowing different Python versions to be used. A separate instance of it would be run for each application in respective virtual environments.
>
> Apologies, I see now that I did not mention this in my original post. Yes, I use virtual environments for all applications. (Two use Django, two use Flask.) Do you see any disadvantages in such a setup?
>
> Graham:> That combined with a typical Apache installation not being setup very well for Python applications, means that total memory usage may not be much different, especially if using frameworks like Django. In fact, the better configuration of mod_wsgi-express may make it better overall if you weren't using daemon mode, as you can tune further each mod_wsgi-express instance to needs of a specific application. This means if you have one application that needs a multi process configuration, you aren't then bloating out size used by others unnecessarily. Memory use would be least of my concerns.
>
> I'm not sure what you're saying here. The setup I thought you were suggesting was one mod_wsgi (express) for each application, all of them behind the "system" Apache functioning as a proxy.

And I am. When you run mod_wsgi-express, you are still running Apache with mod_wsgi, you are just running a separate instance to the main system Apache, with a configuration generated by mod_wsgi-express. So comparing memory usage from the mod_wsgi-express instances to running similar with system Apache only.

> I don't know if I am currently us "daemon mode", Apache setup is a black art to me. I have currently made no special provisions for the applications (that I can think of), other than specify WSGIScriptAlias. I did do (I see now) some configuration in wsgi.conf: WSGIDaemonProcess and WSGIProcessGroup.

The WSGIDaemonProcess and WSGIProcessGroup directives are setting up daemon mode. But if not using WSGIRestrictEmbedded, and WSGIApplicationGroup %{GLOBAL} (in context of each daemon process group), plus various other Apache settings to control its memory usage, then the Apache instance would be using more memory than it needs to. When you use mod_wsgi-express, the generated Apache configuration it uses contains all the tricks to reduce memory usage and why saying it could well end up using less.

> I don't expect tuning to be necessary, but it would be nice, if I felt the urge to try some, that the results be contained to one site, and not influence all of them like now. It seems the mod_wsgi per virtualenv is ideal for that.
>
> Graham:
>> There wasn't any real interest in the community for having curated container images for Python web application hosting that used best practices. Everyone wanted to build their own from scratch. After all, docker was a shiny new tool and was silly to think you could deprive people of their opportunity to waste lots of time learning it, making lots of mistakes, but still come up with only a half baked solution that sort of worked. ;-)
>
> I've had too many occasions of not understanding Apache and/or mod_wsgi configuration not to appreciate having that done right for me. :-) It is unfortunate that the images didn't get more interest.
>
> I would like to move to the latest available Python version. Docker was certainly an option for me, but it seems that it would require perhaps even more tinkering than virtualenv+mod_wsgi-express. So currently that is my favorite option. (I won't have time to actually try something until the weekend.)
>
> Thanks again, regards,
> Gertjan.
>
>
>
>
>

> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
> To post to this group, send email to mod...@googlegroups.com.
> Visit this group at https://groups.google.com/group/modwsgi.

> To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/qdbor5%246dul%241%40blaine.gmane.org.

[Gertjan Klein]

[Graham]

> When you run mod_wsgi-express, you are still running Apache with mod_wsgi, you are just running a separate instance to the main system Apache, with a configuration generated by mod_wsgi-express. So comparing memory usage from the mod_wsgi-express instances to running similar with system Apache only.

Great. I'm finally beginning to understand roughly how this works. It
runs an instance of the system Apache, but with a compiled-for-it
mod_wsgi, and a configuration tuned for mod_wsgi. (Correct?)

Is it correct that the only documentation for mod_wsgi-express is on the
PyPI page?

I now have a test copy of one of the sites that uses a virtual
environment with mod_wsgi-express. It works, including the proxy!
(Fortunately, the site itself needed zero changes from Python 2.7 to
3.7.) There are a few things I'm wondering about, though:

* In my previous setup, I used Alias statements in the (main) Apache
configuration for my static files. It appears these don't work in
combination with the ProxyPass statement. I would like to keep using
these, though. What I've done now is add an Include to the bottom of the
generated httpd.conf, including a file that has these statements. Is
this an appropriate way to do it?

* The PyPI page says to add 'mod_wsgi.server' to the installed apps. I
didn't do that, yet the site works. Is this instruction merely to add
the manage.py commands? Is there any reason I should prefer these over
the generated apachectl that I'm using now?

Thanks again for your help, regards,
Gertjan.

[Graham Dumpleton]

On 10 Jun 2019, at 5:43 am, Gertjan Klein <gkl...@xs4all.nl> wrote:

[Graham]

When you run mod_wsgi-express, you are still running Apache with mod_wsgi, you are just running a separate instance to the main system Apache, with a configuration generated by mod_wsgi-express. So comparing memory usage from the mod_wsgi-express instances to running similar with system Apache only.

Great. I'm finally beginning to understand roughly how this works. It runs an instance of the system Apache, but with a compiled-for-it mod_wsgi, and a configuration tuned for mod_wsgi. (Correct?)

Is it correct that the only documentation for mod_wsgi-express is on the PyPI page?

And various blog posts own my site, plus ensure you run it as:

    mod_wsgi-express start-server --help

I now have a test copy of one of the sites that uses a virtual environment with mod_wsgi-express. It works, including the proxy! (Fortunately, the site itself needed zero changes from Python 2.7 to 3.7.) There are a few things I'm wondering about, though:

* In my previous setup, I used Alias statements in the (main) Apache configuration for my static files. It appears these don't work in combination with the ProxyPass statement.

Depends on how you set up Proxy statements and the order you have directives in the Apache configuration file. Impossible to comment without seeing what you are using. Can you show what you are using?

I would like to keep using these, though. What I've done now is add an Include to the bottom of the generated httpd.conf, including a file that has these statements. Is this an appropriate way to do it?

The alternative if can't work out what is wrong with the Apache config for proxy, is to have mod_wsgi-express host static files by using --url-alias option.

* The PyPI page says to add 'mod_wsgi.server' to the installed apps. I didn't do that, yet the site works. Is this instruction merely to add the manage.py commands? Is there any reason I should prefer these over the generated apachectl that I'm using now?

When using as manage.py, it is still generating the Apache configuration. The difference is that by using runmodwsgi management command, it will automatically set up mod_wsgi-express to host your Django static media files and work out where the Django WSGI application entry point is.

See:

http://blog.dscpl.com.au/2015/04/using-modwsgi-express-with-django.html

http://blog.dscpl.com.au/2015/04/integrating-modwsgi-express-as-django.html

Thanks again for your help, regards,
Gertjan.

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/qdl8o2%246qa7%241%40blaine.gmane.org.

[Gertjan Klein]

Graham:

>> On 10 Jun 2019, at 5:43 am, Gertjan Klein <gkl...@xs4all.nl

>> <mailto:gkl...@xs4all.nl>> wrote:
[...]

>> * In my previous setup, I used Alias statements in the (main) Apache
>> configuration for my static files. It appears these don't work in
>> combination with the ProxyPass statement.
>
> Depends on how you set up Proxy statements and the order you have
> directives in the Apache configuration file. Impossible to comment
> without seeing what you are using. Can you show what you are using?

I don't have that configuration anymore. However, I've studied the help
generated with --help, experimented a bit, and find I don't actually
need the aliases anymore. I now run (in the base directory for the
application, with the virtual environment activated):

mod_wsgi-express setup-server daglengte/wsgi.py --port=8000 \
--user gklein --group www-data \
--server-root=../apache --http2 --host localhost --setup-only \
--document-root ../../www \
--url-alias /static static \
--include-file ../apache_wsgi.conf \
--trust-proxy-header X-Forwarded-Host \
--trust-proxy-header X-Forwarded-Port \
--trust-proxy-header X-Forwarded-For \
--trust-proxy-header X-Forwarded-Scheme

This appears to do exactly what I want. Static files from the Django
application are handled by the --url-alias. Files like robots.txt,
lets-encrypt challenges etc. are handled by --document-root. All
automatically setup with the proper Apache configuration. I still have
the --include-file in there, but it's actually empty. :) (I am quite
impressed with how easy all this actually is with mod_wsgi-express!)

The virtual host configuration of the main Apache now handles, in
essence, (common) logging and SSL. The proxying does the rest.

I have one last task left for this application: integrate into the
system startup sequence. I have done something similar before, I'll be
able to figure that out. Then it's on to the other sites!

Thanks again, regards,
Gertjan.