Running mod_wsgi under docker.

[Graham Dumpleton]

If anyone is a user of docker and/or has already been playing around with mod_wsgi-express, you might like to monitor my latest little project.

https://registry.hub.docker.com/u/grahamdumpleton/mod-wsgi-docker/

Hard to see how one could make it simpler.

Graham

[Michael Blake]

Works flawlessly.  I added a few pre-build commands, and my Flask app starts right up on port 80 of the container, and it runs... fast!  Definitely production grade (at least for my purposes).

The next step I need is to setup TLS on the site, and a few redirects.  I'm poking around in the .whiskey/apache area to see where to add in the config, but it's not immediately apparent where, or correct way to add the necessary SSLCertificate / rewrite directives.  I checked over the docs and was looking for more about the .whiskey folder, and how that is setup.  

Any help is appreciated, thanks!

Mike

[Jason Garber]

I could be completely off base, but if you can run nginx in that container it solves 3 important issues:

1. Slow client buffering
2. TLS is super easy
3. Static asset serving is blazing fast

I can provide configs if desired...

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Graham Dumpleton]

Jason

The whole point of me providing a docker image is to make it a 'best of breed' solution so people do not have to muck around having to configure everything themselves.

It uses mod_wsgi-express and so people do not need to worry about server configuration themselves. By suggesting using nginx in the same container when the pre configured setup will be more than adequate for 95+% of people sort of defeats the whole point of why the image was created.

As to TLS, mod_wsgi-express already makes that easy.

Static file serving is also more than adequate and mod_wsgi-express provides special configuration to allow special layering capabilities with the way that static file handling is done to make that easy as well.

Finally, in a docker world, embedding nginx in the same container sort of goes against the philosophy of how you use containers. The approach one would use if there was some need to offload some work to nginx, especially if using it for load balancing, would be to have the nginx running in a different container which was linked to the containers running the Python web application.

Anyway, I will followup with the TLS options and how to set it up in a while after have dumped kids at school.

Graham

[Graham Dumpleton]

On 11/05/2015, at 5:53 AM, Michael Blake <ubi...@gmail.com> wrote:

> Works flawlessly. I added a few pre-build commands, and my Flask app starts right up on port 80 of the container, and it runs... fast! Definitely production grade (at least for my purposes).
>
> The next step I need is to setup TLS on the site, and a few redirects. I'm poking around in the .whiskey/apache area to see where to add in the config, but it's not immediately apparent where, or correct way to add the necessary SSLCertificate / rewrite directives. I checked over the docs and was looking for more about the .whiskey folder, and how that is setup.

Where you currently might have a Dockerfile of:

FROM grahamdumpleton/mod-wsgi-docker:python-2.7-onbuild
CMD [ "hello.wsgi" ]

You can add any additional mod_wsgi-express options into 'CMD'.

So as far as production goes, depending on your specific requirements, you may want to adjust the number of processes/threads used from the default of one process and five threads. Thus:

FROM grahamdumpleton/mod-wsgi-docker:python-2.7-onbuild
CMD [ "--processes", "3", "--threads", "3", "hello.wsgi" ]

For setting up HTTPS as well as HTTP, similarly use the appropriate options and just add them to the Dockerfile, plus expose port 443, which is default that mod_wsgi-express will use for HTTPS port.

Presuming you have added the certificate and key file into the top level directory of the project that got copied into the Docker image as 'server.crt' and 'server.key', and that the FQDN for the host name used in the SSL certificate was 'www.example.com', you would use:

FROM grahamdumpleton/mod-wsgi-docker:python-2.7-onbuild
EXPOSE 443
CMD [ "--enable-https", "--server-name", "www.example.com", "--ssl-certificate", "server", "hello.wsgi" ]

If you wanted only HTTPS and not HTTP, then you can also add "--https-only". This will result in access to HTTP URLs being redirected automatically to HTTPS URL.

If you wanted to set up a HSTS policy to make browsers only use HTTPS, then you can use "--hsts-policy" with the policy argument.

Because "--server-name" is used to satisfy host name requirements for SSL certificate, then it must be accessed as that host name else will fail.

If you have a wildcard SSL certificate, then you can use "--server-alias" with pattern for other matching host names that should be accepted.

Probably makes no sense in Docker case, but if doing this on normal host OS and wanted to be able to still access using HTTPS by 'localhost', you can also use "--allow-localhost". Because this isn't going to match name in SSL certificate, then you will get browser complaining about SSL certificate not matching which would have to tell browser to ignore.

If you were after other typical HTTPS/HTTP behaviour beyond that then let me know what you are after. There is also support for client certificates and forcing SSL certificate information into the WSGI environment.

Graham

[Graham Dumpleton]

As for the redirects, which forgot to mention, for minor additional Apache configuration, you can create in the top level directory of your project a 'server.conf' file for Apache configuration snippets. You would then use the option "--include-file" with "server.conf' as argument.

Anything in there would get added at the end of the Apache configuration file which is generated, so depends a bit on what you want to do as to whether that will work. This is because this will sit outside of the VirtualHost definitions. In general this shouldn't be an issue as Apache will fall through to the definitions setup outside of scope of VirtualHost anyway.

So for example, if wanted to block access to a particular part of a site to certain client IPs, could add into "server.conf":

<Location /admin>
Require ip 192.168.2.0/24
</Location>

As to redirects using mod_rewrite, depends on what you are wanting to do. The mod_rewrite module will always be loaded at least as mod_wsgi-express relies on it for certain things.

Graham

[Jason Garber]

Thanks for the clarification Graham. Makes sense.

[Graham Dumpleton]

On 04/06/2015, at 6:37 AM, Eibriel Inv <eibri...@gmail.com> wrote:

Hi!! I'm trying to run a Flask aplication, but I'm stuck with the following error:

ImportError: No module named 'server.wsgi'

Can I see the full traceback to see where this import is occurring from?

And can I see the full command for mod_wsgi-express you are using and indicate where you are running that relative to where your Django project is?

A description of how your files/directories are laid out would help to clarify.

Do I need to create the "server.wsgi"? (the django example don't have the "example.wsgi" invoked by Dockerfile) Where should I put it?

In saying 'server.wsgi' or 'example.wsgi', then 'server' or 'example' is meant for Django to be the name of the Django project.

That is, there should already be a file called:

    projectname/wsgi.py

along side of the settings.py file for the project.

Graham

I've tryed on several levels of the folder tree with no luck.

If I remove the following line:

"--application-type", "module", "example.wsgi"

I get the weird whiskey picture.

Docker version:

Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64

I'm also using docker-compose:

web:
    build: .
    ports:
     - "8000:80"
    links:
     - db
     
db:
    image: mongo:latest

thanks!

[Eibriel Inv]

Hi!
I've noticed that every time I make a change on the code I need to rebuild the container and the build script install the pip requirements (which is time consuming).

How can I install my own requirements from Dockerfile to prevent that?

Thanks!!

El miércoles, 26 de noviembre de 2014, 9:20:13 (UTC-2), Graham Dumpleton escribió:

[Graham Dumpleton]

On 16/06/2015, at 10:16 AM, Eibriel Inv <eibri...@gmail.com> wrote:

> Hi!
> I've noticed that every time I make a change on the code I need to rebuild the container and the build script install the pip requirements (which is time consuming).
>
> How can I install my own requirements from Dockerfile to prevent that?

Is the issue with this because you are trying to use the docker image during development and so are making fast turn around changes?

The reason you are seeing what you are seeing is because the 'onbuild' docker image is written as:

FROM grahamdumpleton/mod-wsgi-docker:python-2.7

WORKDIR /app

ONBUILD COPY . /app
ONBUILD RUN mod_wsgi-docker-build

EXPOSE 80
ENTRYPOINT [ "mod_wsgi-docker-start" ]

Because the requirements.txt file is a part of the directory that needs to be copied into the docker image, then any change to the project directory will result in the latter build phase being triggered. It is the build phase that pip is run from to install the requirements.

This sort of arrangement is quite typical for Python docker images.

If you weren't using this image with the mod_wsgi-docker-build script and doing things explicitly, you could probably get away with using something like:

WORKDIR /app

COPY requirements.txt /app/requirements.txt

RUN pip install -r requirements.txt

COPY . /app

With this, because the requirements.txt file is copied separately, so long as that file along isn't changed, the pip wouldn't be run.

The whole point with mod_wsgi-docker-build though is that it does other stuff before pip is run to properly set up the environment. This means it cannot be split out in that way.

If the problem is because you are trying to use it for quick turnaround changes in a development environment, there may be a better way, although I would have to tweak how the images work.

What one other person did, perhaps due to also wanting to do things in a development environment, was to copy my docker images and change them to have the application directory mounted in as a volume.

They possibly weren't thinking of the pip install problem, but wanted to be able to modify code on the fly, touch the WSGI script file and have the mod_wsgi in the docker instance automatically reload it.

In other words, wouldn't be necessary to stop the container.

Rather than copy my docker images and change them, notionally you could achieve the same thing by doing:

docker build -t myapp .
docker run -it -v `pwd`:/app myapp

That is, build the docker image once, which will result in all the packages being installed, and then run it, with it using the live code from your host. You would only need to do a rebuild if you change the requirements.txt file.

Unfortunately, as written this will not work.

The problem is that the docker images themselves stored various stuff inside of the /app directory and what is in it isn't just your application code. Thus the script to start everything up isn't found.

What I could do though is change the images such that the special stuff from the base image isn't stored under /app where your application is placed, and puts it somewhere else.

If I do that, then the mounting volume trick should work and allow for this sort of behaviour.

So can you explain the scenario of where this is a problem so I can understand better the needs you have and whether this change would work?

Thanks.

Graham

[Graham Dumpleton]

I am just doing a final build and push of updated images which will permit the volume mounting trick.

Besides that, there is one other thing one can do which is not to avoid running pip, but to use a manual step to first prebuild Python wheels for all the packages from inside Docker back into a local directory on the host. You then setup the build hook script to check for a cache directory of wheels and set an environment variable so pip knows where it is. When you want to build the image using the cache, you would mount as a volume the local cache directory with the wheels into the location which pip is being setup to grab them from.

By using Python wheels you would speed things up significantly as it avoids needing to download packages from the network and recompile them if C extensions.

I will work on a recipe for that, but if you can let me know the scenario you are targeting which is the problem do still let me know.

Graham

[Eibriel Inv]

Hi Graham, thanks for your response.

I want to use docker for Development to be sure I'm running under the same conditions that Production.
I'm using "onbuild" images, you can see the code here: https://github.com/Eibriel/widu-community-server/blob/master/Dockerfile

I've no problem rebuilding the container each time (avoiding Volumes if possible), if the build time is short enough.

I'm wondering, Why the virtual environment is needed? Without it we will be able to just pip install on Dockerfile.

Thanks!!

[Graham Dumpleton]

Unless I am confused by what you are asking, there isn't a virtual environment.

The images have a separate Python installation from any system Python installation so it can ensure is latest version and when you are doing a pip install it is installing into the separate Python installation, not a virtual environment constructed from the Python installation.

So the build scripts aren't setting up a virtual environment and that therefore isn't the issue. The issue is more that the build scripts support you being able to specify hooks to perform actions and set extra environment variables dynamically, which need to be done before the pip install can be run.

If you aren't using the build hooks, then technically you could do a pip install direct from the Docker file, however not when inheriting from the on build image. Instead you would want to copy the intent of the on build into your Dockerfile, actually dropping the on build and then insert your pip installs.

So instead use something like:

FROM grahamdumpleton/mod-wsgi-docker:python-2.7

WORKDIR /app

RUN pip install mypackage

COPY . /app

RUN mod_wsgi-docker-build

EXPOSE 80
ENTRYPOINT [ "mod_wsgi-docker-start" ]

Any particular reason volumes would be an issue?

FWIW, when I had some time I was going to play with wheel deploys was a variant of some ideas described in:

https://glyph.twistedmatrix.com/2015/03/docker-deploy-double-dutch.html

That is, use one docker container once to build all the wheels in and then the results would be copied into final image to speed up its build.

Graham

[Eibriel Inv]

Thanks!
This solution worked just fine for me, I wasn't using hooks.

I think Volumes should be for data instead of code (but maybe I'm over reacting).

my Dockerfile:

FROM grahamdumpleton/mod-wsgi-docker:python-3.4

WORKDIR /app

RUN pip install Flask
RUN pip install eve

COPY . /app

RUN mod_wsgi-docker-build

EXPOSE 80
ENTRYPOINT [ "mod_wsgi-docker-start" ]

CMD [ "--working-directory", "main_server", \
      "--url-alias", "/main_server/static", "static", \
      "main_server.wsgi" ]

Now the script skips the requirements because are already installed, is only installing modwsgi, and is fast enough to use it on development :)

[Eibriel Inv]

(modwsgi is only installed the first time)

El jueves, 18 de junio de 2015, 21:42:42 (UTC-3), Graham Dumpleton escribió:

[Graham Dumpleton]

The mod_wsgi package should be installed every time you build an image with that Dockerfile.

Anyway, I have pushed up new docker images which install mod_wsgi as part of the base image now so even that will not happen. So pull down latest images and you should see that change.

I am working on the recipe to explain the use of wheels for quicker builds. The volume only comes into play when building the wheels, not necessarily when building your application image, and definitely now when running it.

Environment variables for pip are giving me a bit of a problem right now, so trying to sort that out.

Graham

[Graham Dumpleton]

I have updated the Docker image again. I actually had to embed the knowledge of dealing with a wheel directory into the image as there is no way for a user to override environment variables during the build phase except by not using the onbuild image and creating a manual one again, but want to avoid that for various reasons.

With the very latest images what you can now do is.

1. Build the image.

$ time docker build -t myapp .Sending build context to Docker daemon   574 kB

Sending build context to Docker daemon

Step 0 : FROM grahamdumpleton/mod-wsgi-docker:python-2.7-onbuild

# Executing 2 build triggers

Trigger 0, COPY . /app

Step 0 : COPY . /app

Trigger 1, RUN mod_wsgi-docker-build

Step 0 : RUN mod_wsgi-docker-build

 ---> Running in fa3adb991c09

 -----> Installing dependencies with pip

Collecting Django<1.8 (from -r requirements.txt (line 1))

  Downloading Django-1.7.8-py2.py3-none-any.whl (7.4MB)

Installing collected packages: Django

Successfully installed Django-1.7.8

 -----> Running .whiskey/action_hooks/build

….

61 static files copied to '/app/example/htdocs'.

 ---> c72e831182e0

Removing intermediate container cc5bdce4213b

Removing intermediate container fa3adb991c09

Step 1 : CMD --working-directory example --url-alias /static example/htdocs --application-type module example.wsgi

 ---> Running in 0e70fe87ecc8

 ---> 5da3bdf7d5e1

Removing intermediate container 0e70fe87ecc8

Successfully built 5da3bdf7d5e1

real 0m15.185s

user 0m0.181s

sys 0m0.014s

So this is like what you would normally be doing now.

2. Run the image but don't start the WSGI server, just enter into the shell by setting the entry point. Using a special shell command here so pick up same environment as will exist when deploying. Also mount the current directory source as well so it actually overrides what is in the image. As it is a built image you can use 'pip freeze' to see that packages were installed.

$ docker run -it --rm --entrypoint mod_wsgi-docker-shell -v `pwd`:/app myapp

Django==1.7.8

mod-wsgi==4.4.13

virtualenv==13.0.3

wheel==0.24.0

3. We are now going to create wheels for the packages that we want installed each time. In this case what is in the requirements.txt file. We will store the wheels in the .whiskey/wheelhouse directory. You might have to create the .whiskey directory if it doesn't already exist.

root@095b6ad1ff25:/app# pip wheel --wheel-dir .whiskey/wheelhouse -r requirements.txt

Collecting Django<1.8 (from -r requirements.txt (line 1))

  Downloading Django-1.7.8-py2.py3-none-any.whl (7.4MB)

    100% |████████████████████████████████| 7.4MB 60kB/s

  Saved ./.whiskey/wheelhouse/Django-1.7.8-py2.py3-none-any.whl

Skipping Django, due to already being wheel.

root@095b6ad1ff25:/app# ls -las .whiskey/wheelhouse/

total 7252

   0 drwxr-xr-x 1 1000 staff     102 Jun 20 06:12 .

   0 drwxr-xr-x 1 1000 staff     136 Jun 20 06:12 ..

7252 -rw-r--r-- 1 1000 staff 7423353 Jun 20 06:12 Django-1.7.8-py2.py3-none-any.whl

root@095b6ad1ff25:/app# exit

Because we mounted the original source directory, this has actually written it back to your host. Once done we can exit.

From the host you can see that files are now present.

$ ls -las .whiskey/wheelhouse/

total 14504

    0 drwxr-xr-x  3 graham  staff      102 20 Jun 16:12 .

    0 drwxr-xr-x  4 graham  staff      136 20 Jun 16:12 ..

14504 -rw-r--r--  1 graham  staff  7423353 20 Jun 16:12 Django-1.7.8-py2.py3-none-any.whl

4. Build the image again. This time it the wheelhouse directory will be copied into the image and the wheel files there will be used to do pip installation.

$ time docker build -t myapp .

Sending build context to Docker daemon 7.998 MB

Sending build context to Docker daemon

Step 0 : FROM grahamdumpleton/mod-wsgi-docker:python-2.7-onbuild

# Executing 2 build triggers

Trigger 0, COPY . /app

Step 0 : COPY . /app

Trigger 1, RUN mod_wsgi-docker-build

Step 0 : RUN mod_wsgi-docker-build

 ---> Running in aed306a4db30

 -----> Detected wheelhouse for pip

 -----> Installing dependencies with pip

Collecting Django<1.8 (from -r requirements.txt (line 1))

Installing collected packages: Django

Successfully installed Django-1.7.8

 -----> Running .whiskey/action_hooks/build

….

61 static files copied to '/app/example/htdocs'.

 ---> 9f4681e0d152

Removing intermediate container aed306a4db30

Removing intermediate container 1f9033a7887e

Step 1 : CMD --working-directory example --url-alias /static example/htdocs --application-type module example.wsgi

 ---> Running in d56665f7d8be

 ---> 6d7ee644df22

Removing intermediate container d56665f7d8be

Successfully built 6d7ee644df22

real 0m9.264s

user 0m0.256s

sys 0m0.040s

For just Django, this time it is 6 seconds quicker. For packages which need code compilation, would be a lot quicker.

So the general idea is that keep doing builds and will use all the wheels. If you need to update wheels, change the requirements.txt file and run the shell again to generate the wheels for new versions and then builds will pick them up and make things quicker.

So the mounting of a volume is only done when you want to generate the wheels.

When wanting to build a final image for production release, you can remove the wheelhouse directory if want to force everything to be rebuilt from scratch. Also make sure wheelhouse directory is ignored in git repository so don't accidentally check it in to repository.

Graham

[Armazi]

Hello
Thanks a lot for this image. It saved me from a lot of time.
However, the static files of my Django app are not served and I don't know why.
Could you help me ?

[Graham Dumpleton]

Did you add a build action hook for running ‘python manage.py collectstatic’ to generate the static files?

If you did, is the ‘build’ file executable?

Is STATIC_ROOT set in the Django settings file.

Can you post the output from ‘docker build’ so can see what was run?

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.

Visit this group at https://groups.google.com/group/modwsgi.