WSGIDaemonProcess segfault inside docker Alpine linux

[Konstantin Suvorov]

Hello,

I try to pack WSGI application into docker container. I use Alpine linux.

Everything works until I enable daemon mode with WSGIDaemonProcess, in this case I get "Segmentation fault".

This is reproducible with the simplest setup possible.

Dockerfile:

FROM alpine:3.4

RUN apk add --no-cache apache2-mod-wsgi && mkdir -p /run/apache2 && mkdir -p /wsgi

ADD wsgi-test.conf /etc/apache2/conf.d/

ADD wsgi-test.wsgi /wsgi/

CMD ["/bin/sh"]

wsgi-test.conf:

Listen 8080

<VirtualHost *:8080>

  ServerName localhost

  WSGIScriptAlias /test /wsgi/wsgi-test.wsgi

# WSGIDaemonProcess example.com processes=2 threads=2

# WSGIProcessGroup example.com

  <Directory /wsgi>

  Require all granted

  </Directory>

</VirtualHost>

wsgi-test.wsgi:

def application(environ, start_response):

    status = '200 OK'

    output = 'Hello World!\n'

    response_headers = [('Content-type', 'text/plain'),

                        ('Content-Length', str(len(output)))]

    start_response(status, response_headers)

    return [output]

This works as expected – I run "httpd -DFOREGROUND" inside the container and can curl http://docker:8080/test for 'Hello World!'.

If I uncomment daemon options, the same command gives:

/ # httpd -DFOREGROUND

AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.5. Set the 'ServerName' directive globally to suppress this message

Segmentation fault

 Any hints how to fix this?

[Konstantin Suvorov]

I've tried mod_wsgi-express on a clean python:2.7-apline image.

Just made:

apk --no-cache add build-base bash
pip install mod_wsgi-httpd
pip install mod_wsgi
addgroup -S apache
adduser -S -G apache apache
mod_wsgi-express start-server /wsgi/wsgi-test.wsgi --user apache --group apache

Gives segfault with hello world application.

Some GDB stuff:

(gdb) r

Starting program: /bin/bash -c mod_wsgi-express\ start-server\ /wsgi/wsgi-test.wsgi\ --user\ apache\ --group\ apache

process 1667 is executing new program: /usr/local/bin/python2.7

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:0

Server Conf        : /tmp/mod_wsgi-localhost:8000:0/httpd.conf

Error Log File     : /tmp/mod_wsgi-localhost:8000:0/error_log (warn)

Request Capacity   : 5 (1 process * 5 threads)

Request Timeout    : 60 (seconds)

Startup Timeout    : 15 (seconds)

Queue Backlog      : 100 (connections)

Queue Timeout      : 45 (seconds)

Server Capacity    : 20 (event/worker), 20 (prefork)

Server Backlog     : 500 (connections)

Locale Setting     : en_US.UTF-8

process 1667 is executing new program: /bin/busybox

process 1667 is executing new program: /bin/bash

process 1667 is executing new program: /usr/local/lib/python2.7/site-packages/mod_wsgi_packages/httpd/bin/httpd

Program received signal SIGSEGV, Segmentation fault.

wsgi_add_daemon_process (cmd=0x7fffffffe800, mconfig=<optimized out>, args=0x55555583eefd "") at src/server/mod_wsgi.c:7569

7569 src/server/mod_wsgi.c: No such file or directory.

(gdb) bt

#0  wsgi_add_daemon_process (cmd=0x7fffffffe800, mconfig=<optimized out>, args=0x55555583eefd "") at src/server/mod_wsgi.c:7569

#1  0x00005555555a2776 in invoke_cmd (cmd=0x7ffff567d8d0 <wsgi_commands+80>, parms=parms@entry=0x7fffffffe800, mconfig=0x555555839e38,

    args=0x55555583ecc8 "localhost:8000    display-name='(wsgi:localhost:8000:0)'    home='/'    threads=5    maximum-requests=0    python-path=''    python-eggs='/tmp/mod_wsgi-localhost:8000:0/python-eggs'    lang='en_US.UTF"...) at config.c:879

#2  0x00005555555a4f1d in ap_walk_config_sub (section_vector=0x55555581e508, parms=0x7fffffffe800, current=0x55555583aa88) at config.c:1309

#3  ap_walk_config (current=0x55555583aa88, parms=parms@entry=0x7fffffffe800, section_vector=0x55555581e508) at config.c:1342

#4  0x00005555555a622d in ap_process_config_tree (s=<optimized out>, conftree=<optimized out>, p=0x5555557ec0a8, ptemp=<optimized out>)

    at config.c:2059

#5  0x0000555555582d5d in main (argc=10, argv=0x7fffffffe9b8) at main.c:739

This leads me to this:

        if (!strcmp(entry->name, name))

            return "Name duplicates previous WSGI daemon definition.";

gdb output from packaged apache and mod_wsgi suggest the same place:

Program received signal SIGSEGV, Segmentation fault.

0x00007ffff7dc1abd in strcmp () from /lib/ld-musl-x86_64.so.1

Any ideas?

[Graham Dumpleton]

The very minimal Apache configuration they supply is somehow resulting in Apache not being initialised properly in some way that mod_wsgi expects. Or more likely the changes I made have simply changed the memory layout enough that it works.

If I use:

Listen 8080

LogLevel warn

WSGIDaemonProcess example.com processes=2 threads=2

<VirtualHost *:8080>

  ServerName localhost

  WSGIScriptAlias /test /wsgi/wsgi-test.wsgi

 WSGIProcessGroup example.com

  <Directory /wsgi>

  Require all granted

  </Directory>

</VirtualHost>

then it works fine.

The two changes were, set LogLevel directive to have a value rather than rely on compiled in default.

Move the WSGIDaemonProcess directive outside of the VirtualHost.

As long as do both, it isn’t crashing.

I have suspected that there is a memory usage issue in mod_wsgi for a very long time and have never been able to find it.

I have seen two different issues and both only occur with Apache from MacOS X on Yosemite.

The first is that if use -DFOREGROUND and -DONE_PROCESS it would crash on startup.

The second is that sometimes on shutdown with mod_wsgi-express, the Python interpreter will crash as it being destroyed within the process.

Whenever I have attempted to stick a debugger on it to try and track down, the issue never occurs.

BTW, with the mod_wsgi-express way of triggering it, can you instead run:

    mod_wsgi-express start-server /wsgi/wsgi-test.wsgi —user apache —group apache —enable-gdb

This will launch httpd under gdb from the outset, in the -DFOREGROUND and -DONE_PROCESS mode. I want to see if in doing that it still crashes.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Graham Dumpleton]

Seems using —enable-gdb will not work as it doesn’t like something in my command file for gdb.

/ # mod_wsgi-express start-server /wsgi/wsgi-test.wsgi --user apache --group apache --enable-gdb

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:0

Server Conf        : /tmp/mod_wsgi-localhost:8000:0/httpd.conf

Error Log File     : /tmp/mod_wsgi-localhost:8000:0/error_log (warn)

Request Capacity   : 5 (1 process * 5 threads)

Request Timeout    : 60 (seconds)

Startup Timeout    : 15 (seconds)

Queue Backlog      : 100 (connections)

Queue Timeout      : 45 (seconds)

Server Capacity    : 20 (event/worker), 20 (prefork)

Server Backlog     : 500 (connections)

Locale Setting     : en_US.UTF-8

GNU gdb (GDB) 7.11

Copyright (C) 2016 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-alpine-linux-musl".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/local/lib/python2.7/site-packages/mod_wsgi_packages/httpd/bin/httpd...done.

warning: Error disabling address space randomization: Operation not permitted

/tmp/mod_wsgi-localhost:8000:0/gdb.cmds:1: Error in sourced command file:

During startup program exited with code 1.

Graham

[Konstantin Suvorov]

This config doesn't rise segfault, nor does it work.

When you  try to fetch /test url it gives 500 error with the following in logs:

[Fri Sep 30 07:18:19.019193 2016] [wsgi:error] [pid 19] [client 192.168.168.161:53243] No WSGI daemon process called 'example.com' has been configured: /wsgi/wsgi-test.wsgi

пятница, 30 сентября 2016 г., 3:27:13 UTC+3 пользователь Graham Dumpleton написал:

[Konstantin Suvorov]

result of --enable-gdb:

 # mod_wsgi-express start-server test.wsgi --user apache --group apache --enable-gdb

Server URL         : http://localhost:8000/

Server Root        : /tmp/mod_wsgi-localhost:8000:0

Server Conf        : /tmp/mod_wsgi-localhost:8000:0/httpd.conf

Error Log File     : /tmp/mod_wsgi-localhost:8000:0/error_log (warn)

Request Capacity   : 5 (1 process * 5 threads)

Request Timeout    : 60 (seconds)

Startup Timeout    : 15 (seconds)

Queue Backlog      : 100 (connections)

Queue Timeout      : 45 (seconds)

Server Capacity    : 20 (event/worker), 20 (prefork)

Server Backlog     : 500 (connections)

Locale Setting     : en_US.UTF-8

GNU gdb (GDB) 7.11

Copyright (C) 2016 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-alpine-linux-musl".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/local/lib/python2.7/site-packages/mod_wsgi_packages/httpd/bin/httpd...done.

Program received signal SIGSEGV, Segmentation fault.

0x00007ffff5457f72 in wsgi_add_import_script (cmd=0x7fffffffe7a0, mconfig=<optimized out>,

    args=0x555555832492 "process-group='%{GLOBAL}' application-group=%{GLOBAL}") at src/server/mod_wsgi.c:4996

4996 src/server/mod_wsgi.c: No such file or directory.

[Konstantin Suvorov]

Some more research...

If I run a fresh container with packaged mod_wsgi from my first post, I can start httd with the following wsgi-test.conf:

Listen 8080

WSGIDaemonProcess example0verylongname processes=2 threads=2

WSGIScriptAlias /test /wsgi/wsgi-test.wsgi

WSGIProcessGroup example0verylongname

<Directory /wsgi>

Require all granted

</Directory>

Note verylong process name.

But this one gives segfault:

Listen 8080

WSGIDaemonProcess example.com processes=2 threads=2

WSGIScriptAlias /test /wsgi/wsgi-test.wsgi

WSGIProcessGroup example.com

<Directory /wsgi>

Require all granted

</Directory>

And more to say, if I revert config back to "example0verylongname" after catching segfault, httpd will segfault with it too!!!

Also I've noted that if I strip ".com" from your example leaving only "example" process name, it segfaults.

If I use "example0" in your example, it starts.

Do you think it's possible to fix this? Or should I forget the idea of running mod_wsgi with Alpine linux?

[Graham Dumpleton]

For now I would give up on Alpine Linux. Any advantages of using Alpine Linux with Docker are overblown.

For now just play with:

    https://github.com/GrahamDumpleton/mod_wsgi-docker

Graham

[Konstantin Suvorov]

Ok, got it.

And last question: is there any downsides of running mod_wsgi in embedded mode in this setup? It seems to be running fine without dedicated daemon.

Httpd process in this container is serving only this wsgi application and nothing else.

[Graham Dumpleton]

Daemon mode is still much better as it provides a whole lot of timeout mechanisms which embedded mode doesn’t. These can be used to ensure your application stays alive, automatically restarting processes when things hang etc. When using mod_wsgi-express these timeouts are configured. If manually setting up mod_wsgi yourself you would need to work out what timeouts are and configure them yourself.

So daemon mode is better and especially using mod_wsgi-express where I do the configuration for you.

Graham

[Sean Gallagher]

Problem solved..

See here https://github.com/GrahamDumpleton/mod_wsgi/issues/620

Sorry to necro-post but this thread is the #1 google result for this issue.