Re: [mod_python] mod_wsgi (pid=88536): Unable to change working directory
Graham Dumpleton
> On 12/01/2010 21:24, mog wrote:
>>
>> Hi,
>>
>> I'm seeing this error in my main Apache log when I switched to using
>> daemon mode:
>>
>> [Tue Jan 12 21:13:47 2010] [alert] (2)No such file or directory: mod_wsgi
>> (pid=88536): Unable to change working directory to '/nonexistent'.
>>
>> Not sure why it would be happening though, why mod_wsgi would need to
>> change directories, or what the "correct" solution is? If someone could
>> please point me in the right direction, it would be greatly appreciated.
>>
>> Thanks,
>> mog
>>
>
> I read that mod_wsgi's daemon mode is the way forward, so let's "make it
> so". It doesn't seem to want to play nicely though.
>
> The 'unable to change working directory' problems sort of go away if I
> specify 'WSGIDaemonProcess home=/root', but that just doesn't feel right to
> me. Is there something more appropriate I should be using for that?
By rights it perhaps shouldn't be an alert but a warning. You can ignore it.
It occurs because in daemon mode mod_wsgi will change working
directory to the home directory of the user that the daemon process
runs as. On your system, the default Apache user doesn't have a home
directory and so the message. It will not stop anything from working.
If you really want to get rid of it, then override home as you have
done, setting it to '/tmp' even. Alternatively, create a dedicated
user, or use another existing user account that does have a home
directory, specifying the user/group options to WSGIDaemonProcess so
that the daemon process runs as that user and not the default Apache
user.
> My Apache vhost config is pretty normal with these WSGI specifics in:
>
> WSGIDaemonProcess meow.elasticmind.net processes=1 threads=5 home=/root
> display-name=%{GROUP}
> WSGIProcessGroup meow.elasticmind.net
> WSGIApplicationGroup %{GLOBAL}
> WSGIScriptAlias / /home/test/trunk/test/test.wsgi
>
> These are all present inside the VirtualHost directive but not contained
> within any Directory directives etc.
>
> Unfortunately though, it doesn't seem happy with these things. The only
> error messages I'm seeing are these:
>
> [error] Script timed out before returning headers: test.wsgi
>
> The content of test.wsgi is:
>
> def application(environ, start_response):
> status = '200 OK'
> output = 'Hello World!'
> response_headers = [('Content-type', 'text/plain'),
> ('Content-Length', str(len(output)))]
> start_response(status, response_headers)
> return [output]
>
> So without the DaemonProcess process stuff, this works absolutely fine. With
> the daemon mode process stuff, it breaks :(
Are you still loading mod_python into same Apache?
You likely have a conflicting version of mod_python loaded which is
causing problems. Disable mod_python and try again.
Please use mod_wsgi list in future as I already pointed out.
Graham
Graham Dumpleton
> On 12/01/2010 22:55, Graham Dumpleton wrote:
>>
>> By rights it perhaps shouldn't be an alert but a warning. You can ignore
>> it.
>> It occurs because in daemon mode mod_wsgi will change working
>> directory to the home directory of the user that the daemon process
>> runs as. On your system, the default Apache user doesn't have a home
>> directory and so the message. It will not stop anything from working.
>>
>> If you really want to get rid of it, then override home as you have
>> done, setting it to '/tmp' even. Alternatively, create a dedicated
>> user, or use another existing user account that does have a home
>> directory, specifying the user/group options to WSGIDaemonProcess so
>> that the daemon process runs as that user and not the default Apache
>> user.
>>
>
>
>
>> Are you still loading mod_python into same Apache?
>> You likely have a conflicting version of mod_python loaded which is
>> causing problems. Disable mod_python and try again.
>>
>
> really shouldn't be any remnants of mod_python lurking around; but even if
> there are, I removed the mod_python LoadModule directive from httpd.conf and
> have stopped and restarted Apache numerous times - so it really should be
> gone.
Just the LoadModule line for mod_python is enough to cause the
problem. This is because it is still loaded even if not used.
Also make sure you did a complete 'stop' of Apache and not just a
Linux init reload, as sometimes mod_python doesn't unload properly the
Python library.
If still a problem, you perhaps have SELinux extensions enabled and it
should be disabled or properly configured.
Graham
>> Please use mod_wsgi list in future as I already pointed out.
>>
>
> Oooops sorry, I don't remember any mention of that but I really should have
> thought to switch myself. I'll get myself subscribed to the mod_wsgi list
> pronto.
>
> Thanks again everyone!
> _______________________________________________
> Mod_python mailing list
> Mod_p...@modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
>
mog
> Just the LoadModule line for mod_python is enough to cause the
> problem. This is because it is still loaded even if not used.
>
> Also make sure you did a complete 'stop' of Apache and not just a
> Linux init reload, as sometimes mod_python doesn't unload properly the
> Python library.
>
> If still a problem, you perhaps have SELinux extensions enabled and it
> should be disabled or properly configured.
>
> Graham
It's a bit strange. Last night I made a point of removing the port, the
LoadModule line from httpd.conf and everything connected with
mod_python. Apache has been fully stopped and started again a number of
times. I also use FreeBSD and don't have any Linux compatibility or
ports stuff installed, so that shouldn't be an issue.
I will check things out again to see if I can find out what the problem
is. *fingers crossed* it goes away.
I was hoping to avoid it, but if there's nothing else that might be
causing it, I'll install Apache and mod_wsgi on a fresh box that has
never had any web stuff on it before, and see if that makes a difference.
Thanks again,
mog
Graham Dumpleton
configuration?
For such a simple script it shouldn't come into play, but if Timeout
has been set to 1 second and there was a delay in WSGI script file
loading and initialisation, you may be able to trip it I guess. I
would have to think about it as have never seen that timeout message
before and so have to work out when it could occur or whether it could
even occur under normal operation.
Graham
2010/1/13 mog <li...@elasticmind.net>:
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to
> modwsgi+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/modwsgi?hl=en.
>
>
>
>
Graham Dumpleton
> In extra/httpd-default.conf, the 'Timeout' is set to 300.
Which is the default value, equating to 5 minutes.
Do you get back a 500 error in browser immediately or only after a 5
minute wait?
Graham
> Please let me know if you think of anything, in the meantime I'll see if I
> can coerce it into behaving.
>
> With thanks,
> mog
mog
On 13/01/2010 01:08, Graham Dumpleton wrote:
> 2010/1/13 mog<li...@elasticmind.net>:
>
>> In extra/httpd-default.conf, the 'Timeout' is set to 300.
>>
> Which is the default value, equating to 5 minutes.
>
Indeed.
> Do you get back a 500 error in browser immediately or only after a 5
> minute wait?
>
I get the 500 code after a 5 minute wait. At the same time these are
written to the vhost log file:
[Wed Jan 13 01:14:34 2010] [error] Script timed out before returning
headers: saweb.wsgi
[Wed Jan 13 01:14:44 2010] [error] Script timed out before returning
headers: saweb.wsgi
mog
Graham Dumpleton
What I would like you to do is set:
LogLevel debug
WSGIVerboseLogging On
in Apache configuration.
The LogLevel directive is standard Apache directive, so you will need
to find where it is and change it. If it has been set in virtual host
context as well, then also change it there.
The other directive is mod_wsgi specific and just increases the amount
of debug logging.
Would like to then see the error log output (main Apache error log and
any virtual host log file) from period of time from request starting
to timeout.
Thanks.
Graham
mog
WSGIVerboseLogging directive. No matter where I put it, Apache complains
about it being there and wont start up.
Should I proceed with the test without it? Or?
Kind regards,
mog.
Graham Dumpleton
> Just trying to do these things, but it doesn't like the WSGIVerboseLogging
> directive. No matter where I put it, Apache complains about it being there
> and wont start up.
>
> Should I proceed with the test without it? Or?
Sorry, should be WSGIVerboseDebugging.
You do need mod_wsgi 3.0+ however.
Graham
> Kind regards,
> mog.
>
>
> On 13/01/2010 01:53, Graham Dumpleton wrote:
>>
>> What I would like you to do is set:
>>
>> LogLevel debug
>> WSGIVerboseLogging On
>>
>> in Apache configuration.
>>
>> The LogLevel directive is standard Apache directive, so you will need
>> to find where it is and change it. If it has been set in virtual host
>> context as well, then also change it there.
>>
>> The other directive is mod_wsgi specific and just increases the amount
>> of debug logging.
>>
>> Would like to then see the error log output (main Apache error log and
>> any virtual host log file) from period of time from request starting
>> to timeout.
>>
>> Thanks.
>>
>> Graham
>>
>
mog
> 2010/1/13 mog<li...@elasticmind.net>:
>
>> Just trying to do these things, but it doesn't like the WSGIVerboseLogging
>> directive. No matter where I put it, Apache complains about it being there
>> and wont start up.
>>
>> Should I proceed with the test without it? Or?
>>
> Sorry, should be WSGIVerboseDebugging.
>
> You do need mod_wsgi 3.0+ however.
>
No worries, both log levels are set.
Here goes... *waits for it to time out*
This is from the main Apache httpd-error log file:
[Wed Jan 13 02:16:28 2010] [notice] Apache/2.2.14 (FreeBSD)
mod_ssl/2.2.14 OpenSSL/0.9.8e DAV/2 mod_wsgi/3.1 Python/2.6.4 PHP/5.2.12
with Suhosin-Patch configured -- resuming normal operations
[Wed Jan 13 02:16:28 2010] [info] Server built: Jan 8 2010 14:09:05
[Wed Jan 13 02:16:28 2010] [debug] prefork.c(1013): AcceptMutex: flock
(default: flock)
[Wed Jan 13 02:16:28 2010] [info] mod_wsgi (pid=93063): Initializing Python.
[Wed Jan 13 02:16:28 2010] [info] mod_wsgi (pid=93064): Initializing Python.
[Wed Jan 13 02:16:28 2010] [info] mod_wsgi (pid=93062): Initializing Python.
[Wed Jan 13 02:16:28 2010] [info] mod_wsgi (pid=93065): Initializing Python.
[Wed Jan 13 02:16:28 2010] [info] mod_wsgi (pid=93061): Attach
interpreter ''.
[Wed Jan 13 02:16:28 2010] [info] mod_wsgi (pid=93065): Attach
interpreter ''.
[Wed Jan 13 02:16:28 2010] [info] mod_wsgi (pid=93063): Attach
interpreter ''.
[Wed Jan 13 02:16:28 2010] [info] mod_wsgi (pid=93064): Attach
interpreter ''.
[Wed Jan 13 02:16:28 2010] [info] mod_wsgi (pid=93062): Attach
interpreter ''.
<Apache start-up was complete at this point>
[Wed Jan 13 02:18:31 2010] [debug] mod_wsgi.c(11882): mod_wsgi
(pid=93061): Request server was 'test.elasticmind.net|0'.
[Wed Jan 13 02:18:31 2010] [info] mod_wsgi (pid=93071): Initializing Python.
[Wed Jan 13 02:18:31 2010] [info] mod_wsgi (pid=93071): Attach
interpreter ''.
<The above is what happened when I made the request>
[Wed Jan 13 02:23:31 2010] [debug] mod_wsgi.c(11882): mod_wsgi
(pid=93064): Request server was 'test.elasticmind.net|0'.
<The above is the result from the request timing out I think>
This is all the virtual host log-file said in the time period:
[Wed Jan 13 02:23:31 2010] [error] Script timed out before returning
headers: saweb.wsgi
Sorry this doesn't seem very useful at all to me, but it's literally all
there is :(
I am, however, building a jail so I've got a completely clean testing
environment to work with. I will set Apache and mod_wsgi up in there and
see what happens. If it's just an environmental thing it should work in
that.
With thanks,
mog
Graham Dumpleton
Not a lot, but jogged my memory over what this may be. It has come up
once, maybe twice before only and is in part why mod_wsgi was changed
to generate a timeout like that.
What was occurring in prior case, which was investigated a bit, was
that the Apache server child process was getting a successful result
for connecting to the listener socket of the daemon process and so
started proxying the request, but on the daemon side it was never
getting any indication at all that a connection had been received and
hadn't even accepted a connection. Thus daemon process was just
sitting there not doing anything yet the Apache server child process
was waiting for a response.
In the main case where this happened before it was only happening
every so often and not every time as seems to be your case. There was
suspicion over another case like yours where it happened every time
but don't remember the outcome of that. It is possible they just gave
up and went away without providing opportunity to debug it, or that
they rebuilt lots of stuff and it went away, I just can't remember.
One discussion about prior issue is:
http://groups.google.com/group/trac-dev/browse_frm/thread/bcaedbbb7fe5b6c
Unfortunately people don't always get back to me about whether issue
indeed went away or what they worked out the problem was. In this case
all I could do was to patch the code to make it more tolerant of
unexpected behaviour.
Give me a little while to think about this. I may put some more
debugging in code and get you to compile up a version from head of
subversion repository.
Graham
mog
Hi Graham,
Once again, thanks for this.
After spending a couple of hours setting up a pristine jail to test it
all, I was hoping the problem would just go away when I tried running
the example code in it. However, I was again frustrated by failure.
I tried another couple of things but still nothing. Then, really by
chance, I mentioned the annoying problem to a fellow FreeBSD person and
he told me what mod_wsgi configuration settings he was using on his
system, allowing me to compare with him what I was using.
I tried removing and adding things from the config to match up with his,
and after a few attempts I finally figured out what was causing the
problem. Basically by changing 'display-name=%{GROUP}' to
'display-name=arbitrarystring' and completely removing the
'WSGIProcessGroup test.elasticmind.net' line, it magically started working.
I think for some reason I thought 'WSGIProcessGroup' could be called
anything I wanted so I was using the domain name of the vhost. But
reading a bit about that directive suggests that this is not how it
works and it actually needs something special (although I'm not entirely
sure what). In a number of example configurations I see the
WSGIProcessGroup directive being used, but having just read the
Configuration Directives information on it, I'm not really sure what its
benefits are and why that is so - considering the problems it has caused
for me (although admittedly I'm really tired so I wouldn't be surprised
if my head is having a little trouble assimilating that information
right now, heh).
Does that make sense at all?
Kind regards,
mog
Graham Dumpleton
By removing WSGIProcessGroup you are no longer running your
application in daemon mode, but instead are using embedded mode. Since
this particular issue that is affecting you only affects daemon mode,
switching to embedded mode will as a result make it go away.
For a bit of a description about different modes and process models of
Apache/mod_wsgi have a read of:
http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
For the dangers of using embedded mode read:
http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
So, at least so you can move forward try with embedded mode, but I may
keep quizzing you about things so can work out why daemon mode isn't
working on FreeBSD.
BTW, can you supply me the output from running:
httpd -V
Graham
mog
> By removing WSGIProcessGroup you are no longer running your
> application in daemon mode, but instead are using embedded mode. Since
> this particular issue that is affecting you only affects daemon mode,
> switching to embedded mode will as a result make it go away.
>
Oh bugger. Yeah, so the reason it started working in the first place,
was because removing 'WSGIProcessGroup' was telling it to revert back to
embedded mode. That's really frustrating, heh :|
> For a bit of a description about different modes and process models of
> Apache/mod_wsgi have a read of:
>
> http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
>
> For the dangers of using embedded mode read:
>
> http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
>
Thanks for these. I remember reading the load spikes and memory usage
one before when you showed it to someone else. Although I wasn't using
mod_wsgi at the time, it seemed interesting and something to bear in
mind for the future. To be honest, I think it's the main reason why I've
been trying to get it to run in daemon mode (for memory conservation).
It's a real shame that it seems a bit broken. I'll read through the
processes and threading one too just as soon as I'm a bit more awake.
> So, at least so you can move forward try with embedded mode, but I may
> keep quizzing you about things so can work out why daemon mode isn't
> working on FreeBSD.
>
Yeah I'll do that, I need to crack on with the new project and make some
progress with migrating the old ones. That's no problem, if I can help
or provide more info that will assist with solving the issue, I'll
certainly do what I can. I'd still like to be able to use daemon mode
one day :)
> BTW, can you supply me the output from running:
>
> httpd -V
>
Sure...
Server version: Apache/2.2.14 (FreeBSD)
Server built: Jan 13 2010 02:48:24
Server's Module Magic Number: 20051115:23
Server loaded: APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_FLOCK_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local"
-D SUEXEC_BIN="/usr/local/bin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="/var/run/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="/var/log/httpd-error.log"
-D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
-D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"
Thanks again for all your help in advancing my understanding of all this
and for your help with getting it working.
Kind regards,
mog
Giel van Schijndel
On 14 jan, 00:27, mog <li...@elasticmind.net> wrote:
> On 13/01/2010 21:47, Graham Dumpleton wrote:
>> By removing WSGIProcessGroup you are no longer running your
>> application in daemon mode, but instead are using embedded mode. Since
>> this particular issue that is affecting you only affects daemon mode,
>> switching to embedded mode will as a result make it go away.
>
> Oh bugger. Yeah, so the reason it started working in the first place,
> was because removing 'WSGIProcessGroup' was telling it to revert back to
> embedded mode. That's really frustrating, heh :|
I seem to have the exact same problem on a FreeBSD system as well. It
runs well in embedded mode but just times out in daemon mode.
Except for the timestamps my logs are almost exactly the same to
mog's,
thus I won't quote them here right away (unless you'd want me to).
>> So, at least so you can move forward try with embedded mode, but I may
>> keep quizzing you about things so can work out why daemon mode isn't
>> working on FreeBSD.
>
> Yeah I'll do that, I need to crack on with the new project and make some
> progress with migrating the old ones. That's no problem, if I can help
> or provide more info that will assist with solving the issue, I'll
> certainly do what I can. I'd still like to be able to use daemon mode
> one day :)
>
>> BTW, can you supply me the output from running:
>
>> httpd -V
Thus I'm subscribing as another tester should you need one.
Version data:
> [giel@hyde:mgt/htdocs]$ uname -a
> FreeBSD hyde.il.fontys.nl 7.2-STABLE FreeBSD 7.2-STABLE #0: Fri Jun 12 00:01:27 CEST 2009 ro...@hyde.il.fontys.nl:/usr/obj/usr/src/sys/HYDE i386
> [giel@hyde:mgt/htdocs]$ httpd -V
> Server version: Apache/2.2.13 (FreeBSD)
> Server built: Sep 9 2009 00:15:52
> Server's Module Magic Number: 20051115:23
> Server loaded: APR 1.3.8, APR-Util 1.3.9
> Compiled using: APR 1.3.8, APR-Util 1.3.9
> Architecture: 32-bit
> Server MPM: Prefork
> threaded: no
> forked: yes (variable process count)
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/prefork"
> -D APR_HAS_SENDFILE
> -D APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D APR_USE_FLOCK_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
> -D APR_HAS_OTHER_CHILD
> -D AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=128
> -D HTTPD_ROOT="/usr/local"
> -D SUEXEC_BIN="/usr/local/sbin/suexec"
> -D DEFAULT_PIDLOG="/var/run/httpd.pid"
> -D DEFAULT_SCOREBOARD="/var/run/apache_runtime_status"
> -D DEFAULT_LOCKFILE="/var/run/accept.lock"
> -D DEFAULT_ERRORLOG="/var/log/httpd-error.log"
> -D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
> -D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"
--
With kind regards,
Giel van Schijndel
- Interlink <www.il.fontys.nl>
Graham Dumpleton
There were changes to how threads were managed on daemon process side
in mod_wsgi 3.1 and want to eliminate that as a cause.
Thanks.
Graham
2010/1/15 Giel van Schijndel <mortis....@gmail.com>:
Giel van Schijndel
> 2010/1/15 Giel van Schijndel <mortis....@gmail.com>:
>>> On 13/01/2010 21:47, Graham Dumpleton wrote:
>>>> By removing WSGIProcessGroup you are no longer running your
>>>> application in daemon mode, but instead are using embedded mode.
>>>> Since this particular issue that is affecting you only affects
>>>> daemon mode, switching to embedded mode will as a result make it go
>>>> away.
>>>
>>> Oh bugger. Yeah, so the reason it started working in the first
>>> place, was because removing 'WSGIProcessGroup' was telling it to
>>> revert back to embedded mode. That's really frustrating, heh :|
>>
>> I seem to have the exact same problem on a FreeBSD system as well. It
>> runs well in embedded mode but just times out in daemon mode.
>>
>> Except for the timestamps my logs are almost exactly the same to
>> mog's, thus I won't quote them here right away (unless you'd want me
>> to).
>>
>>>> So, at least so you can move forward try with embedded mode, but I may
>>>> keep quizzing you about things so can work out why daemon mode isn't
>>>> working on FreeBSD.
>>>
>>> Yeah I'll do that, I need to crack on with the new project and make
>>> some progress with migrating the old ones. That's no problem, if I
>>> can help or provide more info that will assist with solving the
>>> issue, I'll certainly do what I can. I'd still like to be able to
>>> use daemon mode one day :)
>>
>
>
> There were changes to how threads were managed on daemon process side
> in mod_wsgi 3.1 and want to eliminate that as a cause.
Using this before:
> [giel@hyde:www/mod_wsgi3]$ pkg_info | grep wsgi
> mod_wsgi-3.1 Python WSGI adapter module for Apache
now:
> [giel@hyde:www/mod_wsgi3]$ pkg_info | grep wsgi
> mod_wsgi-2.8 Python WSGI adapter module for Apache
Seems, you're right, there's a regression from 2.8 to 3.1. Downgrading
back to 2.8 seems to work with a daemon process (except for the
WSGIVerboseDebugging directive).
Thus perhaps some deadlock or something similar has been introduced that
somehow only gets triggered on FreeBSD? Either way, I didn't look at the
code so I'm not going to be guessing much further than that.
You're suggestion seems to work, I'll thus be running 2.8 for now. If
you have a patch and would like me to test it just mail me.
Graham Dumpleton
2010/1/15 Graham Dumpleton <graham.d...@gmail.com>:
> 2010/1/15 Graham Dumpleton <graham.d...@gmail.com>:
>> 2010/1/15 Giel van Schijndel <gi...@il.fontys.nl>:
>> Can you give me the result of running:
>>
>> ldd mod_wsgi.so
>>
>> for .so file from both mod_wsgi 2.8 and mod_wsgi 3.1.
>>
>> I am presuming here you are installing binary packages and not
>> something compiled from source code. Correct?
>>
>> One other possibility besides mod_wsgi 3.1 code changes, is whether
>> linked with same threading libraries. I understand that on some BSD
>> systems there are multiple threading libraries.
>
> Another request. When Apache is installed, it creates a 'build'
> install directory where apxs files are. In that directory is a file
> called 'config.nice', eg.,
>
> /usr/share/httpd/build/config.nice
>
> I would like to see the contents of that. For example, For Apple
> supplied httpd is:
>
> "/SourceCache/apache/apache-747/httpd/configure" \
> "--prefix=/usr" \
> "--enable-layout=Darwin" \
> "--with-apr=/usr" \
> "--with-apr-util=/usr" \
> "--with-pcre=/usr/local/bin/pcre-config" \
> "--enable-mods-shared=all" \
> "--enable-ssl" \
> "--enable-cache" \
> "--enable-mem-cache" \
> "--enable-proxy-balancer" \
> "--enable-proxy" \
> "--enable-proxy-http" \
> "--enable-disk-cache" \
> "$@"
>
> If you don't know where the 'build' directory is located, cat the
> 'apxs' script and look for:
>
> my $installbuilddir = "/usr/share/httpd/build";
Graham Dumpleton
>>> One other possibility besides mod_wsgi 3.1 code changes, is whether
>>> linked with same threading libraries. I understand that on some BSD
>>> systems there are multiple threading libraries.
Not necessarily up to date as for older FreeBSD, but read:
http://blag.whit537.org/2007/07/freebsd-threads-apache-and-modwsgi.html
This gives an indication of the problems with threading on FreeBSD in the past.
Graham
Graham Dumpleton
I have a FreeBSD 7.2 image installed under VmWare and think I have
ports installed.
How do I install mod_wsgi 2.8/3.1 ports given that they aren't in
standard ports tree, plus, even the ones at freshports are setup for
Apache 1.3 and not Apache 2.2.
http://www.freshports.org/www/mod_wsgi/
http://www.freshports.org/www/mod_wsgi3/
I have never installed FreeBSD before or used ports, so good
instructions for getting Apache 2.2 and newest mod_wsgi versions
installed from ports appreciated.
Graham
mog
> I have a FreeBSD 7.2 image installed under VmWare and think I have
> ports installed.
>
> How do I install mod_wsgi 2.8/3.1 ports given that they aren't in
> standard ports tree, plus, even the ones at freshports are setup for
> Apache 1.3 and not Apache 2.2.
>
> http://www.freshports.org/www/mod_wsgi/
> http://www.freshports.org/www/mod_wsgi3/
>
> I have never installed FreeBSD before or used ports, so good
> instructions for getting Apache 2.2 and newest mod_wsgi versions
> installed from ports appreciated.
>
> Graham
>
The system I was using to get things working on to begin with runs
FreeBSD 7.2 (amd64), then the machine I built a jail on for a clean
testing environment is running FreeBSD 7.2 (i386).
First you need to make sure your ports tree is present and up-to-date.
For most people, the best way to do this is using portsnap(8). To run
portsnap for the first time, you need to use the following commands:
portsnap fetch
portsnap extract
Naturally the first time portsnap runs, it has quite a lot of work to
do, so it may take a while. Future updates should happen quicker and can
be performed just by issuing the 'portsnap fetch update' command. For
more information on portsnap you may wish to check out the handbook page:
http://www.freebsd.org/doc/en/books/handbook/portsnap.html
Once the ports tree is updated, you can start installing things.
Personally, I'm using perl-5.10 at the moment (and Apache will want it)
so I like to install it first to get it out the way. If another port has
already installed a different version, it probably doesn't matter for
what you're doing.
To install perl I do:
/usr/ports/lang/perl5.10 && make install clean
A number of ports will display an Options menu when you go to install
them. This is just a simple list of checkbox options that can allow you
to tweak common configure/build options the port uses in a user-friendly
manner. For most ports you'll usually find that the default Options
settings are fine.
As you'd expect, the ports system will automatically install any
dependencies required by the port you are trying to install, so don't be
surprised if it shows you an Options menu for something else at some point.
Next you might like to install Python. You can do this like so:
cd /usr/ports/lang/python26 && make install clean
Then Apache:
cd /usr/ports/www/apache22 && make install clean
Then mod_wsgi 2.x:
cd /usr/ports/www/mod_wsgi && make install clean
Or for mod_wsgi 3.x use:
cd /usr/ports/www/mod_wsgi3 && make install clean
Note that the ports system (quite sensibly) will not allow you to have
both of these installed at the same time. To install a different
version, you must first remove the version currently installed. For
example, you might remove mod_wsgi like this.
pkg_info | grep mod_wsgi
mod_wsgi-3.1 Python WSGI adapter module for Apache
pkg_delete mod_wsgi-3.1
The pkg_info(1) command simply generates a list of all currently
installed ports / packages. Then the pkg_delete(1) command will delete
whatever port / package you tell it to.
Once installed, you'll find your Apache configuration files located in:
/usr/local/etc/apache22/
Note that the mod_wsgi ports will automatically add the LoadModule
directive into your httpd.conf for you. But it always makes sense to
check through it all anyway just in case.
You can let FreeBSD's rc.d help you manage the starting and stopping of
Apache. Doing these commands as root will also tell Apache to start when
your system boots up:
echo 'apache22_enable="YES"' >> /etc/rc.conf
echo 'apache22_http_accept_enable="YES"' >> /etc/rc.conf
And then use the rc.d control script to start Apache:
/usr/local/etc/rc.d/apache22 start
/usr/local/etc/rc.d/apache22 stop
/usr/local/etc/rc.d/apache22 graceful
etc
Do '/usr/local/etc/rc.d/apache22' with no arguments for a list of
available options.
Hope that helps. If I've missed anything out or something doesn't make
sense, just give me a shout.
mog.
Giel van Schijndel
> 2010/1/15 Graham Dumpleton <graham.d...@gmail.com>:
>>>> 2010/1/15 Giel van Schijndel <mortis....@gmail.com>:
>>>>> On 14 jan, 00:27, mog <li...@elasticmind.net> wrote:
>>>>>> On 13/01/2010 21:47, Graham Dumpleton wrote:
>>>>>>> So, at least so you can move forward try with embedded mode, but I may
>>>>>>> keep quizzing you about things so can work out why daemon mode isn't
>>>>>>> working on FreeBSD.
>>>>>>
>>>>>> Yeah I'll do that, I need to crack on with the new project and make
>>>>>> some progress with migrating the old ones. That's no problem, if I
>>>>>> can help or provide more info that will assist with solving the
>>>>>> issue, I'll certainly do what I can. I'd still like to be able to
>>>>>> use daemon mode one day :)
>>>>>
>>>>> Thus I'm subscribing as another tester should you need one.
>>>>
>>>> If you are both using mod_wsgi 3.1, can you instead try mod_wsgi 2.8.
>>>>
>>>> There were changes to how threads were managed on daemon process side
>>>> in mod_wsgi 3.1 and want to eliminate that as a cause.
>>
>>
>> ldd mod_wsgi.so
>>
>> for .so file from both mod_wsgi 2.8 and mod_wsgi 3.1.
> [giel@hyde:~]$ ldd /usr/tmp/usr/ports/www/mod_wsgi/work/mod_wsgi-2.8/.libs/mod_wsgi.so
> /usr/tmp/usr/ports/www/mod_wsgi/work/mod_wsgi-2.8/.libs/mod_wsgi.so:
> libpython2.6.so => /usr/local/lib/libpython2.6.so (0x28300000)
> libutil.so.7 => /lib/libutil.so.7 (0x281a8000)
> libm.so.5 => /lib/libm.so.5 (0x281b6000)
> libc.so.7 => /lib/libc.so.7 (0x28080000)
> libthr.so.3 => /lib/libthr.so.3 (0x281cb000)
> [giel@hyde:~]$ ldd /usr/tmp/usr/ports/www/mod_wsgi3/work/mod_wsgi-3.1/.libs/mod_wsgi.so
> /usr/tmp/usr/ports/www/mod_wsgi3/work/mod_wsgi-3.1/.libs/mod_wsgi.so:
> libpython2.6.so => /usr/local/lib/libpython2.6.so (0x28300000)
> libutil.so.7 => /lib/libutil.so.7 (0x281ad000)
> libm.so.5 => /lib/libm.so.5 (0x281bb000)
> libc.so.7 => /lib/libc.so.7 (0x28080000)
> libthr.so.3 => /lib/libthr.so.3 (0x281d0000)
>> I am presuming here you are installing binary packages and not
>> something compiled from source code. Correct?
I'm using FreeBSD ports. Which is a source-based distribution system.
(Similar to Gentoo's portage, if you're familiar with that).
>> One other possibility besides mod_wsgi 3.1 code changes, is whether
>> linked with same threading libraries. I understand that on some BSD
>> systems there are multiple threading libraries.
>
> Another request. When Apache is installed, it creates a 'build'
> install directory where apxs files are. In that directory is a file
> called 'config.nice', eg.,
>
> /usr/share/httpd/build/config.nice
>
> I would like to see the contents of that.
Its contents:
> [giel@hyde:~]$ cat /usr/local/share/apache22/build/config.nice
> #! /bin/sh
> #
> # Created by configure
>
> CC="cc"; export CC
> CFLAGS="-O2 -fno-strict-aliasing -pipe -I/usr/local/include/mysql -DHAVE_MYSQL_H -I/usr/include -DLDAP_DEPRECATED"; export CFLAGS
> CPPFLAGS="-I/usr/local/include"; export CPPFLAGS
> LDFLAGS=" -L/usr/local/lib/mysql -pthread -rpath=/usr/lib:/usr/local/lib -L/usr/lib -L/usr/local/lib"; export LDFLAGS
> "./configure" \
> "--enable-layout=FreeBSD" \
> "--with-perl=/usr/local/bin/perl5.8.9" \
> "--with-port=80" \
> "--with-expat=/usr/local" \
> "--with-iconv=/usr/local" \
> "--enable-http" \
> "--with-pcre=/usr/local" \
> "--disable-authn-file" \
> "--disable-authn-default" \
> "--disable-authz-host" \
> "--disable-authz-groupfile" \
> "--disable-authz-user" \
> "--disable-authz-default" \
> "--disable-auth-basic" \
> "--disable-charset-lite" \
> "--disable-include" \
> "--disable-log-config" \
> "--disable-env" \
> "--disable-setenvif" \
> "--disable-mime" \
> "--disable-status" \
> "--disable-autoindex" \
> "--disable-asis" \
> "--disable-cgid" \
> "--disable-cgi" \
> "--disable-negotiation" \
> "--disable-dir" \
> "--disable-imagemap" \
> "--disable-actions" \
> "--disable-userdir" \
> "--disable-alias" \
> "--disable-filter" \
> "--disable-proxy" \
> "--disable-proxy-connect" \
> "--disable-proxy-ftp" \
> "--disable-proxy-http" \
> "--disable-proxy-ajp" \
> "--disable-proxy-balancer" \
> "--enable-so" \
> "--with-suexec-caller=www-data" \
> "--with-suexec-uidmin=1000" \
> "--with-suexec-gidmin=1000" \
> "--with-suexec-userdir=public_html" \
> "--with-suexec-docroot=/usr/local/www/data" \
> "--with-suexec-safepath=/usr/local/bin:/usr/local/bin:/usr/bin:/bin" \
> "--with-suexec-logfile=/var/log/httpd-suexec.log" \
> "--with-suexec-bin=/usr/local/sbin/suexec" \
> "--enable-mods-shared=auth_basic auth_digest authn_file authn_dbd authn_dbm authn_anon authn_default authn_alias authz_host authz_groupfile authz_user authz_dbm authz_owner authz_default cache disk_cache file_cache dav dav_fs ldap authnz_ldap actions alias asis autoindex cern_meta cgi charset_lite deflate dir dumpio env expires headers imagemap include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias filter version ssl suexec suexec" \
> "--with-dbm=db41" \
> "--with-berkeley-db=/usr/local" \
> "--with-mysql=/usr/local" \
> "--with-pgsql=/usr/local" \
> "--with-sqlite3=/usr/local" \
> "--with-ssl=/usr" \
> "--with-ldap" \
> "--with-ldap-lib=/usr/local/lib" \
> "--with-ldap-include=/usr/local/include" \
> "--enable-v4-mapped" \
> "--enable-threads" \
> "--enable-nonportable-atomics" \
> "--with-mpm=prefork" \
> "--prefix=/usr/local" \
> "--mandir=/usr/local/man" \
> "--infodir=/usr/local/info/" \
> "--build=i386-portbld-freebsd7.2" \
> "build_alias=i386-portbld-freebsd7.2" \
> "CC=cc" \
> "CFLAGS=-O2 -fno-strict-aliasing -pipe -I/usr/local/include/mysql -DHAVE_MYSQL_H -I/usr/include -DLDAP_DEPRECATED" \
> "LDFLAGS= -L/usr/local/lib/mysql -pthread -rpath=/usr/lib:/usr/local/lib -L/usr/lib -L/usr/local/lib" \
> "LIBS=" \
> "CPPFLAGS=-I/usr/local/include" \
> "$@"
Giel van Schijndel
> On 15/01/2010 11:57, Graham Dumpleton wrote:
>> I have a FreeBSD 7.2 image installed under VmWare and think I have
>> ports installed.
>>
>> How do I install mod_wsgi 2.8/3.1 ports given that they aren't in
>> standard ports tree, plus, even the ones at freshports are setup for
>> Apache 1.3 and not Apache 2.2.
Actually those ports specify USE_APACHE=1.3+. Which means they require
an Apache version of 1.3 or more recent to be built. If you make sure to
install Apache 2.2 (www/apache22 port) *before* building mod_wsgi the
dependency resolver will be happy and won't force www/apache13 on you.
As for getting the most recent ports tree, mog's advice below is similar
to what I'd give you.
>> http://www.freshports.org/www/mod_wsgi/
>> http://www.freshports.org/www/mod_wsgi3/
>>
>> I have never installed FreeBSD before or used ports, so good
>> instructions for getting Apache 2.2 and newest mod_wsgi versions
>> installed from ports appreciated.
>
> The system I was using to get things working on to begin with runs
> FreeBSD 7.2 (amd64), then the machine I built a jail on for a clean
> testing environment is running FreeBSD 7.2 (i386).
FreeBSD 7.2 is the "legacy" production release. FreeBSD 8.0 is the
current production release (not to be confused with FreeBSD-CURRENT,
which is a development trunk).
Actually if you don't care for using specific versions you can just
start install the www/apache22 port first, then the www/mod_wsgi or
www/mod_wsgi second (and last). The ports system is capable enough to
pull in all required dependencies.
For Python, lang/python26 is already the default used (if you don't
explicitly use another version). I'm not sure about Perl though.
One thing I'd do as well is use 'make config-recursive install clean',
the config-recursive target will make sure that all build configuration
questions will be asked *before* it starts building (ensuring that the
build process won't pause waiting for user input).
> Note that the ports system (quite sensibly) will not allow you to
> have both of these installed at the same time. To install a
> different version, you must first remove the version currently
> installed. For example, you might remove mod_wsgi like this.
>
> pkg_info | grep mod_wsgi
> mod_wsgi-3.1 Python WSGI adapter module for Apache
>
> pkg_delete mod_wsgi-3.1
>
> The pkg_info(1) command simply generates a list of all currently
> installed ports / packages. Then the pkg_delete(1) command will
> delete whatever port / package you tell it to.
>
> Once installed, you'll find your Apache configuration files located in:
>
> /usr/local/etc/apache22/
>
> Note that the mod_wsgi ports will automatically add the LoadModule
> directive into your httpd.conf for you. But it always makes sense to
> check through it all anyway just in case.
>
> You can let FreeBSD's rc.d help you manage the starting and stopping
> of Apache. Doing these commands as root will also tell Apache to
> start when your system boots up:
>
> echo 'apache22_enable="YES"' >> /etc/rc.conf
> echo 'apache22_http_accept_enable="YES"' >> /etc/rc.conf
The first (enable) directive is required for the rc.d script (below) to
work at all.
At system bootup time something like this piece of shell code is ran:
> for rcscript in /etc/rc.d/* /usr/local/etc/rc.d
> do
> $rcscript start
> done
All those scripts (indirectly) include /etc/rc.conf, that's how they
determine whether to start the pieces of software for real.
> And then use the rc.d control script to start Apache:
>
> /usr/local/etc/rc.d/apache22 start
> /usr/local/etc/rc.d/apache22 stop
> /usr/local/etc/rc.d/apache22 graceful
> etc
> Do '/usr/local/etc/rc.d/apache22' with no arguments for a list of
> available options.
Graham Dumpleton
Haven't had much time to progress this but got everything built and
installed. When running up Apache however, it just hangs when
validating configuration. Hang occurs even when run 'apachectl -t'
directly.
The only no standard option I selected at any point was to enable
threading in APR for Apache build as without that mod_wsgi daemon mode
isn't going to work. At least, assumed I should select that.
FWIW, running truss on 'httpd -t', it is running, but think it is
looping over trying to do some DNS setup. Note that all I have done
with the OS installation is to say use DHCP, nothing else.
Graham
2010/1/16 mog <li...@elasticmind.net>:
Pawel Tyll
non-random reproductible problem with mod_wsgi3 and FreeBSD, I just
wanted to add my 'me too!'
Just burned through 3 hours of my life trying to figure out what's
wrong :)
FreeBSD 8, apache 2.2.14.
Graham Dumpleton
Yes, we are aware of it. :-)
Can you confirm that in your case problem goes away if mod_wsgi 2.8 is
used and that it only affects daemon mode and not embedded mode.
To add another data point, can you supply output of running:
httpd -V
and the contents of the file:
/usr/local/share/apache22/build/config.nice
FWIW, I have been trying to setup FreeBSD under VMWare, but Apache is
failing even to start up, possibly hanging trying to do some DNS
lookup when validating Apache configuration. I have been quite busy as
still not able to find much time to do stuff since moved house. So,
haven't had a chance to even work out why Apache will not start.
http://groups.google.com/group/modwsgi/browse_frm/thread/51a4e3bbb9ed7ecc
Graham
Pawel Tyll
wrote:
> Yes, we are aware of it. :-)
>
> Can you confirm that in your case problem goes away if mod_wsgi 2.8 is
> used and that it only affects daemon mode and not embedded mode.
went away after switching to 2.8.
> To add another data point, can you supply output of running:
> httpd -V
# uname -m -r -s
FreeBSD 8.0-STABLE amd64
# httpd -V
Server version: Apache/2.2.14 (FreeBSD)
Server built: Jan 19 2010 05:29:01
Server's Module Magic Number: 20051115:23
Server loaded: APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_FLOCK_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local"
-D SUEXEC_BIN="/usr/local/bin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="/var/run/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="/var/log/httpd-error.log"
-D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
-D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"
> and the contents of the file:
> /usr/local/share/apache22/build/config.nice
# cat /usr/local/share/apache22/build/config.nice
#! /bin/sh
#
# Created by configure
CC="cc"; export CC
CFLAGS="-O2 -pipe -march=nocona -I/usr/local/include/mysql -
DHAVE_MYSQL_H -I/usr/local/include -fno-strict-aliasing"; export
CFLAGS
CPPFLAGS="-I/usr/local/include"; export CPPFLAGS
LDFLAGS=" -L/usr/local/lib/mysql -pthread -rpath=/usr/local/lib -L/usr/
local/lib -L/usr/local/lib"; export LDFLAGS
LTFLAGS="--tag=CXX"; export LTFLAGS
"--disable-proxy-scgi" \
"--enable-so" \
"--enable-mods-shared=auth_basic auth_digest authn_file authn_dbm
authn_anon authn_default authn_alias authz_host authz_groupfile
authz_user authz_dbm authz_owner authz_default cache disk_cache
file_cache dav dav_fs actions alias asis autoindex cern_meta cgi
charset_lite dir dumpio env expires headers imagemap include info
log_config logio mime mime_magic negotiation rewrite setenvif speling
status unique_id userdir usertrack vhost_alias filter version ssl" \
"--with-dbm=sdbm" \
"--with-mysql=/usr/local" \
"--with-pgsql=/usr/local" \
"--with-sqlite3=/usr/local" \
"--with-ssl=/usr/local" \
"--with-apr=/usr/local/bin/apr-1-config" \
"--with-apr-util=/usr/local/bin/apu-1-config" \
"--enable-v4-mapped" \
"--enable-threads" \
"--with-mpm=prefork" \
"--prefix=/usr/local" \
"--mandir=/usr/local/man" \
"--infodir=/usr/local/info/" \
"--build=amd64-portbld-freebsd8.0" \
"build_alias=amd64-portbld-freebsd8.0" \
"CC=cc" \
"CFLAGS=-O2 -pipe -march=nocona -I/usr/local/include/mysql -
DHAVE_MYSQL_H -I/usr/local/include -fno-strict-aliasing" \
"LDFLAGS= -L/usr/local/lib/mysql -pthread -rpath=/usr/local/lib -L/usr/
local/lib -L/usr/local/lib" \
"LIBS=" \
"CPPFLAGS=-I/usr/local/include" \
"$@"
> FWIW, I have been trying to setup FreeBSD under VMWare, but Apache is
> failing even to start up, possibly hanging trying to do some DNS
> lookup when validating Apache configuration. I have been quite busy as
> still not able to find much time to do stuff since moved house. So,
> haven't had a chance to even work out why Apache will not start.
You've probably already did all the obvious things, like host www.google.com
to check if DNS is even working properly, so I can't think of any
ideas as to why's that happening
mog
> Yes, we are aware of it. :-)
>
> Can you confirm that in your case problem goes away if mod_wsgi 2.8 is
> used and that it only affects daemon mode and not embedded mode.
>
> To add another data point, can you supply output of running:
>
> httpd -V
>
> and the contents of the file:
>
> /usr/local/share/apache22/build/config.nice
>
> FWIW, I have been trying to setup FreeBSD under VMWare, but Apache is
> failing even to start up, possibly hanging trying to do some DNS
> lookup when validating Apache configuration. I have been quite busy as
> still not able to find much time to do stuff since moved house. So,
> haven't had a chance to even work out why Apache will not start.
>
> http://groups.google.com/group/modwsgi/browse_frm/thread/51a4e3bbb9ed7ecc
>
> Graham
>
Heh, in a weird way, it's kind of comforting to know I'm not the only
one who's been insanely busy these past few days :|
Seems like you're having fun with more weirdness :(
Did the fetch and install of the ports work okay? Confirming that the VM
itself can resolve DNS queries and that /etc/resolv.conf doesn't need a
tweak or something?
mog
Graham Dumpleton
The fetching of stuff from ports was working fine, so presumed that
DNS stuff was fine.
Am trying to rebuild Apache again and not fiddle the thread options
this time. Am inside a corporate firewall though, so if needs to
access out I may have problems.
Graham
Graham Dumpleton
Rebuilding itself didn't help and when did rebuild it, didn't give me
options menus for changing configuration anyway.
Still hung, even without mod_wsgi. Setting DNS server manually in
resolv.conf and adding 'freebsd.localdomain' alias against 'localhost'
in hosts file got Apache started.
No problem with hello world WSGI application in embedded mode, but
hangs in daemon mode. Waiting for it to do the 300 second time out.
Hopefully this gives me a test bed for tracking it down.
More later.
Graham
Pawel Tyll
wrote:
> Rebuilding itself didn't help and when did rebuild it, didn't give me
> options menus for changing configuration anyway.
> Hopefully this gives me a test bed for tracking it down.
It does.
Also I noted that socket was created with wrong owning user, but
correct group when defining user= and group= - and umask= was ignored
(or maybe it isn't used to set socket's permissions?). Maybe something
to look at?
Cheers.
Graham Dumpleton
> On Jan 20, 3:45 am, Graham Dumpleton <graham.dumple...@gmail.com>
> wrote:
>> Rebuilding itself didn't help and when did rebuild it, didn't give me
>> options menus for changing configuration anyway.
> Try a 'make config' in port directory. Should work wonders for you ;)
>
>> Hopefully this gives me a test bed for tracking it down.
> It does.
>
> Also I noted that socket was created with wrong owning user, but
> correct group when defining user= and group= -
The socket isn't meant to have permissions matching daemon process.
Has to have permissions which normal Apache server child processes
running as Apache user can access as client.
> and umask= was ignored
> (or maybe it isn't used to set socket's permissions?). Maybe something
> to look at?
The socket permissions I see are 0700 and owner/group is www/wheel,
which looks fine to me as owned by Apache user and group is probably
what Apache was when started as root.
What are the permissions you have?
Anyway, it seems that apr_atomic_cas32() in APR libraries is broken on
FreeBSD for some reason. That or the way I use it isn't quite right
and although it works fine on other platforms, not on FreeBSD.
Basically, the way of use it is as a faster way than a thread mutex of
allowing only one thread into a critical section of code, but on
FreeBSD it is just letting all threads through. They all then do
poll() on listener socket and doing that either stuffs things up, or
they all then go onto accept the connection when it comes and that
fails.
The apr_atomic_cas32() function is used in some of the Apache MPM
modules so if it is broken, would expect Apache core to have issues as
well.
Graham
Pawel Tyll
wrote:
> The socket isn't meant to have permissions matching daemon process.
> Has to have permissions which normal Apache server child processes
> running as Apache user can access as client.
> The apr_atomic_cas32() function is used in some of the Apache MPM
> modules so if it is broken, would expect Apache core to have issues as
> well.
No problems with apache itself here. Maybe try looking at the calls
themselves to double check that you're using them correctly? Oh well,
since everything is fine under other OSes, then you probably are.
Graham Dumpleton
Except that default Apache on FreeBSD is 'prefork'. The
apr_atomic_cas32() call is only used in worker MPM. :-)
Know of people have problems with worker MPM on FreeBSD. Maybe for
prefork the configure options aren't quite the same and so something
not being compiled in correctly. Checking the object files though, it
appears to be correctly compile the routines from ia32.c file in apr
library, so is using Intel instructions.
I'll have to review my code, but it is effectively a copy of something
similar done in one of the old experimental MPMs for Apache. :-(
Graham
Graham Dumpleton
mod_wsgi 3.X.
It appears that FreeBSD doesn't like a thread mutex being unlocked
from a thread different to the thread it was originally locked by.
This was causing use of a condition variable to fail but error result
wasn't being tested and so code was assuming it had been successful in
being notified that it could precede.
The change moves the locking of the mutex into the actual thread when
it is created.
Reading some of the comments against the problem function in APR am
still not sure the original code I copied is completely robust and it
perhaps should be using additional variable flag to know condition has
in fact been satisfied. My code where I got it working does this, but
have only posted the minimal fix at this point.
Let me know how it goes. I'll review the code a bit more and perhaps
also include my other changes to make it more robust. The potential
problems in it may explain some of the other very rare problems on
Linux platforms.
Sorry for how long this has taken, I just haven't had much time of
late due to things still needing to be done related to me moving
house. I have been lucky to steal a couple of hours that I have an am
sure to be roasted over the fact my office/study is still a mess. :-)
Graham
Index: mod_wsgi.c
===================================================================
--- mod_wsgi.c (revision 1523)
+++ mod_wsgi.c (working copy)
@@ -10399,6 +10399,8 @@
WSGIDaemonThread *thread = data;
apr_pool_t *p = apr_thread_pool_get(thd);
+ apr_thread_mutex_lock(thread->mutex);
+
wsgi_daemon_worker(p, thread);
apr_thread_exit(thd, APR_SUCCESS);
@@ -10679,8 +10681,6 @@
sleep(5);
}
- apr_thread_mutex_lock(thread->mutex);
-
/* Now create the actual thread. */
thread->id = i;
2010/1/20 Graham Dumpleton <graham.d...@gmail.com>:
Giel van Schijndel
> 2010/1/20 Graham Dumpleton <graham.d...@gmail.com>:
>> 2010/1/20 Pawel Tyll <of...@ofca.pl>:
>>> On Jan 20, 4:55 am, Graham Dumpleton <graham.dumple...@gmail.com> >>> wrote:
>>>> The apr_atomic_cas32() function is used in some of the Apache MPM
>>>> modules so if it is broken, would expect Apache core to have issues
>>>> as well.
>>> No problems with apache itself here. Maybe try looking at the calls
>>> themselves to double check that you're using them correctly? Oh
>>> well, since everything is fine under other OSes, then you probably
>>> are.
>>
>> Except that default Apache on FreeBSD is 'prefork'. The
>> apr_atomic_cas32() call is only used in worker MPM. :-)
>>
>> Know of people have problems with worker MPM on FreeBSD. Maybe for
>> prefork the configure options aren't quite the same and so something
>> not being compiled in correctly. Checking the object files though, it
>> appears to be correctly compile the routines from ia32.c file in apr
>> library, so is using Intel instructions.
>>
>> I'll have to review my code, but it is effectively a copy of something
>> similar done in one of the old experimental MPMs for Apache. :-(
>
> mod_wsgi 3.X.
>
> It appears that FreeBSD doesn't like a thread mutex being unlocked
> from a thread different to the thread it was originally locked by.
> This was causing use of a condition variable to fail but error result
> wasn't being tested and so code was assuming it had been successful in
> being notified that it could precede.
No, I believe thread lock ownership is directly associated with the
kernel structures used for runqueue objects (i.e. threads & processes).
Either way, it doesn't appear like good practice to me either, i.e.
unlocking a mutex in a different thread than it was locked. That looks a
lot like wishing for race conditions, which mutexes are intended to help
prevent.
> The change moves the locking of the mutex into the actual thread when
> it is created.
>
> Reading some of the comments against the problem function in APR am
> still not sure the original code I copied is completely robust and it
> perhaps should be using additional variable flag to know condition has
> in fact been satisfied. My code where I got it working does this, but
> have only posted the minimal fix at this point.
>
> Let me know how it goes. I'll review the code a bit more and perhaps
> also include my other changes to make it more robust. The potential
> problems in it may explain some of the other very rare problems on
> Linux platforms.
Running it for 30 minutes now. No problems so far. Last time I had
timeout problems immediately, I'm guessing caused by a dead lock which
that patch fixes.
Either way, I'll leave it running for now until problems arise (in which
case I'll gladly notify you again).
> Sorry for how long this has taken, I just haven't had much time of
> late due to things still needing to be done related to me moving
> house. I have been lucky to steal a couple of hours that I have an am
> sure to be roasted over the fact my office/study is still a mess. :-)
No problem. I'm not exactly suffering the luxury of having plenty of
time either.
Graham Dumpleton
No race condition. Each thread had its own dedicated mutex and it was
being locked initially before even the thread which would use had been
created. The way problem manifested was that the mutex was bound to a
thread condition variable, when performing a wait on that condition
variable it would return immediately with an error due to the internal
unlocking of the mutex failing since the mutex was original locked in
parent thread that created that thread. This meant that multiple
threads fell through to a select/poll on listener socket and and when
request came, multiple threads proceeded to call accept on listener
socket. Although multiple calls to accept on listener socket work on
some platforms, it doesn't work on others and FreeBSD appears to be
one of those. For those platforms which do allow multiple accept calls
and don't allow unlocking of mutex in different thread, there would
also technically be a problem, but still managing to work adequately
since multiple accepts work.
Most systems appear to support unlocking mutex in different thread and
it just seems to be FreeBSD that doesn't.
That said, I agree that probably not good practice, and that is what I
get for copying someone else's code. Given the code came from an
experimental MPM in Apache 2.0, not surprised that it never became
part of standard Apache distribution with issues like have had.
>> The change moves the locking of the mutex into the actual thread when
>> it is created.
>>
>> Reading some of the comments against the problem function in APR am
>> still not sure the original code I copied is completely robust and it
>> perhaps should be using additional variable flag to know condition has
>> in fact been satisfied. My code where I got it working does this, but
>> have only posted the minimal fix at this point.
>>
>> Let me know how it goes. I'll review the code a bit more and perhaps
>> also include my other changes to make it more robust. The potential
>> problems in it may explain some of the other very rare problems on
>> Linux platforms.
>
> Running it for 30 minutes now. No problems so far. Last time I had
> timeout problems immediately, I'm guessing caused by a dead lock which
> that patch fixes.
As said above, not a deadlock, but multiple accept calls at same time
on listener socket not working on that platform.
Graham
> Either way, I'll leave it running for now until problems arise (in which
> case I'll gladly notify you again).
>
>> Sorry for how long this has taken, I just haven't had much time of
>> late due to things still needing to be done related to me moving
>> house. I have been lucky to steal a couple of hours that I have an am
>> sure to be roasted over the fact my office/study is still a mess. :-)
>
> No problem. I'm not exactly suffering the luxury of having plenty of
> time either.
>
> --
> With kind regards,
> Giel van Schijndel
> - Interlink <www.il.fontys.nl>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iEYEARECAAYFAktd6xAACgkQZBYm/87l50IP4gCfaXVQ8IGFoxLNamQ7EbOo7yh5
> WoUAoItop9Gz4RjO1pUyOjWlorgdwnd+
> =VQ2W
> -----END PGP SIGNATURE-----
>
>
Giel van Schijndel
> On Mon, Jan 25, 2010 at 10:10:42PM +1100, Graham Dumpleton wrote:
>> Let me know how it goes. I'll review the code a bit more and perhaps
>> also include my other changes to make it more robust. The potential
>> problems in it may explain some of the other very rare problems on
>> Linux platforms.
>
> Running it for 30 minutes now. No problems so far. Last time I had
> timeout problems immediately, I'm guessing caused by a dead lock which
> that patch fixes.
>
> Either way, I'll leave it running for now until problems arise (in which
> case I'll gladly notify you again).
Some time later, and now it seems that the daemon crashed. At least I
think it must have, considering that it didn't leave anything in the
logs, except for timeouts around when the daemon must have gone. No
coredump either.
Graham Dumpleton
I'll give you an updated patch shortly then which includes the other
changes I figured are required to make it more robust on platforms
where conditional wait can actually return even though condition not
satisfied.
Graham
Graham Dumpleton
Revert that prior patch and try this one instead:
Index: mod_wsgi.c
===================================================================
--- mod_wsgi.c (revision 1523)
+++ mod_wsgi.c (working copy)
@@ -1314,6 +1314,7 @@
apr_thread_t *thread;
int running;
int next;
+ int wakeup;
apr_thread_cond_t *condition;
apr_thread_mutex_t *mutex;
} WSGIDaemonThread;
@@ -10099,7 +10100,29 @@
continue;
}
else {
- return apr_thread_cond_wait(thread->condition, thread->mutex);
+ apr_status_t rv;
+
+ if (thread->wakeup) {
+ thread->wakeup = 0;
+
+ return APR_SUCCESS;
+ }
+
+ rv = apr_thread_cond_wait(thread->condition, thread->mutex);
+
+ while (rv == APR_SUCCESS && !thread->wakeup)
+ rv = apr_thread_cond_wait(thread->condition, thread->mutex);
+
+ if (rv != APR_SUCCESS) {
+ ap_log_error(APLOG_MARK, WSGI_LOG_CRIT(rv),
+ wsgi_server, "mod_wsgi (pid=%d): "
+ "Wait on thread %d wakeup condition variable "
+ "failed.", getpid(), id);
+ }
+
+ thread->wakeup = 0;
+
+ return rv;
}
}
}
@@ -10130,9 +10153,8 @@
}
else {
/*
- * Acquire and release the idle worker's mutex
- * to ensure that it's actually waiting on its
- * condition variable
+ * Flag that thread should be woken up and then
+ * signal it via the condition variable.
*/
apr_status_t rv;
@@ -10140,6 +10162,9 @@
APR_SUCCESS) {
return rv;
}
+
+ thread->wakeup = 1;
+
if ((rv = apr_thread_mutex_unlock(thread->mutex)) !=
APR_SUCCESS) {
return rv;
@@ -10399,6 +10424,8 @@
WSGIDaemonThread *thread = data;
apr_pool_t *p = apr_thread_pool_get(thd);
+ apr_thread_mutex_lock(thread->mutex);
+
wsgi_daemon_worker(p, thread);
apr_thread_exit(thd, APR_SUCCESS);
@@ -10679,8 +10706,6 @@
Giel van Schijndel
> 2010/1/26 Graham Dumpleton <graham.d...@gmail.com>:
>> 2010/1/26 Giel van Schijndel <mortis....@gmail.com>:
>>> On Mon, Jan 25, 2010 at 08:03:44PM +0100, Giel van Schijndel wrote:
>>>> On Mon, Jan 25, 2010 at 10:10:42PM +1100, Graham Dumpleton wrote:
>>>>> Let me know how it goes. I'll review the code a bit more and perhaps
>>>>> also include my other changes to make it more robust. The potential
>>>>> problems in it may explain some of the other very rare problems on
>>>>> Linux platforms.
>>>>
>>>> Running it for 30 minutes now. No problems so far. Last time I had
>>>> timeout problems immediately, I'm guessing caused by a dead lock which
>>>> that patch fixes.
>>>>
>>>> Either way, I'll leave it running for now until problems arise (in which
>>>> case I'll gladly notify you again).
>>>
>>> Some time later, and now it seems that the daemon crashed. At least I
>>> think it must have, considering that it didn't leave anything in the
>>> logs, except for timeouts around when the daemon must have gone. No
>>> coredump either.
>>
>> I'll give you an updated patch shortly then which includes the other
>> changes I figured are required to make it more robust on platforms
>> where conditional wait can actually return even though condition not
>> satisfied.
>
> Revert that prior patch and try this one instead:
No immediate regressions so far. I.e. it functions properly within a few
minutes after restarting Apache with it.
Graham Dumpleton
error log, or if you have them already, from both main Apache error
log and any virtual host log which daemon process is within, post them
for me.
Graham
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iEYEARECAAYFAkteU+YACgkQZBYm/87l50JyxwCeOskvPtWAauq+++cCEnUJ9/H5
> Ol4AoInWrbyw4oKQuOvHWELK4lm+LAYl
> =C4an
> -----END PGP SIGNATURE-----
>
>
Graham Dumpleton
http://code.google.com/p/modwsgi/issues/detail?id=176
for this. Have been a bit lax in doing this.
I'll treat the inactivity timeout issue separately, if it proves to be
an actual problem.
Graham
Giel van Schijndel
> 2010/1/26 Giel van Schijndel <mortis....@gmail.com>:
>> On Tue, Jan 26, 2010 at 01:12:00AM +0100, Giel van Schijndel wrote:
>>> On Tue, Jan 26, 2010 at 11:00:42AM +1100, Graham Dumpleton wrote:
>>>> 2010/1/26 Graham Dumpleton <graham.d...@gmail.com>:
>>>>> 2010/1/26 Giel van Schijndel <mortis....@gmail.com>:
>>>>>> think it must have, considering that it didn't leave anything in the
>>>>>> logs, except for timeouts around when the daemon must have gone. No
>>>>>> coredump either.
>>>>>
>>>>> I'll give you an updated patch shortly then which includes the other
>>>>> changes I figured are required to make it more robust on platforms
>>>>> where conditional wait can actually return even though condition not
>>>>> satisfied.
>>>>
>>>> Revert that prior patch and try this one instead:
>>>
>>> No immediate regressions so far. I.e. it functions properly within a few
>>> minutes after restarting Apache with it.
>>
>> It seems that after the daemon times out due to inactivity it gets
>> killed, but it never seems to be respawned when a request arrives after
>> that timeout.
>
> error log, or if you have them already, from both main Apache error
> log and any virtual host log which daemon process is within, post them
> for me.
Logs (last request + everything that follows):
> [Tue Jan 26 15:18:48 2010] [debug] mod_wsgi.c(12695): mod_wsgi (pid=16220): Request server matched was 'ilpam.il.fontys.nl|0'.
> [Tue Jan 26 15:27:33 2010] [info] mod_wsgi (pid=16220): Daemon process deadlock timer expired, stopping process 'ilpam'.
> [Tue Jan 26 15:27:33 2010] [info] mod_wsgi (pid=16220): Shutdown requested 'ilpam'.
> [Tue Jan 26 15:27:33 2010] [info] mod_wsgi (pid=16220): Stopping process 'ilpam'.
> [Tue Jan 26 15:27:33 2010] [info] mod_wsgi (pid=16220): Destroying interpreters.
> [Tue Jan 26 15:27:33 2010] [debug] mod_wsgi.c(5172): mod_wsgi (pid=16220): Create thread state for thread 0 against interpreter 'ilpam.il.fontys.nl|'.
> [Tue Jan 26 15:27:33 2010] [info] mod_wsgi (pid=16220): Destroy interpreter 'ilpam.il.fontys.nl|'.
> [Tue Jan 26 15:27:33 2010] [info] mod_wsgi (pid=16220): Cleanup interpreter ''.
> [Tue Jan 26 15:27:33 2010] [info] mod_wsgi (pid=16220): Terminating Python.
> [Tue Jan 26 15:27:33 2010] [info] mod_wsgi (pid=16220): Python has shutdown.
Looking at the process tree of Apache it seems the daemon has gone
defunct:
> [giel@hyde:~]$ pstree -p 8103
> -+= 00001 root /sbin/init --
> \-+= 08103 root /usr/local/sbin/httpd -f /beast/config/apache/httpd.conf -DNOHTTPACCEPT
> |--- 16220 giel <defunct>
> |--- 16221 www /usr/local/sbin/httpd -f /beast/config/apache/httpd.conf -DNOHTTPACCEPT
> |--- 16222 www /usr/local/sbin/httpd -f /beast/config/apache/httpd.conf -DNOHTTPACCEPT
> |--- 16223 www /usr/local/sbin/httpd -f /beast/config/apache/httpd.conf -DNOHTTPACCEPT
> |--- 16224 www /usr/local/sbin/httpd -f /beast/config/apache/httpd.conf -DNOHTTPACCEPT
> |--- 16225 www /usr/local/sbin/httpd -f /beast/config/apache/httpd.conf -DNOHTTPACCEPT
> |--- 16226 www /usr/local/sbin/httpd -f /beast/config/apache/httpd.conf -DNOHTTPACCEPT
> |--- 16227 www /usr/local/sbin/httpd -f /beast/config/apache/httpd.conf -DNOHTTPACCEPT
> \--- 16228 www /usr/local/sbin/httpd -f /beast/config/apache/httpd.conf -DNOHTTPACCEPT
Further I had a small script running to look at what's happening to the
process:
> while kill -0 16220
> do
> (date ; ps aux | grep -v grep | grep 16220) | tee -a mod_wsgi_daemon.log
> sleep 10
> done
Its output from the time the daemon went defunct:
> Tue Jan 26 15:27:21 CET 2010
> giel 16220 0.0 1.1 28592 22216 ?? I 3:11PM 0:00.81 httpd: ILMAP (httpd)
> Tue Jan 26 15:27:31 CET 2010
> giel 16220 0.0 1.1 28592 22216 ?? I 3:11PM 0:00.81 httpd: ILMAP (httpd)
> Tue Jan 26 15:27:41 CET 2010
> giel 16220 0.1 0.0 0 0 ?? Z 3:11PM 0:00.85 <defunct>
> Tue Jan 26 15:27:51 CET 2010
> giel 16220 0.0 0.0 0 0 ?? Z 3:11PM 0:00.85 <defunct>
I'm guessing that the daemon process shouldn't be allowed to remain
zombie?
Further sending SIGHUP (or using 'apachectl graceful') to the parent
Apache process causes it to be properly cleaned up and respawned.
Graham Dumpleton
Hmmm, you were talking about inactivity timeouts, yet above is
something completely different. The above specifically happens when
something has locked Python GIL for period of deadlock timeout,
default 300 seconds, and as such no other Python code can run.
I can't see from logs, as not enough from before this point, whether
this occurred on inactivity timeout shutdown or whether distinct. My
guess is it is distinct as for inactivity timeout, where shutdown
timeout is 5 seconds by default, there is no way one could get to 300
seconds and trigger this.
Can you explain more about how often/when you are seeing this deadlock
timeout in error logs. Something like that is not good as indicates a
badly implemented Python C extension module that doesn't release GIL
when doing blocking operations.
Graham
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iEYEARECAAYFAkte/xQACgkQZBYm/87l50JueACfVg7NCMKWYT6EhfLsLJI+KoQC
> J/gAmwQvhbne5O/0y4kZ960DD7ccO2AO
> =zB3N
> -----END PGP SIGNATURE-----
>
>
Giel van Schijndel
> 2010/1/27 Giel van Schijndel:
>> On Tue, Jan 26, 2010 at 01:46:53PM +1100, Graham Dumpleton wrote:
>>>> On Tue, Jan 26, 2010 at 01:12:00AM +0100, Giel van Schijndel wrote:
>>>>> On Tue, Jan 26, 2010 at 11:00:42AM +1100, Graham Dumpleton wrote:
>>>>>>> 2010/1/26 Giel van Schijndel:
Except that the above doesn't look like 300 seconds to me, more like
515.
I guess I could test this by setting 'threads=1'?
> I can't see from logs, as not enough from before this point, whether
> this occurred on inactivity timeout shutdown or whether distinct. My
> guess is it is distinct as for inactivity timeout, where shutdown
> timeout is 5 seconds by default, there is no way one could get to 300
> seconds and trigger this.
>
> Can you explain more about how often/when you are seeing this deadlock
> timeout in error logs. Something like that is not good as indicates a
> badly implemented Python C extension module that doesn't release GIL
> when doing blocking operations.
I'm not using any Python C extensions, not directly at least. The only
things (aside from the Python standard library) I use are CherryPy
(3.1.2) and Genshi (0.5.1). It's a rather simple project up until now
[1], currently with as only code a 125 line long Python script [2].
As for how often I see this deadlock. I thought it to be related to the
inactivity timeout because it appears to happen about 500 (+/- 120)
seconds after the last HTTP request gets in.
[1] http://git.il.fontys.nl/git/ilpam.git
[2] http://git.il.fontys.nl/git/ilpam.git/blob/HEAD:/mgt/web/wsgi.py
Graham Dumpleton
I wasn't saying it was linked to the prior logged message, just that
the deadlock timeout is 300 seconds.
That said, since you have additional debug turned on, that would have
been the previous request.
Thus, only possibility is that there was a stuck request from long
before, or a background thread, doing stuff in a loop which at some
point finally didn't release the GIL.
Anyway, that is assuming that it is a user code problem. :-)
> I guess I could test this by setting 'threads=1'?
>
>> I can't see from logs, as not enough from before this point, whether
>> this occurred on inactivity timeout shutdown or whether distinct. My
>> guess is it is distinct as for inactivity timeout, where shutdown
>> timeout is 5 seconds by default, there is no way one could get to 300
>> seconds and trigger this.
>>
>> Can you explain more about how often/when you are seeing this deadlock
>> timeout in error logs. Something like that is not good as indicates a
>> badly implemented Python C extension module that doesn't release GIL
>> when doing blocking operations.
>
> I'm not using any Python C extensions, not directly at least. The only
> things (aside from the Python standard library) I use are CherryPy
> (3.1.2) and Genshi (0.5.1). It's a rather simple project up until now
> [1], currently with as only code a 125 line long Python script [2].
>
> As for how often I see this deadlock. I thought it to be related to the
> inactivity timeout because it appears to happen about 500 (+/- 120)
> seconds after the last HTTP request gets in.
But you have extra debug turned on from what I can tell and there is
no message that process is shutting down due to inactivity timeout. If
there was, then there is an outside possibility that during shutdown
of interpreter the deadlock detection code incorrectly detected a
deadlock as code doesn't check to see whether interpreter in process
of being shutdown. But then, GIL locking still should work even if
interpreter is being shutdown or has been shutdown as it is
effectively a separate lock. I'll still have a look over the code and
see if there is potential for false detection during shutdown.
Graham
> [1] http://git.il.fontys.nl/git/ilpam.git
> [2] http://git.il.fontys.nl/git/ilpam.git/blob/HEAD:/mgt/web/wsgi.py
>
> --
> With kind regards,
> Giel van Schijndel
> - Interlink <www.il.fontys.nl>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iEYEARECAAYFAktgKvQACgkQZBYm/87l50LJrQCaAg7+z1gtgVmSmEGs+zZ7N0fK
> lDcAoIEaYHqeFsRg7ngpSJ1ssHw4X9TP
> =K8EV
> -----END PGP SIGNATURE-----
>
>