Couldn't create worker

[Carl Nobile]

While writing a thumbnail service in Django I found the messages below
in the apache error log for the application. Has anybody ever seen
this before? Touching the Django .wsgi file didn't fix it only
restarting apache seemed to fix it.

We are using mod_wsgi version 2.3

~Carl
----------

[Wed Oct 28 15:01:22 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9449): Starting thread 4 in daemon process 'thumb'.
[Wed Oct 28 15:01:22 2009] [alert] (11)Resource temporarily
unavailable: mod_wsgi (pid=9449): Couldn't create worker thread 3 in
daemon process 'thumb'
.
[Wed Oct 28 15:01:27 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9449): Starting thread 5 in daemon process 'thumb'.
[Wed Oct 28 15:01:27 2009] [alert] (11)Resource temporarily
unavailable: mod_wsgi (pid=9449): Couldn't create worker thread 4 in
daemon process 'thumb'
.
[Wed Oct 28 15:01:32 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9449): Starting thread 6 in daemon process 'thumb'.
[Wed Oct 28 15:01:32 2009] [alert] (11)Resource temporarily
unavailable: mod_wsgi (pid=9449): Couldn't create worker thread 5 in
daemon process 'thumb'
.
[Wed Oct 28 15:01:37 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9449): Starting thread 7 in daemon process 'thumb'.
[Wed Oct 28 15:01:37 2009] [alert] (11)Resource temporarily
unavailable: mod_wsgi (pid=9449): Couldn't create worker thread 6 in
daemon process 'thumb'
.
[Wed Oct 28 15:01:42 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9449): Starting thread 8 in daemon process 'thumb'.
[Wed Oct 28 15:01:42 2009] [alert] (11)Resource temporarily
unavailable: mod_wsgi (pid=9449): Couldn't create worker thread 7 in
daemon process 'thumb'
.
[Wed Oct 28 15:01:47 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9449): Starting thread 9 in daemon process 'thumb'.
[Wed Oct 28 15:01:47 2009] [alert] (11)Resource temporarily
unavailable: mod_wsgi (pid=9449): Couldn't create worker thread 8 in
daemon process 'thumb'
.
[Wed Oct 28 15:01:52 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9449): Starting thread 10 in daemon process 'thumb'.
[Wed Oct 28 15:01:52 2009] [alert] (11)Resource temporarily
unavailable: mod_wsgi (pid=9449): Couldn't create worker thread 9 in
daemon process 'thumb'
.
[Wed Oct 28 15:01:57 2009] [info] mod_wsgi (pid=9449): Shutdown
requested 'thumb'.
[Wed Oct 28 15:01:57 2009] [alert] (11)Resource temporarily
unavailable: mod_wsgi (pid=9449): Couldn't create reaper thread in
daemon process 'thumb'.
[Wed Oct 28 15:01:57 2009] [info] mod_wsgi (pid=9449): Stopping
process 'thumb'.
[Wed Oct 28 15:01:57 2009] [info] mod_wsgi (pid=9449): Cleanup
interpreter ''.
[Wed Oct 28 15:01:57 2009] [info] mod_wsgi (pid=9449): Terminating
Python.
[Wed Oct 28 15:01:58 2009] [info] mod_wsgi (pid=9950): Attach
interpreter ''.
[Wed Oct 28 15:01:58 2009] [debug] mod_wsgi.c(8449): mod_wsgi
(pid=9950): Starting 10 threads in daemon process 'thumb'.
[Wed Oct 28 15:01:58 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9950): Starting thread 1 in daemon process 'thumb'.
[Wed Oct 28 15:01:58 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9950): Starting thread 2 in daemon process 'thumb'.
[Wed Oct 28 15:01:58 2009] [debug] mod_wsgi.c(8455): mod_wsgi
(pid=9950): Starting thread 3 in daemon process 'thumb'.
[Wed Oct 28 15:01:58 2009] [info] mod_wsgi (pid=9950): Enable monitor
thread in process 'thumb'.
[Wed Oct 28 15:01:58 2009] [debug] mod_wsgi.c(8301): mod_wsgi
(pid=9950): Deadlock timeout is 300.
[Wed Oct 28 15:01:58 2009] [debug] mod_wsgi.c(8304): mod_wsgi
(pid=9950): Inactivity timeout is 0.
[Wed Oct 28 15:01:58 2009] [info] mod_wsgi (pid=9950): Enable deadlock
thread in process 'thumb'.
[Wed Oct 28 15:01:58 2009] [alert] (11)Resource temporarily
unavailable: mod_wsgi (pid=9950): Couldn't create worker thread 2 in
daemon process 'thumb'.

[Carl Nobile]

This is in our apache conf file.

WSGISocketPrefix /tmp/wsgi

<VirtualHost *:80>
    ServerName thumb.scrippsing.com
    ServerAlias lb.thumb.scrippsing.com
    ServerAlias dev.thumb.scrippsing.com lb.dev.thumb.scrippsing.com
    ServerAlias stage.thumb.scrippsing.com lb.stage.thumb.scrippsing.com

    # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
    LogLevel debug
    ErrorLog /logs/apache/dev_thumb-error.log
    CustomLog /logs/apache/dev_thumb-access.log combined

    WSGIDaemonProcess thumb user=ellington group=ellington \
      processes=4 threads=10 maximum-requests=1000
    WSGIProcessGroup thumb

    WSGIScriptAlias / /opt/local/apps/apps-django-1.0/thumb/apache/app.wsgi

    Alias /media /opt/local/apps/apps-django-1.0/thumb/thumb/media

</VirtualHost>

--
-------------------------------------------------------------------------------
Carl J. Nobile (Software Engineer)
carl....@gmail.com
-------------------------------------------------------------------------------

[Graham Dumpleton]

2009/10/29 Carl Nobile <carl....@gmail.com>:

> This is in our apache conf file.
>
> WSGISocketPrefix /tmp/wsgi

It is not recommended that /tmp/wsgi be used as socket prefix.
Directory they are in should only be writable by root. This is
documented in:

http://code.google.com/p/modwsgi/wiki/ConfigurationIssues#Location_Of_UNIX_Sockets

>> While writing a thumbnail service in Django I found the messages below
>> in the apache error log for the application. Has anybody ever seen
>> this before? Touching the Django .wsgi file didn't fix it only
>> restarting apache seemed to fix it.
>>
>> We are using mod_wsgi version 2.3
>>
>> ~Carl
>> ----------
>>
>> [Wed Oct 28 15:01:22 2009] [debug] mod_wsgi.c(8455): mod_wsgi
>> (pid=9449): Starting thread 4 in daemon process 'thumb'.
>> [Wed Oct 28 15:01:22 2009] [alert] (11)Resource temporarily
>> unavailable: mod_wsgi (pid=9449): Couldn't create worker thread 3 in
>> daemon process 'thumb'

Likely from:

pthread_create() will fail if:

[EAGAIN] The system lacked the necessary resources to create
another thread, or the system-imposed limit on the
total number of threads in a process
[PTHREAD_THREADS_MAX] would be exceeded.

This suggests threads per process, but since those threads are only
created on startup one wouldn't expect that unless there could be
other system wide limits that were being hit.

Graham

[Carl Nobile]

Graham, It looks like that when the max thread count was hit my app started writing files as root user. Our sysadmins told me that we are using a fused file system where our media is located in our staging environment, but we are not doing this in production. The kernel module for this is fuse-unionfs.

So I am looking at possible two issues here. One when we have hit a max process/thread count which was 950 and has been reset to 1500 per user. And two that somehow apache/mod_wsgi dropped my app into root when the max thread count was reached or the fused file system caused the apps file creations to be done as root even though the app was not running as a non root user.

I really hope that neither apache or mod_wsgi caused this app to become root since that could be a serious security issue. I thought you should know in case this is a mod_wsgi issue. Though I find it hard to beleive that either apache or mod_wsgi could do that.

~Carl

[Graham Dumpleton]

2009/10/30 Carl Nobile <carl....@gmail.com>:

> Graham, It looks like that when the max thread count was hit my app started
> writing files as root user. Our sysadmins told me that we are using a fused
> file system where our media is located in our staging environment, but we
> are not doing this in production. The kernel module for this is
> fuse-unionfs.
>
> So I am looking at possible two issues here. One when we have hit a max
> process/thread count which was 950 and has been reset to 1500 per user. And
> two that somehow apache/mod_wsgi dropped my app into root when the max
> thread count was reached or the fused file system caused the apps file
> creations to be done as root even though the app was not running as a non
> root user.
>
> I really hope that neither apache or mod_wsgi caused this app to become root
> since that could be a serious security issue. I thought you should know in
> case this is a mod_wsgi issue. Though I find it hard to beleive that either
> apache or mod_wsgi could do that.

If it mod_wsgi that wasn't dropping privileges properly, then you
would have expected to at least see a logged message.

/* Setup the daemon process real and effective user. */

if (setuid(daemon->group->uid) == -1) {
ap_log_error(APLOG_MARK, WSGI_LOG_ALERT(errno), wsgi_server,
"mod_wsgi (pid=%d): Unable to change to uid=%ld.",
getpid(), (long)daemon->group->uid);
}

The only errors on MacOS X returned by setuid are:

[EINVAL] The value of the {group,user} ID argument is invalid
and is not supported by the implementation.

[EPERM] The process does not have appropriate privileges and
the ID argument does not match the real ID or the
saved set-{group,user}-ID.

EPERM shouldn't be able to happen, as don't execute this code if not
running as root.

EINVAL could technically happen if user account was deleted between
time that Apache was started and when daemon process is forked.
Because of daemon process restarts at later time, that is actually a
big window.

Even though the latter is not generally going to happen unless system
administrator acts without thinking, I should perhaps on any failure
actually kill the daemon process rather than carry on, because right
now could carry on as root. Possibly better to not even check for
failure, but later on before initiating main loop that see if still
running as root for any reason and just kill off process.

Anyway, any issue in this space would mean that process was always
running as root and not that it just changed back to root somehow
later when resources started to get exhausted. It doesn't seem to
explain your particular problem so looks more likely an issue with the
file system kernel module you are using getting corrupted when start
having resource issues.

Graham

[Jason Garber]

On Thu, Oct 29, 2009 at 7:32 PM, Graham Dumpleton <graham.d...@gmail.com> wrote:

Even though the latter is not generally going to happen unless system
administrator acts without thinking, I should perhaps on any failure
actually kill the daemon process rather than carry on, because right
now could carry on as root.

+1 for the following change:

 

Possibly better to not even check for
failure, but later on before initiating main loop that see if still
running as root for any reason and just kill off process.

JG 

[Graham Dumpleton]

2009/10/31 Jason Garber <bo...@gahooa.com>:

Looking at this further, not sure there is an issue. At least not
anything one can do anything about.

The setuid() function will only fail if the uid is out of range.

What exactly that means is a bit vague but presumably that means a uid
outside of range of what system allows. It doesn't mean a uid for
which no account exists at the time. It looks like you can still
change to a uid even if no account exists matching that uid.

To even get that far in code the user given by 'user' option, whether
by name or by "#nnn" would already have been validated in various ways
and certainly know whether or not it was in valid range.

The real problem comes later as process crashes anyway. This can seem
to happen in various places, so my guess is that something caches
password file entry information or something and that gets invalidated
and later attempt to use it causes crash.

Anyway, seems that not much I can do about it. It is just a very nasty
thing to do, remove account while something wanting to run as that
user, and likely that a lot of programs aren't going to behave well.

Important thing is that process will not simply run as root.

Graham

[Carl Nobile]

I was curious if mod_wsgi could drop me in to root, as it seems it cannot do that. This is good to know since it eliminated one thing from our issue list. As it happens the fuse-unionfs seems to do this all on it's own. As I worked through this issue I talked to other engineers here at Scripps that have had similar issues. Though this has nothing to do with Apache or mod_wsgi people should be aware of this.

The only confusing part of all this is why was mod_wsgi not able to create threads at the same time fuse-unionfs wrote files with root permissions. Luckly our prod environment doesn't use fuse-unionfs, so we should be safe there.

~Carl