How do you find if fcntl or flock is being used

[kusimari share]

Hi,

I have two instances of a apache, mod_wsgi and flask based http rest service. The only difference between the two services are the files being loaded (python pickle files on flask start) and AWS regions, one is in US while the other is in EU. The flask service itself is a wrapper over scikit learn predictions which are the reason for the python pickle files mentioned earlier.

Confusingly the instance in US works fine. No issues. However the EU instance keeps getting into a deadlock situation because of which mod_wsgi restarts. Apache is still up. I have had no luck using the different mod_wsgi debug mechanisms, apparently there is no deadlock in the flask, scikit learn side of the equation.

Today, found that on deadlock mod_wsgi emitted:

21720 [Mon Aug 29 21:31:26.979825 2016] [wsgi:crit] [pid 4317:tid 140495687612160] (35)Resource deadlock avoided: │EOE

      mod_wsgi (pid=4317): Couldn't acquire accept mutex '/path-to-apache/var/state.14843.0.1.sock'. Shutting down daemon process 

Apache conf contains

Mutex file:/path-to-pache/var/state mpm-accept

My suspicion is that one of the instance is using fcntl while the other uses flock, and probably the one using fcntl has deadlocks based on apache documentation. But I have no way to verify it.

Is there a way to find which of fcntl or flock is being used by Apache?

httpd -V shows

 -D APR_HAS_SENDFILE

 -D APR_HAS_MMAP

 -D APR_USE_SYSVSEM_SERIALIZE

 -D APR_USE_PTHREAD_SERIALIZE

 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT

 -D APR_HAS_OTHER_CHILD

 -D AP_HAVE_RELIABLE_PIPED_LOGS

 -D DYNAMIC_MODULE_LIMIT=256

Thanks

Santhosh

[Graham Dumpleton]

If you start up the Apache web server with LogLevel set to ‘debug’, you should see a line like:

    [mpm_prefork:debug] [pid 82122] prefork.c(1027): AH00165: Accept mutex: none (default: flock)

I think this is what mod_wsgi will also use. There was an issue at some point where Apache stopped allowing me to see the choice it made and so I had to start calculating it myself.

That in itself shouldn’t cause a problem though as Apache’s own cross process mutex locks are separate to those mod_wsgi creates, so there cannot be a clash if types end up being different.

I would suggest the issue may be an ownership/permissions issue on the directory rather than a mismatch of lock types. The locks on this are all managed in mod_wsgi so can’t see how the type of locking used could differ at different times.

So what are the ownership/permissions on the directory:

    /path-to-apache/var/

Can you try setting the directive:

    WSGISocketPrefix /tmp

as a test to see if that helps. The sockets and lock files should then be placed in /tmp instead, which if this is a single user machine should be fine.

You can also override what the mutex type used by mod_wsgi  for its own locks is by using:

    WSGIAcceptMutex flock

Graham

[kusimari share]

Ok, I will try that and post back.

Difficulty has been due to the fact that it happens only in one instance and not in the other. That is on the box which is deployed on linux in AWS US, never see any deadlocks even though the number of requests are higher. 

However in the box deployed on linux in AWS EU, no matter the number of process and threads, the deadlock always happens. Further, if the number of processes and threads are increased the deadlock happens frequently. As of now I have restricted to 2 process and 15 threads for WSGI. So the system chugs along fine for 3-8 hours and then goes into a tail spin of deadlock/restart for about 3 or 4 times and then it continues again for 3-8 hours.

In the AWS US box, it never deadlocks and goes on for more than a couple week.

Is there a way query the .sock file? Something similar to cat /proc/locks?

Santhosh

[kusimari share]

Some additional data

Apache debug logs show:

[mpm_worker:debug] [pid 6870:tid 140091454019328] worker.c(1829): AH00294: Accept mutex: fcntl (default: sysvsem)

This I assume implies: mpm worker uses fcntl while mod_wsgi uses sysvsem. With this default /path-to-apache/var contained only .sock file.

I added two lines to apache conf

Mutex file:/path-to-apache/var/state mpm-accept

WSGIAcceptMutex flock

Now apache debug logs show:

[mpm_worker:debug] [pid 15647:tid 140091454117350] worker.c(1829): AH00294: Accept mutex: flock (default: sysvsem)

And /path-to-apache/var contains both .sock and .lock file.

I still don't know if this will eliminate deadlocks happening in one box while the other box chugs away happily without issues. On the box which has issues, deadlock reason seems to vary. Saw these in the logs at different times. Don't know why the reason for deadlock also changes. (Note this is using fcntl default sysvsem)

• Logs for the most common of the deadlock situations. Note that pid for the timeout and deadlock timer are different.

[wsgi:error] [pid 11561:tid 139804096313088] [client 10.4.71.118:28639] Timeout when reading response headers from daemon process 'app': /path-to-apache/bin/app.wsgi

[wsgi:info] [pid 11559:tid 139804272707328] mod_wsgi (pid=11559): Daemon process deadlock timer expired, stopping process 'app'.

[wsgi:info] [pid 11559:tid 139804333795072] mod_wsgi (pid=11559): Shutdown requested 'app'.    

• Logs for one of the deadlock situation:

[wsgi:crit] [pid 4317:tid 140495687612160] (35)Resource deadlock avoided: mod_wsgi (pid=4317): Couldn't acquire accept mutex '/path-to-apache/var/state.14843.0.1.sock'. Shutting down daemon process

• One of the rare situation which I saw in passing, but did not grab the log quickly enough, showed logging api failure to lock log file

I am not sure what's at fault. If it was due to fcntl and sysvsem, then that should have happened in both the boxes and not just one. The only two difference between the two boxes are

 - One is in AWS US region and the other is in AWS EU region. Though uname and httpd -V shows both being the same.

 - The US region loads different set of scikit-learn pickles than the EU region one. But both pickles are exports from the same scikit-learn classes (SGDClassifier)

Mysterious at best. Any clues would help. Definitely don't want to try using gunicorn/nginx just for experimenting if something in apache is messing things up.

Thanks

Santhosh

[Graham Dumpleton]

On 30 Aug 2016, at 4:17 PM, kusimari share <kusimar...@gmail.com> wrote:

Some additional data

Apache debug logs show:

[mpm_worker:debug] [pid 6870:tid 140091454019328] worker.c(1829): AH00294: Accept mutex: fcntl (default: sysvsem)

This I assume implies: mpm worker uses fcntl while mod_wsgi uses sysvsem. With this default /path-to-apache/var contained only .sock file.

If using Apache 2.4, then maybe.

The logic in mod_wsgi is:

#if !defined(AP_ACCEPT_MUTEX_TYPE)

    sconfig->lock_mechanism = ap_accept_lock_mech;

#else

    sconfig->lock_mechanism = APR_LOCK_DEFAULT;

#endif

From memory they got rid of ap_accept_lock_mech and is impossible to work out what Apache used.

So mod_wsgi will use default. If the Apache configuration override default explicitly, then quite likely different.

As I said, it shouldn’t matter unless the locking mechanism is somehow broken or unreliable for that platform.

I added two lines to apache conf

Mutex file:/path-to-apache/var/state mpm-accept

WSGIAcceptMutex flock

So that will guarantee using the same, presuming that Apache was overriding it and flock is more reliable than sysvsem on your system.

I personally have never seen it, but have seen one person complain that sysvsem were unreliable in Apache because if Apache crashed, or had kill -9 done on it, the sysvsem wouldn’t be cleaned up properly and you would run out of them. Who does kill -9 on Apache and then restarts it and if Apache as a whole was crashing you would know about it.

Now apache debug logs show:

[mpm_worker:debug] [pid 15647:tid 140091454117350] worker.c(1829): AH00294: Accept mutex: flock (default: sysvsem)

And /path-to-apache/var contains both .sock and .lock file.

I still don't know if this will eliminate deadlocks happening in one box while the other box chugs away happily without issues. On the box which has issues, deadlock reason seems to vary. Saw these in the logs at different times. Don't know why the reason for deadlock also changes. (Note this is using fcntl default sysvsem)

• Logs for the most common of the deadlock situations. Note that pid for the timeout and deadlock timer are different.

[wsgi:error] [pid 11561:tid 139804096313088] [client 10.4.71.118:28639] Timeout when reading response headers from daemon process 'app': /path-to-apache/bin/app.wsgi

[wsgi:info] [pid 11559:tid 139804272707328] mod_wsgi (pid=11559): Daemon process deadlock timer expired, stopping process 'app'.

This means your Python code is deadlocking, not Apache or mod_wsgi code. This is usually caused by Python C extension modules that aren’t written properly so they work in sub interpreters.

Make sure that in using daemon mode you are delegating the WSGI application to run in the application group %{GLOBAL}.

What is the mod_wsgi setup you are using?

From memory some of the numpy/scipy stuff has this exact problem and this is required.

See:

    http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#python-simplified-gil-state-api

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[kusimari share]

WSGI setup uses global and mpm_worker.

# apache workers config to start 2 server with 75 max requests.

# changes to workload implies that the below needs to be tuned

<IfModule mpm_worker_module>

    StartServers          2

    MinSpareThreads      10

    MaxSpareThreads      35

    ThreadsPerChild      25

    MaxClients           75

    MaxRequestsPerChild   0

 </IfModule>

Mutex flock:/patch-to-apache/var/state mpm-accept 

 

WSGIScriptReloading Off

WSGISocketPrefix /path-to-apache/var/state

WSGIPassAuthorization On

# see mod_wsgi documentation on embedded vs daemon

WSGIRestrictEmbedded On

# force wsgi to use flock instead of default. added to check fcntl vs sysvsem vs flock

WSGIAcceptMutex flock

# Port configuration

Listen 8080

<VirtualHost *:8080>

    ServerName server-name:8080

    UseCanonicalName Off

    #app configuration for current load at 10k+ classification reqs / sec

    WSGIDaemonProcess app processes=3 threads=30 display-name=atlas

    WSGIScriptAlias / /patch-to-apache/bin/app.wsgi

    WSGIProcessGroup app

    WSGIApplicationGroup %{GLOBAL}

    <Directory /path-to-apache/bin/>

        Order deny,allow

        Allow from all

    </Directory>

</VirtualHost> 

My initial suspicion was something in scikit/numpy/scipy/nltk was the reason for deadlock, even though I was using GLOBAL following http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#python-simplified-gil-state-api. However, if that is correct, then the deadlock should happen in both boxes as both use the same python flask app code.

From apache conf, my later suspicion was that prefork was an issue, so loaded only mpm_worker. That delays the deadlock, but it happens never the less.

Don't know if both Mutex and WSGISocketPrefix pointing to the same path has issues. Again if it was an issue, then both boxes should deadlock.

Finally, deadlocks happen all of a sudden without a pattern. Its NOT triggered because of increased load. And once deadlock is triggered, then I see that the server goes for a spin. A spin is repeated deadlock by each of the process for about 3 or 4 times post which they all settle down and work fine. This is what made me suspect that locks was a problem. If not why would deadlocks happen across all process/threads at the same time and continue for 3 or 4 cycles and then settle down.

I followed the touch to thread dump method at http://modwsgi.readthedocs.io/en/develop/user-guides/debugging-techniques.html, but getting thread dumps when I saw apache freezing up did not give anything meaningful to debug too.

-Santhosh

[Graham Dumpleton]

On 30 Aug 2016, at 4:56 PM, kusimari share <kusimar...@gmail.com> wrote:

WSGI setup uses global and mpm_worker.

# apache workers config to start 2 server with 75 max requests.

# changes to workload implies that the below needs to be tuned

<IfModule mpm_worker_module>

    StartServers          2

    MinSpareThreads      10

    MaxSpareThreads      35

Min and Max spare threads should really be multiples of ThreadsPerChild.

Apache doesn’t dynamically grow and shrink at thread level, but by process. Thus it either increments in batches of ThreadsPerChild threads at a time. Not having Min and Max as a multiple of that, can cause strange behaviour with Apache’s mechanism for dynamically adjusting the number of worker processes it uses.

Because though you are using daemon mode and disable Python in the worker processes, the problems with the potential increased process churn aren’t likely not going to be as significant, but still better to perhaps use:

MinSpareThreads 25

MaxSpareThreads 50

    ThreadsPerChild      25

    MaxClients           75

    MaxRequestsPerChild   0

 </IfModule>

Mutex flock:/patch-to-apache/var/state mpm-accept 

 

WSGIScriptReloading Off

WSGISocketPrefix /path-to-apache/var/state

It doesn’t matter that this is using same directory as even if using file based lock files, the naming convention for mod_wsgi lock files doesn’t clash.

WSGIPassAuthorization On

# see mod_wsgi documentation on embedded vs daemon

WSGIRestrictEmbedded On

# force wsgi to use flock instead of default. added to check fcntl vs sysvsem vs flock

WSGIAcceptMutex flock

# Port configuration

Listen 8080

<VirtualHost *:8080>

    ServerName server-name:8080

    UseCanonicalName Off

    #app configuration for current load at 10k+ classification reqs / sec

    WSGIDaemonProcess app processes=3 threads=30 display-name=atlas

High numbers of threads in daemon mode is always worrying. Especially if you are doing compute intensive tasks which are more CPU bound that I/O bound, cramming too many threads in a process can be a bad idea as can cause GIL contention. I would only use a higher number of threads in one process if I/O bound and even then would be careful about increasing it too much.

There are metrics that one can get from recent mod_wsgi versions which can tell you if you are over specifying the number of threads and are creating more than you need, but you need a monitoring system to report them into. Also, even if creating more than required, only wastes a bit of memory for I/O bound as mod_wsgi uses them in a way that if doesn’t need that many threads, will use threads it has used before and leave unused threads dormant if it can.

So is code your code more I/O bound vs CPU bound? I generally like to use in range of 3-5 threads per process if more CPU intensive.

    WSGIScriptAlias / /patch-to-apache/bin/app.wsgi

    WSGIProcessGroup app

    WSGIApplicationGroup %{GLOBAL}

    <Directory /path-to-apache/bin/>

        Order deny,allow

        Allow from all

    </Directory>

</VirtualHost> 

My initial suspicion was something in scikit/numpy/scipy/nltk was the reason for deadlock, even though I was using GLOBAL following http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#python-simplified-gil-state-api. However, if that is correct, then the deadlock should happen in both boxes as both use the same python flask app code.

From apache conf, my later suspicion was that prefork was an issue, so loaded only mpm_worker. That delays the deadlock, but it happens never the less.

Don't know if both Mutex and WSGISocketPrefix pointing to the same path has issues. Again if it was an issue, then both boxes should deadlock.

Finally, deadlocks happen all of a sudden without a pattern. Its NOT triggered because of increased load. And once deadlock is triggered, then I see that the server goes for a spin. A spin is repeated deadlock by each of the process for about 3 or 4 times post which they all settle down and work fine. This is what made me suspect that locks was a problem. If not why would deadlocks happen across all process/threads at the same time and continue for 3 or 4 cycles and then settle down.

I am not sure right now. I have seen various strange things over the years with deadlocking systems, but nothing like you are describing. Have had funny ones where someones database backups were locking everything up for 5-10 minutes every night. I have also seen stories, albeit not lockups, where when running on AWS, other customers of AWS whose applications were on same host were somehow cross impacting an application because of how the virtualisation worked for the shared system.

All I can suggest right now is reviewing what you have for processes/threads for daemon process groups and work out whether that is appropriate. I never like the high thread count per process.

Graham

[kusimari share]

Thanks Graham.

It took me a week to get to a stability where I can get deadlock happen only once or twice a day. Will need to see deeper on what is tripping the system. BTW, the jobs are all CPU bound. Once the request is parsed, then all the logic is in matrix opearations in scikit/numpy/scipy/nltk. Of course logs are the only IO ops.

As next steps:

 - Will increase process count while leaving threads at 5-6

 - Use flock for both mpm and wsgi

Let's see how successful these turn out. Will update the thread +ve or -ve.

Thanks

Santhosh

[Graham Dumpleton]

The implications of it being CPU bound when using numpy may not be so bad given that the design of bumpy is such as to ensure that the GIL is released when doing the CPU bound stuff. This is possible because it uses its own C level data structures rather than Python data structures which would need the GIL to still be held.

Graham