Apache2 modwsgi

[Paul Kraus]

I have an internal web application that imports some records via csv and then imports them into our legacy systems database using ODBC. The click the import button it goes to just a normal @expose() method that does the import and when finished redirects to a results page. No multithread or async stuff going on. In development it just works without issues. On our production server it dies with an error 500 and watching the logs this is the error produced...

[core:error] [pid 45376:tid 139899256612608] [client 10.0.7.1:50186] End of script output before headers: app.wsgi

Kind of getting twisted around the axle here so any help would be greatly appreciated.

[Michael Pedersen]

I've seen similar, and the times I've seen it it's been file permissions. Check where the csv file is being written to, along with *all* the parent directories. Make sure that the user running the apache process has permissions to write to those directories.

--
You received this message because you are subscribed to the Google Groups "TurboGears" group.
To unsubscribe from this group and stop receiving emails from it, send an email to turbogears+...@googlegroups.com.
To post to this group, send email to turbo...@googlegroups.com.
Visit this group at http://groups.google.com/group/turbogears.
For more options, visit https://groups.google.com/d/optout.

[Paul Kraus]

I don't actually read or write the file from the file system. It's imported via a form and then processed. I examined the app log and that's what i received below but i just now tested it again and examined the apache error log and this happens at the exact same moment ...

[core:notice] [pid 45371:tid 139899616245632] AH00094: Command line: '/usr/sbin/apache2'

[core:notice] [pid 45371:tid 139899616245632] AH00051: child pid 45375 exit signal Segmentation fault (11), possible coredump in /etc/apache2

[core:notice] [pid 45371:tid 139899616245632] AH00051: child pid 55064 exit signal Segmentation fault (11), possible coredump in /etc/apache2

[Paul Kraus]

I thought it might be something different between my virtualenv on my dev box and the virtualenv being using by modwsgi on the production server. So i loaded the app up using gearbox serve -c development.ini with the same virtualenv as the production server, and no issues. So it only seems to blow up when using apache2/modwsgi.

Very frustrating.

[lebouquetin]

If your form uses a <input type="file"> for choosing a file, then the uploaded file is written to a temporary directory and what Michael said is that you might get an error because apache user is not allowed. Right access is a classical case of errors when wsgi-ing your app while running it with gearbox serve (which, I assume you run as root)

Damien

[Paul Kraus]

It does, and that's excellent. Would it keep the same file name, what would be the default location of the file?

Paul Kraus
IT Director / Project Management Director
(216) 916.9801 Direct Dial
(216) 267.6176 Fax
www.pelsupply.com

--
You received this message because you are subscribed to a topic in the Google Groups "TurboGears" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/turbogears/VHKKc0iN1F4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to turbogears+...@googlegroups.com.

[Paul Kraus]

Actually here are the order of events.

Form accepts file upload, file is parsed and written to a database table where error checking is done against the data to make sure its cleaned up and able to be sent to our legacy system. The user is then presented with a view of the data and any information they need to decide what changes should be bad to bad data. If everything is good they click import, it reads the records from the table, and using odbc writes them to the legacy system.

So its happening after all of the file stuff.

Paul Kraus
IT Director / Project Management Director
(216) 916.9801 Direct Dial
(216) 267.6176 Fax
www.pelsupply.com

[Michael Pedersen]

It could also be some sort of print statement in the code. sys.stdout and sys.stderr don't exist under mod_wsgi, but they do in gearbox.

You received this message because you are subscribed to the Google Groups "TurboGears" group.

To unsubscribe from this group and stop receiving emails from it, send an email to turbogears+...@googlegroups.com.

[Alessandro Molina]

On Tue, Jun 16, 2015 at 4:39 PM, Paul Kraus <pkr...@pelsupply.com> wrote:

[core:notice] [pid 45371:tid 139899616245632] AH00094: Command line: '/usr/sbin/apache2'

[core:notice] [pid 45371:tid 139899616245632] AH00051: child pid 45375 exit signal Segmentation fault (11), possible coredump in /etc/apache2

[core:notice] [pid 45371:tid 139899616245632] AH00051: child pid 55064 exit signal Segmentation fault (11), possible coredump in /etc/apache2

If Apache crashes when using mod_wsgi it is usually due to some python module that has C speedups/extensions that do not cope well with mod_wsgi.

One common reason is that the module uses the simplified GIL API, which only works in the main python interpreter and not in sub-interpreters (which mod_wsgi uses).

In many cases using "WSGIApplicationGroup %{GLOBAL}" inside your application configuration solves the issues.

It that doesn't help you might want to check you are not linking to multiple versions of the same library, for example if you have PHP in apache and also use MySQL in your python application you might end up with two different mysql versions being linked one from PHP and the other from python which leads to crashes. This can be checked by running "ldd" against the apache2 modules to see which library they are linking to.

In any case when you get a segfault is 90% related to a python module relying on C code which doesn't cope with mod_wsgi, ODBC might be a candidate, try uninstall the python odbc library and run your code against SQLITE to see if it still crashes.

[Paul Kraus]

On Tue, Jun 16, 2015 at 5:35 PM, Alessandro Molina <alessand...@gmail.com> wrote:

In many cases using "WSGIApplicationGroup %{GLOBAL}"

So i should trying putting "WSGIApplicationGroup %{GLOBAL}" in my production.ini file or app.conf. I am not familiar with what %{GLOBAL} does or this directive or do you mean in the apache config? Actually thinking this through you mean the apache config. Can you tell me what the GLOBAL does?

[Alessandro Molina]

https://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIApplicationGroup

It's a mod_wsgi option, so it should go in your apache virtualhost 

--

[Paul Kraus]

The link helped but didn't solve my issues. I reduced my threads from 5 to 1 with processes not set and that made it go away. To be honest i don't understand how processs and threads are used and what there meaning is in the  mod-wsgi settings. Does this now mean apache is using i a single process and thread to handle all requests? That can't be good. How are these variables used and what are the ramifications on how i have them set (processes ommited and threads=1).

Thank you very much for all of your help.

Paul

Paul Kraus
IT Director / Project Management Director
(216) 916.9801 Direct Dial
(216) 267.6176 Fax
www.pelsupply.com

--
You received this message because you are subscribed to a topic in the Google Groups "TurboGears" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/turbogears/VHKKc0iN1F4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to turbogears+...@googlegroups.com.

[Alessandro Molina]

On Wed, Jun 17, 2015 at 4:18 PM, Paul Kraus <pkr...@pelsupply.com> wrote:

The link helped but didn't solve my issues. I reduced my threads from 5 to 1 with processes not set and that made it go away. To be honest i don't understand how processs and threads are used and what there meaning is in the  mod-wsgi settings.

Processes define how many python instances to start

Threads define how many threads to start inside each python instance (but as threads in python suffer from the GIL it just provides a way to serve requests while the I/O is moving forward, so it has no great benefit to set it higher than 2-3-4-5 threads for each process).

 

Does this now mean apache is using i a single process and thread to handle all requests? That can't be good. How are these variables used and what are the ramifications on how i have them set (processes ommited and threads=1).

If threads=1 if means that each process will be able to serve a single request at time. If you have only 1 process with only 1 thread that means you can only serve 1 request at time.

Usually setting an high number of processes should not have any issues as they are totally independent and unrelated python instances, while using more than 1 thread might actually lead to problems (race conditions, crashes and so on) if your python code is not thread safe or relies on C extensions that are not thread safe or do not cope well with the GIL.

In case you have issues with C modules, it might often help to set an high number of processes and use a single thread for each one of them.

I still suspect that your problems are caused by pyodbc by the way

[Paul Kraus]

On Wed, Jun 17, 2015 at 4:41 PM, Alessandro Molina <alessand...@gmail.com> wrote:

In case you have issues with C modules, it might often help to set an high number of processes and use a single thread for each one of them.

I still suspect that your problems are caused by pyodbc by the way

Just want to start out with saying i can't thank you enough for taking time to help me work through this!

I am using mxodbc(commercial product) but your point stands and i 100% agree. If you leave off the processes=X declaration from mod_wsgi config, does it just let apache manage how many processes to spawn? Looking at processes running i see 5 /usr/sbin/apache2 running which (unless i am missing something) means apache has 5 processes running using 1 thread but i am kind of guessing. The systems piece of this is not my strong point.

I did read over this earlier today and based on what i read it shouldn't be the issues but the segfault occurs during the odbc calls.

15.1 Running mxODBC from a CGI script

ODBC drivers and managers are usually compiled as a shared library. When running CGI scripts most HTTP daemons (or web servers) don't pass through the path for the dynamic loader (e.g. LD_LIBRARY_PATH) to the script, thus importing the mxODBC C extension will fail with unresolved symbols because the loader doesn't find the ODBC driver/manager's libs.

To have the loader find the path to those shared libs you can either wrap the Python script with a shell script that sets the path according to your system configuration or tell the HTTP daemon to set or pass these through (see the daemon's documentation for information on how to do this; for Apache the directives are named SetEnvandPassEnv).

On Windows, you also have to take into account that the ODBC data sources defined in the ODBC manager are usually restricted to specific user accounts. You can work around this by either setting up the ODBC data sources for the web server service account or by configuring the data as system data sources.

15.2 Running mxODBC with mod_wsgi

Using mxODBC with mod_wsgi is generally possible. However, since the script will run under a restricted user account, some care has to be taken to make the setup work. Please see 15.1Running mxODBC from a CGI script for more details on getting ODBC drivers to work in such an environment.

mod_wsgi and Python 2.7

On Windows, there is also another issue to consider when running the combination Apache, mod_wsgi and Python 2.7. Due to changes in Python 2.7, manifests for the Visual C++ runtime environment, needed by Windows to find the right DLL to load, are no longer added to Python extensions, since this caused problems with loading them into Python processes (see Python Issue 4120).

Unfortunately, neither mod_wsgi nor Apache appear to include the required manifests either. This causes an import error when trying to load mxODBC into a mod_wsgi run process, since Windows cannot resolve the DLL references in mxODBC without the manifest.

Since this affects not only mxODBC, but other Python C extensions as well, you may want to use a work-around until either Apache or the mod_wsgi team solves the problem:

Manifest work-around

Adding the VC++ manifests to the Apache process is explained in this posting.

You will also have to install the MS VC++ 2008 CRT SP1 redistributable package on the server running Apache.

With those changes in place, mxODBC should load without problems.