Got errors when installing Tethys 1.2 in production mode on Centos

[sdc50]

Hi Scott and Nathan,

I performed two clean production installations of tethys 1.2 on local centos6.6 VMs and got errors on both. 

1) WSGI 3.5 (appsdev and apps are using the same version of wsgi)

[Tue Sep 01 00:10:48 2015] [error] /var/www/.tethyscluster

[Mon Aug 31 18:10:48 2015] [error] [client 10.2.115.79] Premature end of script headers: wsgi.py, referer: http://10.2.115.214/

This is exactly the same error I got when trying to upgrade another local production VM from 1.1 to 1.2

2) WSGI 4.4

[Mon Aug 31 20:07:38 2015] [error] /var/www/.tethyscluster

[Mon Aug 31 20:07:38 2015] [error] [client 10.2.115.79] Truncated or oversized response headers received from daemon process 'tethys_default'': /usr/lib/tethys/src/tethys_portal/wsgi.py, referer: http://10.2.115.202/

Looks like both errors are somehow related to  /var/www/.tethyscluster. Is there a way that we can temporally disable tethyscluster feature?

Thanks

Drew

[sdc50]

I don't know of a way to deactivate the TethysCluster feature. You could try commenting out tethys_compute under INSTALLED_APPS in your settings.py file, but I think that will cause some import errors.

I'm not convinced that TethysCluster is at the root of the problem. Have you looked at this discussion on StackOverflow: http://stackoverflow.com/questions/24829085/truncated-or-oversized-response-headers-received-from-daemon-process?answertab=votes#tab-top

The first response by Graham Dumpelton (the developer of mod_wsgi) seems like it is describing the problem you're having. Let me know if that leads to anything helpful.

[zyli2004]

Scott,

Can you please send me a copy of the apache config file "tethys-default.conf" on your redhat production server? on my centos it is at /etc/httpd/conf.d/tethys-default.conf. Also, what versions of apache and wsgi are you using on that server? My centos apache is 2.2.15 and wsgi is 3.5 or 4.4.

Thanks

Drew 

[sdc50]

The RedHat server is running Apache/2.2.15 and mod_wsgi/4.2.4 with Python/2.7.6.

Here is the config file:

WSGISocketPrefix run/wsgi

<VirtualHost 0.0.0.0:80>

    ServerName demo.tethysplatform.org

    ServerAlias demo.tethysplatform.org

    Alias /static/ /var/www/tethys/static/

    <Directory /var/www/tethys/static/>

        #Require all granted

    </Directory>

    WSGIScriptAlias / /usr/lib/tethys/src/tethys_portal/wsgi.py

    <Directory /usr/lib/tethys/src/tethys_portal>

        <Files wsgi.py>

            #Require all granted

        </Files>

    </Directory>

    # Daemon config

    WSGIDaemonProcess tethys_default \

     python-path=/usr/lib/tethys/src/tethys_portal:/usr/lib/tethys/lib/python2.7/site-packages:/var/www/tethys/apps

    WSGIProcessGroup tethys_default

    # Logs

    ErrorLog /var/log/httpd/tethys_default.error.log

    CustomLog /var/log/httpd/tethys_default.custom.log combined

</VirtualHost>

It looks like the first line is not included in the Tethys installation docs. It may be something that is only needed for enterprise Linux.

[zyli2004]

I compared these cfg files, nothing different.

In the production instruction there is a section about the "Setup Email Capabilities", which requires: 

1) install and config postfix. 

2) activate some settings in /usr/lib/tethys/src/tethys_apps/settings.py. 

I just went through step 1), so the settings in  settings.py are still disabled. Will this affect something?

thanks

Drew

[Nathan Swain]

The email settings are optional, so it should not affect anything to have postfix installed and email disabled.

--

Nathan Swain, PhD

Aquaveo

801.709.2943

[zyli2004]

Scott and Nathan,

Sad to say, I am still having this error....

I increased the "header-buffer-size" to 999999 as suggested in this post (http://stackoverflow.com/questions/24829085/truncated-or-oversized-response-headers-received-from-daemon-process?answertab=votes#tab-top), but no luck. Also, I tried mod_wsgi 3.5, 4.2 and 4.4 (latest).

I notice section "4. Set WSGI Python Home" in tethys 1.2 production instruction. But there is no file "/etc/apache2/apache2.conf" on CentOS, so I add line "WSGIPythonHome /usr/local/pythonenv/BASELINE" to file "/etc/httpd/conf.d/tethys-default.conf". I don't see this line on RedHat's config file you sent me.

Now my tethys-default.conf file looks like:

WSGISocketPrefix run/wsgi

WSGIPythonHome /usr/local/pythonenv/BASELINE

WSGIPythonPath /usr/lib/tethys/lib/python2.7:/usr/lib/tethys/lib/python2.7/site-packages:/var/www/tethys/apps

<VirtualHost 0.0.0.0:80>

    ServerName 10.2.115.206

    ServerAlias 10.2.115.206 

    Alias /static/ /var/www/tethys/static/

    <Directory /var/www/tethys/static/>

        #Require all granted

    </Directory>

    WSGIScriptAlias / /usr/lib/tethys/src/tethys_portal/wsgi.py

    <Directory /usr/lib/tethys/src/tethys_portal>

        <Files wsgi.py>

            #Require all granted

        </Files>

    </Directory>

    # Daemon config

    WSGIDaemonProcess tethys_default \

     python-path=/usr/lib/tethys/src/tethys_portal:/usr/lib/tethys/lib/python2.7/site-packages:/var/www/tethys/apps \

     header-buffer-size = 999999

    WSGIProcessGroup tethys_default

    #Logs

    ErrorLog /var/log/httpd/tethys_default.error.log

    CustomLog /var/log/httpd/tethys_default.custom.log combined

</VirtualHost>

Another thing may be related to this error: Scott you told me you directly upgraded RetHat's built-in python 2.6.6 to python 2.7.6, right? so you have only one python instance (I have two on centos). Do you have some notes on how to upgrade it? I installed a separate python instance because it is said direct upgrade will break the "yum" command. but we need "yum" to install other dependencies.

Thanks

Drew

[Scott Christensen]

Drew,

I'm not sure how we got the system Python on our RedHat machine upgraded to 2.7. I think Nathan was worked with a system administrator at the UoU where our server is located to get things to work.

Nathan,

Do you remember how you got things to work on RedHat?

--
You received this message because you are subscribed to a topic in the Google Groups "Tethys Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tethysplatform/rbZmC5i8zxs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tethysplatfor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tethysplatform/e9c364d5-0f4b-4456-b3cb-c913b327b90b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[zyli2004]

I just wrote a "hello world" app to test the mod_wsgi according to this post https://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide. And the "hello world" is working.

So I think my mod_wsgi is installed properly. I am guessing are there any new dependencies of tethys 1.2, like the libraries listed in requirements.txt, may cause the WSGI problem?

[zyli2004]

Update: I think this is the problem we are facing...

Quoted from https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions

Q1: Why when first request is made against a WSGI application does the Apache server process handling the request crash with a 'segmentation fault'?

A1: This is nearly always caused due to a shared library version conflict. That is, Apache or some Apache module is linked against a different version of a library than that which is being used by a particular Python module that the WSGI application makes use of. The most common culprits are the expat and MySQL libraries, but it can also occur with other shared libraries.

Another cause of a process crash only upon the first request can be a third party C extension module for Python which has not been implemented so as to work within a secondary Python sub interpreter. The Python bindings for Subversion are a particular example, with the Python module only working correctly if the WSGI application is forced to run within the first interpreter instance created by Python.

Further information on these problems can be found in various sections of Application Issues. The problems with the expat library are also gone into in more detail in Issues With Expat Library.

Q2: Why am I seeing the error message 'premature end of script headers' in the Apache error logs.

A2: If using daemon mode, this is a symptom of the mod_wsgi daemon process crashing when handling a request. You would probably also see the message 'segmentation fault'. See answer for question about 'segmentation fault' above.

----------------------------------------------------------------------------------------------------------

We got Q1 logged in /etc/httpd/logs/error_log.txt

[Wed Sep 02 14:44:26 2015] [notice] child pid 3139 exit signal Segmentation fault (11)

[Wed Sep 02 14:44:26 2015] [info] mod_wsgi (pid=3139): Process 'tethys_default' has died, deregister and restart it.

[Wed Sep 02 14:44:26 2015] [info] mod_wsgi (pid=3139): Process 'tethys_default' terminated by signal 11

[Wed Sep 02 14:44:26 2015] [info] mod_wsgi (pid=3139): Process 'tethys_default' has been deregistered and will no longer be monitored.

[Wed Sep 02 14:44:26 2015] [info] mod_wsgi (pid=3173): Starting process 'tethys_default' with uid=48, gid=48 and threads=15.

[Wed Sep 02 14:44:26 2015] [info] mod_wsgi (pid=3173): Python home /usr/local/pythonenv/BASELINE.

[Wed Sep 02 14:44:26 2015] [info] mod_wsgi (pid=3173): Initializing Python.

Also, we got Q2 logged in /etc/httpd/logs/ tethys_default.error.log:

[Wed Sep 02 18:44:25 2015] [error] /var/www/.tethyscluster

[Wed Sep 02 14:44:26 2015] [error] [client 10.2.115.202] Premature end of script headers: wsgi.py, referer: http://10.2.115.214/

Looks like having two different python instances can cause a lot of problems.... 

[Nathan Swain]

Are the versions of your BASELINE Python environment and the Tethys Python environment the same? Also, is your Mod WSGI compiled for that version? You may need to get the source for Mod WSGI and compile it against the version of Python you using.

--
You received this message because you are subscribed to the Google Groups "Tethys Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tethysplatfor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tethysplatform/939dc1dc-c52d-4fe8-aed0-1dd289d44f37%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nathan Swain

[zyli2004]

Yes, I installed python 2.7.6 and then pip before installing  tethys and mod_wsgi . Tethys (/usr/lib/tethys) and WSGI (/usr/local/pythonenv/BASELINE) are using different virtual environments, but both of them are created from the same python 2.7.6. Also, mod_wsgi is compiled against BASELINE.

[zyli2004]

Nathan,

I have a question: Why do we need two python virtual environments? one for development  and one for production. In development mode, we already used pip to install tethys dependencies. But in production mode, we used another python virtual environment to install mod_wsgi? Can mod_wsgi access to the libraries we installed into development python environment?

Thanks

Drew

[Scott Christensen]

I can't answer the question about why two virtual environments are needed.

On a different but related note, Ezra is going to be attacking the mod_wsgi problem on our end. Hopefully together we can figure out how to make this work on CentOS. To help bring him up to speed it would be helpful if you could share any notes that you have on installing the development environment of Tethys. Particularly, if you know which packages are needed as dependencies on CentOS that would be helpful.

Thanks!

You received this message because you are subscribed to a topic in the Google Groups "Tethys Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tethysplatform/rbZmC5i8zxs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tethysplatfor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tethysplatform/5fdf0d26-23aa-4963-b9f5-5f05e15e4fba%40googlegroups.com.

[zyli2004]

I got a email from Nathan about the two python virtual environments. I pasted it and my reply below so we could keep tracking all discussions and updates regarding this issue.

From: Nathan Swain
Date: Thu, Sep 3, 2015 at 8:41 AM
Subject: Re: Got errors when installing Tethys 1.2 in production mode on Centos
To: zyli2004 

The BASELINE environment was a recommendation of working with virtual environments with mod_wsgi. Basically, it is to prevent conflicts with any dependencies that your system Python may be using. You don't really want to mess with the system Python, because the operating system uses it for maintenance operations. You can read more about the BASELINE environment here:

https://code.google.com/p/modwsgi/wiki/VirtualEnvironments#Baseline_Environment

How did installation on CentOS 7 go?

---------------------------------------------------------------------------------------------------------------------------------------------

From: Zhiyu Li 
Date: Thu, Sep 3, 2015 at 9:01 AM
Subject: Re: Got errors when installing Tethys 1.2 in production mode on Centos
To: Nathan Swain 

OK,  I want to make sure I didn't misunderstand something. So two or more virtual environments based on one python instance (either system python or user installed python) will not cause problem, because they are pointing to the same python instance. 

The situation mentioned in https://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions is about two different python instances, like one system python 2.6.6 and one user installed python 2.7.6, right?

I am working on CentOS 7 now, will keep you updated.

Thanks

Drew.

[zyli2004]

Some updates on CentOS 7:

1. I got a same error as described in https://github.com/tethysplatform/tethys/issues/96. Workaround is I need to manually start postgis container:   sudo docker run -d -p 5435:5432 --restart=always --name postgis ciwater/postgis 

2.  The apache version on CentOS 7 is 2.4.6 while it is 2.2 on CentOS 6.6/6.7. Looks like some changes on /etc/httpd/conf.d/tethys-default.conf are needed because 2.4 updates its access control policy ( see: http://httpd.apache.org/docs/2.4/upgrading.html)

So now the conf looks like (change are in red):

WSGISocketPrefix run/wsgi

WSGIPythonHome /usr/local/pythonenv/BASELINE

WSGIPythonPath /usr/lib/tethys/lib/python2.7:/usr/lib/tethys/lib/python2.7/site-packages:/var/www/tethys/apps

<VirtualHost 0.0.0.0:80>

    ServerName 10.2.115.212

    ServerAlias 10.2.115.212

    Alias /static/ /var/www/tethys/static/

    <Directory /var/www/tethys/static/>

         Require all granted

    </Directory>

    WSGIScriptAlias / /usr/lib/tethys/src/tethys_portal/wsgi.py

    <Directory /usr/lib/tethys/src/tethys_portal>

        <Files wsgi.py>

              Require all granted

        </Files>

    </Directory>

    # Daemon config

    WSGIDaemonProcess tethys_default \

     python-path=/usr/lib/tethys/src/tethys_portal:/usr/lib/tethys/lib/python2.7/site-packages:/var/www/tethys/apps

    WSGIProcessGroup tethys_default

    #Logs

    ErrorLog /var/log/httpd/tethys_default.error.log

    CustomLog /var/log/httpd/tethys_default.custom.log combined

</VirtualHost>

Also I upgraded Apache LogLevel form "warn" to "info" (/etc/httpd/conf/httpd.conf). Tried accessing tethys, i got these:

tethys_default.error.log

[Thu Sep 03 13:00:36.965127 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Attach interpreter ''.

[Thu Sep 03 13:00:36.965241 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Adding '/usr/lib/tethys/src/tethys_portal' to path.

[Thu Sep 03 13:00:36.965389 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Adding '/usr/lib/tethys/lib/python2.7/site-packages' to path.

[Thu Sep 03 13:00:36.966498 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Adding '/var/www/tethys/apps' to path.

[Thu Sep 03 13:05:27.088434 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Create interpreter '10.2.115.212|'.

[Thu Sep 03 13:05:27.089531 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Adding '/usr/lib/tethys/src/tethys_portal' to path.

[Thu Sep 03 13:05:27.089750 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Adding '/usr/lib/tethys/lib/python2.7/site-packages' to path.

[Thu Sep 03 13:05:27.090875 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Adding '/var/www/tethys/apps' to path.

[Thu Sep 03 13:05:27.091210 2015] [wsgi:info] [pid 17585] [remote 10.2.115.202:44] mod_wsgi (pid=17585, process='tethys_default', application$

[Thu Sep 03 13:05:27.278157 2015] [wsgi:error] [pid 17585] /usr/share/httpd/.tethyscluster

[Thu Sep 03 13:05:27.278246 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44] mod_wsgi (pid=17585): Target WSGI script '/usr/lib/tethys$

[Thu Sep 03 13:05:27.278254 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44] mod_wsgi (pid=17585): Exception occurred processing WSGI $

[Thu Sep 03 13:05:27.278267 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44] Traceback (most recent call last):

[Thu Sep 03 13:05:27.278284 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]   File "/usr/lib/tethys/src/tethys_portal/wsgi.py", line $

[Thu Sep 03 13:05:27.278337 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]     application = get_wsgi_application()

[Thu Sep 03 13:05:27.278346 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]   File "/usr/lib/tethys/lib/python2.7/site-packages/djang$

[Thu Sep 03 13:05:27.278381 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]     django.setup()

[Thu Sep 03 13:05:27.278389 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]   File "/usr/lib/tethys/lib/python2.7/site-packages/djang$

[Thu Sep 03 13:05:27.278422 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]     apps.populate(settings.INSTALLED_APPS)

[Thu Sep 03 13:05:27.278430 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]   File "/usr/lib/tethys/lib/python2.7/site-packages/djang$

[Thu Sep 03 13:05:27.278540 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]     app_config = AppConfig.create(entry)

[Thu Sep 03 13:05:27.278551 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]   File "/usr/lib/tethys/lib/python2.7/site-packages/djang$

[Thu Sep 03 13:05:27.278615 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]     module = import_module(entry)

[Thu Sep 03 13:05:27.278624 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]   File "/usr/lib64/python2.7/importlib/__init__.py", line$

[Thu Sep 03 13:05:27.278661 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]     __import__(name)

[Thu Sep 03 13:05:27.278669 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]   File "/usr/lib/tethys/src/tethys_compute/__init__.py", $

[Thu Sep 03 13:05:27.278713 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44]     os.mkdir(TETHYSCLUSTER_CFG_DIR)

[Thu Sep 03 13:05:27.278739 2015] [wsgi:error] [pid 17585] [remote 10.2.115.202:44] OSError: [Errno 13] Permission denied: '/usr/share/httpd/$

error_log.log

[Thu Sep 03 13:00:35.869562 2015] [mpm_prefork:notice] [pid 17530] AH00170: caught SIGWINCH, shutting down gracefully

[Thu Sep 03 13:00:36.911637 2015] [core:notice] [pid 17583] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0

[Thu Sep 03 13:00:36.912285 2015] [suexec:notice] [pid 17583] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

[Thu Sep 03 13:00:36.922783 2015] [auth_digest:notice] [pid 17583] AH01757: generating secret for digest authentication ...

[Thu Sep 03 13:00:36.923330 2015] [lbmethod_heartbeat:notice] [pid 17583] AH02282: No slotmem from mod_heartmonitor

[Thu Sep 03 13:00:36.925479 2015] [mpm_prefork:notice] [pid 17583] AH00163: Apache/2.4.6 (CentOS) mod_wsgi/4.4.13 Python/2.7.5 configured -- $

[Thu Sep 03 13:00:36.925501 2015] [mpm_prefork:info] [pid 17583] AH00164: Server built: Aug 24 2015 18:11:25

[Thu Sep 03 13:00:36.925512 2015] [core:notice] [pid 17583] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

[Thu Sep 03 13:00:36.925627 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Starting process 'tethys_default' with uid=48, gid=48 and thr$

[Thu Sep 03 13:00:36.926007 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Python home /usr/local/pythonenv/BASELINE.

[Thu Sep 03 13:00:36.926025 2015] [wsgi:info] [pid 17585] mod_wsgi (pid=17585): Initializing Python.

[Thu Sep 03 13:00:36.932124 2015] [wsgi:info] [pid 17586] mod_wsgi (pid=17586): Python home /usr/local/pythonenv/BASELINE.

[Thu Sep 03 13:00:36.932124 2015] [wsgi:info] [pid 17586] mod_wsgi (pid=17586): Initializing Python.

[Thu Sep 03 13:00:36.934598 2015] [wsgi:info] [pid 17587] mod_wsgi (pid=17587): Python home /usr/local/pythonenv/BASELINE.

[Thu Sep 03 13:00:36.934626 2015] [wsgi:info] [pid 17587] mod_wsgi (pid=17587): Initializing Python.

[Thu Sep 03 13:00:36.938713 2015] [wsgi:info] [pid 17588] mod_wsgi (pid=17588): Python home /usr/local/pythonenv/BASELINE.

[Thu Sep 03 13:00:36.938744 2015] [wsgi:info] [pid 17588] mod_wsgi (pid=17588): Initializing Python.

[Thu Sep 03 13:00:36.943807 2015] [wsgi:info] [pid 17589] mod_wsgi (pid=17589): Python home /usr/local/pythonenv/BASELINE.

[Thu Sep 03 13:00:36.943837 2015] [wsgi:info] [pid 17589] mod_wsgi (pid=17589): Initializing Python.

[Thu Sep 03 13:00:36.946571 2015] [wsgi:info] [pid 17590] mod_wsgi (pid=17590): Python home /usr/local/pythonenv/BASELINE.

[Thu Sep 03 13:00:36.946598 2015] [wsgi:info] [pid 17590] mod_wsgi (pid=17590): Initializing Python.

[Thu Sep 03 13:00:36.957097 2015] [wsgi:info] [pid 17586] mod_wsgi (pid=17586): Attach interpreter ''.

[Thu Sep 03 13:00:36.957167 2015] [wsgi:info] [pid 17586] mod_wsgi (pid=17586): Adding '/usr/lib/tethys/lib/python2.7' to path.

[Thu Sep 03 13:00:36.957452 2015] [wsgi:info] [pid 17586] mod_wsgi (pid=17586): Adding '/usr/lib/tethys/lib/python2.7/site-packages' to path.

[Thu Sep 03 13:00:36.961212 2015] [wsgi:info] [pid 17589] mod_wsgi (pid=17589): Attach interpreter ''.

[Thu Sep 03 13:00:36.961255 2015] [wsgi:info] [pid 17589] mod_wsgi (pid=17589): Adding '/usr/lib/tethys/lib/python2.7' to path.

[Thu Sep 03 13:00:36.961430 2015] [wsgi:info] [pid 17589] mod_wsgi (pid=17589): Adding '/usr/lib/tethys/lib/python2.7/site-packages' to path.

[Thu Sep 03 13:00:36.969560 2015] [wsgi:info] [pid 17587] mod_wsgi (pid=17587): Attach interpreter ''.

[Thu Sep 03 13:00:36.969603 2015] [wsgi:info] [pid 17587] mod_wsgi (pid=17587): Adding '/usr/lib/tethys/lib/python2.7' to path.

[Thu Sep 03 13:00:36.969796 2015] [wsgi:info] [pid 17587] mod_wsgi (pid=17587): Adding '/usr/lib/tethys/lib/python2.7/site-packages' to path.

Looks like tethys is trying create a folder under  /usr/share/httpd/ but failed. I assume it is creating the ".tethyscluster" folder. It is strange because we already have a folder at /var/www/.tethyscluster. The ownership of /usr/share/httpd/  is root so I changed it to apache, but got the same error. 

[sdc50]

Here is a list of the Python packages installed on the demo server:

azure (0.10.0)

backports.ssl-match-hostname (3.4.0.2)

boto (2.25.0)

cffi (1.1.2)

condorpy (0.1.6)

cryptography (1.0)

decorator (3.4.0)

Django (1.7.10)

django-bootstrap3 (5.1.1)

django-gravatar2 (1.2.1)

django-social-auth (0.7.28)

docker-py (1.2.3)

ecdsa (0.13)

enum34 (1.0.4)

futures (3.0.3)

GeoAlchemy2 (0.2.5)

gisdata (0.5.4)

gsconfig (0.6.13)

gsshapy (2.0.0)

httplib2 (0.9.1)

idna (2.0)

ipaddress (1.0.14)

iptools (0.6.1)

iso8601 (0.1.8)

Jinja2 (2.7.2)

mapkit (1.1.0)

MarkupSafe (0.23)

netCDF4 (1.1.8)

numpy (1.9.2)

oauth2 (1.5.211)

oauthlib (1.0.3)

optcomplete (1.2-devel)

OWSLib (0.8.13)

paramiko (1.15.2)

Paste (1.7.5.1)

PasteDeploy (1.5.2)

PasteScript (1.7.5)

pip (7.0.3)

psycopg2 (2.6.1)

pyasn1 (0.1.8)

pycparser (2.14)

pycrypto (2.6.1)

PyJWT (1.4.0)

pyOpenSSL (0.14)

python-crontab (1.9.3)

python-dateutil (2.4.2)

python-openid (2.2.5)

python-social-auth (0.2.12)

pytz (2015.4)

requests (2.7.0)

requests-oauthlib (0.5.0)

requests-toolbelt (0.4.0)

scp (0.7.1)

setuptools (18.1)

six (1.9.0)

SQLAlchemy (0.9.10)

tethys-dataset-services (1.1.0)

tethys-platform (1.2.0, /usr/lib/tethys/src)

tethysapp-adhydro-streamflow (0.0.1)

tethysapp-canned-gssha (0.1.0, /var/www/tethys/apps/tethysapp-canned_gssha)

tethysapp-dam-break (0.0.1, /var/www/tethys/apps/tethysapp-dam_break)

tethysapp-erfp-tool (0.0.1, /var/www/tethys/apps/tethysapp-erfp_tool)

tethysapp-gsshaindex (0.0.1, /var/www/tethys/apps/tethysapp-gsshaindex)

tethysapp-observed-data (0.0)

tethysapp-parleys-creek-management (0.0.1, /var/www/tethys/apps/tethysapp-parleys_creek_management)

tethysapp-tmaps-app (0.0.1, /var/www/tethys/apps/tethysapp-tmaps_app)

TethysCluster (0.1.6)

websocket-client (0.32.0)

workerpool (0.9.2)

xlrd (0.9.3)

On Tuesday, September 1, 2015 at 9:47:01 AM UTC-6, sdc50 wrote:

[zyli2004]

Scott,

Thanks for sending this. I have another question: what is the exact tethys version (commit code) running on your demo server?

I notice the commit code of 1.2 release is 92f34032470a0278d884230b92418eb5354011d8 while the latest master is 436cce74436dd5faa0c222a5c81840951d93127d.

Thanks

Drew

[zyli2004]

Update: Turning off SELinux on CentOS resolved the problem. A quick test shows that Tethys 1.2.1 works well on the following platforms:

CentOS 6.7/6.7: non-system python 2.7.6, apache 2.2, mod_wsgi 3.5/4.2/4.4 

CentOS 7: system python 2.7.5, apache 2.4, mod_wsgi 4.2/4.4

cat /etc/sysconfig/selinux 

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

#       enforcing - SELinux security policy is enforced.

#       permissive - SELinux prints warnings instead of enforcing.

#       disabled - SELinux is fully disabled.

SELINUX=disabled

# SELINUXTYPE= type of policy in use. Possible values are:

#       targeted - Only targeted network daemons are protected.

#       strict - Full SELinux protection.

[Nathan Swain]

I'm glad it is working. I vaguely remember running into issues with SELinux on our Redhat Server. Looks like we disabled it (though we set it to permissive) as well. Doing so may make the server vulnerable to security threats. Here are a couple of articles:

http://blog.endpoint.com/2010/02/selinux-httpd-modwsgi-26-rhel-centos-5.html

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/SELinux_Guide/selg-preface-0011.html

We should add an issue to investigate deploying Tethys on CentOS and investigate SELinux more.