new setup-web2py-nginx-uwsgi-ubuntu.sh

[Niphlod]

I run uwsgi using the emperor mode for some time now.... based on my config I tried to upgrade the existing one. Result is available at https://www.dropbox.com/s/n7chteos9sh6p2h/setup-web2py-nginx-uwsgi-ubuntu.sh

I'm looking forward to improve things both on uwsgi niceties and general bash customization (the moment I saw fabric I almost stopped writing bash :-P)

Improvements:
- no uwsgi from debian packages (overcomplicated configurations). uwsgi is installed from pip (so you can upgrade with a simple pip install --upgrade uwsgi)
- optional response.static_version friendly static directories configuration (added also cache headers as an option)
- improved syntax for web2py.xml (no app, no plugin)
- example of uwsgi cron facility (clean sessions script included)
- uwsgi emperor mode, managed with Upstart
- optional upstart pre-script command to fix permissions and compress static files

If interested, I can make a fabfile for that.

<offtopic on> I use Redis as cache but for the "I love embedded" guys uwsgi has a cache framework. I'm accepting votes to make a uwsgi's cache compatible module for web2py, but you will have to test it (no time for that, sorry)
<offtopic off>

[Paolo]

Hi and great job!

One thing that I don't like is at line 10

apt-get -y dist-upgrade

this was also in the previous version, actually I see this too restrictive, for example I have ubuntu 12.04 and don't want to move to 12.10

Paolo

[Niphlod]

mee too, didn't notice in the cut-paste madness. Fixed

PS: "revised setup-web2py-nginx-uwsgi-ubuntu.sh, thanks Vinicious": really ? :P

[paolo....@gmail.com]

sorry, one more thing, what about checking if you have the permission to install stuff and so on?

2012/12/12 Niphlod <nip...@gmail.com>

mee too, didn't notice in the cut-paste madness.

--
 
 
 

--
 Paolo

[Niphlod]

meaning checking if the script is run as root ?

[paolo....@gmail.com]

yeap :)

2012/12/12 Niphlod <nip...@gmail.com>

--
 
 
 

--
 Paolo

[Niphlod]

Updated. don't have a shell lying around, could you try it ?

[Paolo]

I've just tested, you should use " with !=

something like that seems to work

if [ "$UID" != "0" ]; then...

Paolo

[paolo....@gmail.com]

Sorry, but this one should be even better:

http://askubuntu.com/questions/15853/how-can-a-script-check-if-its-being-run-as-root/15856#15856

Paolo

2012/12/12 Paolo <paolo....@gmail.com>

--
 
 
 

--
 Paolo

[Richard Vézina]

Didn't test it, but I take the SSL creation commands from setup-web2py-ubuntu.sh that ask for location and owner of certificat that I like better and adapt them for the setup-web2py-nginx-uwsgi-ubuntu.sh script in order to make these script more homogen.

Here the part I change

ln -s /etc/nginx/sites-available/web2py /etc/nginx/sites-enabled/web2py

rm /etc/nginx/sites-enabled/default

mkdir /etc/nginx/ssl

cd /etc/nginx/ssl

openssl genrsa 1024 > self_signed.key

chmod 400 self_signed.key

openssl req -new -x509 -nodes -sha1 -days 365 -key self_signed.key > self_signed.cert

openssl x509 -noout -fingerprint -text < self_signed.cert > self_signed.info

Thanks to include it... 

I will test your script under 12.10 later to day with these mods to see if it works.

Richard

--
 
 
 

[Massimo Di Pierro]

Email me a patch of send me a pull request if this works and you feel should be included

[Niphlod]

@Richard: fixed. Can you test if this is the behaviour that you want ?

@Paolo: thanks for the pointer, fixed.

@Massimo: I "released" the script here to test it before sending the patch to you just because of these small hiccups here and there. This could definitely become the default installation script on debian derivatives of web2py (with one of the speediest stacks out there and quite frankly the easiest to maintain :-P). Next steps are fabric/cuisine scripts for all kinds of Linuxes.

[Richard Vézina]

I am testing it right now, waiting the VM finish to install :)

Richard

--
 
 
 

[Richard Vézina]

502 Bad Gateway

Under 12.10

Richard

[Richard Vézina]

Here error log :

 cat /var/log/nginx/error.log 

2012/12/12 13:12:40 [emerg] 17676#0: SSL_CTX_use_certificate_chain_file("/etc/nginx/ssl/web2py.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory error:20074002:BIO routines:FILE_CTRL:system lib error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib)

2012/12/12 13:12:40 [emerg] 17677#0: SSL_CTX_use_certificate_chain_file("/etc/nginx/ssl/web2py.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory error:20074002:BIO routines:FILE_CTRL:system lib error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib)

2012/12/12 13:15:34 [emerg] 17721#0: SSL_CTX_use_certificate_chain_file("/etc/nginx/ssl/self_signed.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory error:20074002:BIO routines:FILE_CTRL:system lib error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib)

2012/12/12 13:15:34 [emerg] 17722#0: SSL_CTX_use_certificate_chain_file("/etc/nginx/ssl/self_signed.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory error:20074002:BIO routines:FILE_CTRL:system lib error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib)

2012/12/12 13:16:45 [emerg] 17756#0: SSL_CTX_use_certificate_chain_file("/etc/nginx/ssl/self_signed.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory error:20074002:BIO routines:FILE_CTRL:system lib error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib)

2012/12/12 13:16:45 [emerg] 17757#0: SSL_CTX_use_certificate_chain_file("/etc/nginx/ssl/self_signed.crt") failed (SSL: error:02001002:system library:fopen:No such file or directory error:20074002:BIO routines:FILE_CTRL:system lib error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib)

2012/12/12 13:17:29 [crit] 17770#0: *5 connect() to unix:///tmp/web2py.socket failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.102, server: sgddms1, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/web2py.socket:", host: "192.168.1.190"

2012/12/12 13:17:29 [crit] 17770#0: *5 connect() to unix:///tmp/web2py.socket failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.102, server: sgddms1, request: "GET /favicon.ico HTTP/1.1", upstream: "uwsgi://unix:///tmp/web2py.socket:", host: "192.168.1.190"

2012/12/12 13:18:34 [crit] 17770#0: *9 connect() to unix:///tmp/web2py.socket failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.102, server: sgddms1, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/web2py.socket:", host: "192.168.1.190"

2012/12/12 13:18:34 [crit] 17770#0: *9 connect() to unix:///tmp/web2py.socket failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.102, server: sgddms1, request: "GET /favicon.ico HTTP/1.1", upstream: "uwsgi://unix:///tmp/web2py.socket:", host: "192.168.1.190"

2012/12/12 13:18:38 [crit] 17770#0: *9 connect() to unix:///tmp/web2py.socket failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.102, server: sgddms1, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///tmp/web2py.socket:", host: "192.168.1.190"

2012/12/12 13:18:38 [crit] 17770#0: *9 connect() to unix:///tmp/web2py.socket failed (2: No such file or directory) while connecting to upstream, client: 192.168.1.102, server: sgddms1, request: "GET /favicon.ico HTTP/1.1", upstream: "uwsgi://unix:///tmp/web2py.socket:", host: "192.168.1.190"

Please ignore the lines about certificat, I didn't have the last version of the script. So I change the wrong name of the certificat in order that nginx config use the proper files.

Richard

[Richard Vézina]

uwsgi doesn't seem to start, I have not log for it in /var/log/uwsgi/

Try to start it like that :

/etc/init$ sudo start uwsgi-emperor

start: Job failed to start

Richard

[Niphlod]

ok, wait a second, I'll try with my VM. the previous traceback is due to the changes done to the certificate generation: I took them for granted but there must be an error somewhere.

edit: seen the "forget about the certificates" message right now. Investigating in your "job failed to start".
PS: can you post the output of /var/log/upstart/uwsgi-emperor.log ?

[Niphlod]

fixed in PM. uwsgi wasn't installed on Richard's VM because pip path was reported incorrectly at the first installation (meaning that pip install --upgrade pip worked ok but the next pip install --upgrade pip was not finding pip binary).

Fixed that, script is now running ok even in 12.10.

@Paolo: added the shebang to execute it in bash instead of sh. now check for root permission works.

In the next few days I'll write down a fabfile, it's usually cleaner and safer (e.g. if a network hiccup happens and a package doesn't get correctly installed, this script goes on like nothing happened)

On Wednesday, December 12, 2012 9:09:51 PM UTC+1, Niphlod wrote:

ok, wait a second, I'll try with my VM. the previous traceback is due to the changes done to the certificate generation: I took them for granted but there must be an error somewhere.

edit: seen the "forget about the certificates" message right now. Investigating in your "job failed to start".
PS: can you post the output of /var/log/upstart/uwsgi-emperor.log ?

On Wednesday, December 12, 2012 7:48:54 PM UTC+1, Richard wrote:

[Richard Vézina]

Hello Simone,

As I told you just try the script with a fresh VM. I notice that, the server were not accessible after the script executed. I had to stop both uwsgi and nginx and restart them, then the welcome show up.

I think that the last line is erroneous...

I replace :

/etc/init.d/nginx restart

By :

service nginx start

Could also have work with (NOT TESTED) :

/etc/init.d/nginx start

And retest with a snapshot of the fresh VM and it works then.

Cheer!

Richard

--
 
 
 

[Niphlod]

uhm, in my tests worked (and that line comes from the old script). After you install nginx it is started automatically, so "restart" is a good choice.

However, let's not take it for granted and investigate.....

doing cat /etc/init.d/nginx in 12.10 we can find

restart|force-reload)
        echo -n "Restarting $DESC: "
        start-stop-daemon --stop --quiet --pidfile \
            /var/run/$NAME.pid --exec $DAEMON || true
        sleep 1
        test_nginx_config
        # Check if the ULIMIT is set in /etc/default/nginx
        if [ -n "$ULIMIT" ]; then
            # Set the ulimits
            ulimit $ULIMIT
        fi
        start-stop-daemon --start --quiet --pidfile \
            /var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true
        echo "$NAME."
        ;;

So, it works both if nginx is yet started or not. In fact, it's simply doing a stop followed by a start.

service wraps up the same "command interface" for both /etc/init.d/ scripts (System V) and /etc/init (upstart).

Doing service nameofsomething start (or restart) does the same thing as /etc/init.d/nameofsomething start  if /etc/init.d/nameofsomething is there

Changing that line from "start" to "restart" have a side effect: if nginx is already up (as in normal install just after apt-get install nginx-full), the config will not be reloaded, so there will be no configuration to let nginx talk with uwsgi.

So, are you totally sure that everything is going on correctly with the install and this line is the culprit ?

[Richard Vézina]

I am not sure, the only thing that I know is that Nginx is not showing up until I stop and restart it. Maybe it was needing a reload as you said at last...

Richard

[Richard Vézina]

Hello Simone

Here some improvement :

The following is an example configuration, to reduce the CPU load it is recommended to run one worker process only and to enable keep-alive connections:

worker_processes 1;
http {
  server {
    listen               443;
    ssl                  on;
    ssl_certificate      /usr/local/nginx/conf/cert.pem;
    ssl_certificate_key  /usr/local/nginx/conf/cert.key;
    keepalive_timeout    70;
  }
}

When using a chain of certificates, just append the extra certificates to your .crt file (cert.pem in the example). The server certificate needs to be the first on the file, otherwise you'll get a mismatch between private and public keys.

Since Nginx version 0.7.14 the preferred way of enabling SSL is by using the `ssl` parameter of the `listen` directive:

server {
  listen 443 default_server ssl;
  ssl_certificate      /usr/local/nginx/conf/cert.pem;
  ssl_certificate_key  /usr/local/nginx/conf/cert.key;  
  ...
}

Ref : http://wiki.nginx.org/HttpSslModule

I put the parameter and the reason in bold.

You do what you want, I report because they are suggested best practices that I found reading a bit about ssl and Nginx.

Richard

[Niphlod]

At this point I did a bit of research.....

https://github.com/ioerror/duraconf/blob/master/configs/nginx/nginx.conf

updated the script to reflect those, but I can't test it. Can you please ?

[Richard Vézina]

I will, when I finish my research.

I am planning to write a slice about deployment of Redmine beside Web2py with Nginx. So until I finish my Nginx config in case I found other tweaks, I wait for testing the script again. But I have a fresh VM in place that I can clone rapidly to for testing purpose, so when I am ready, it will not be a big task.

:)

Richard

--
 
 
 

[Arnon Marcus]

Hey guys, I'm also using web2py and redmine, and was concidering moving it into a linux-based VM and running it with nginx. Good to know your on it. Keep up the good work - I'm a neub in linux-land...

Couple of questions:

1. Are you using postgreSQL in this thing?

2. How can one replicate your achievement ones it works?

3. I saw that BitNami has an ubuntu image of RedMine with MySQL with optionally choosing nginex as the web-server, and also a PostgreSQL plugin for these stacks (they also have a "Redmine on Postgres" stack, if I remember, but I think that combination doesn't exist for the ubuntu-image flavor, oddly enough...). Can that be used as a base for your configuration for integrating web2py support on-top of it?

[Paolo]

Hi all, I've just tried to run the script on a VM (ubuntu 12.04.1). The compilation of uwsgi failed:

*** uWSGI linking ***

gcc -pthread -o /usr/local/bin/uwsgi  core/utils.o core/protocol.o core/socket.o core/logging.o core/master.o core/master_utils.o core/emperor.o core/notify.o core/mule.o core/subscription.o core/stats.o core/sendfile.o core/offload.o core/setup_utils.o core/clock.o core/init.o core/buffer.o core/plugins.o core/lock.o core/cache.o core/daemons.o core/queue.o core/event.o core/signal.o core/cluster.o core/rpc.o core/gateway.o core/loop.o lib/rbtree.o core/rb_timers.o core/uwsgi.o proto/base.o proto/uwsgi.o proto/http.o proto/fastcgi.o lib/linux_ns.o core/async.o core/ini.o core/yaml.o core/snmp.o core/xmlconf.o core/spooler.o plugins/python/python_plugin.o plugins/python/pyutils.o plugins/python/pyloader.o plugins/python/wsgi_handlers.o plugins/python/wsgi_headers.o plugins/python/wsgi_subhandler.o plugins/python/web3_subhandler.o plugins/python/pump_subhandler.o plugins/python/gil.o plugins/python/uwsgi_pymodule.o plugins/python/profiler.o plugins/python/symimporter.o plugins/python/tracebacker.o plugins/gevent/gevent.o plugins/ping/ping_plugin.o plugins/cache/cache.o plugins/nagios/nagios.o plugins/rrdtool/rrdtool.o plugins/carbon/carbon.o plugins/rpc/rpc_plugin.o plugins/corerouter/cr_common.o plugins/corerouter/cr_map.o plugins/corerouter/corerouter.o plugins/fastrouter/fastrouter.o plugins/http/http.o plugins/ugreen/ugreen.o plugins/signal/signal_plugin.o plugins/syslog/syslog_plugin.o plugins/rsyslog/rsyslog_plugin.o plugins/logsocket/logsocket_plugin.o plugins/router_uwsgi/router_uwsgi.o plugins/router_redirect/router_redirect.o plugins/router_basicauth/router_basicauth.o plugins/zergpool/zergpool.o plugins/redislog/redislog_plugin.o plugins/mongodblog/mongodblog_plugin.o plugins/router_rewrite/router_rewrite.o plugins/router_http/router_http.o plugins/logfile/logfile.o plugins/router_cache/router_cache.o plugins/rawrouter/rawrouter.o -lpthread -lm -rdynamic -ldl -lssl -lcrypto -L/usr/lib/x86_64-linux-gnu -lxml2 -lpthread -ldl -lutil -lm -L/usr/lib/python2.7/config -lpython2.7 -lcrypt

*** error linking uWSGI ***

----------------------------------------

Command /usr/bin/python -c "import setuptools;__file__='/tmp/pip-build/uwsgi/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-lRnXgl-record/install-record.txt --single-version-externally-managed failed with error code 1 in /tmp/pip-build/uwsgi

Exception information:

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 107, in main

    status = self.run(options, args)

  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 261, in run

    requirement_set.install(install_options, global_options)

  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1166, in install

    requirement.install(install_options, global_options)

  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 589, in install

    cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)

  File "/usr/local/lib/python2.7/dist-packages/pip/util.py", line 612, in call_subprocess

    % (command_desc, proc.returncode, cwd))

InstallationError: Command /usr/bin/python -c "import setuptools;__file__='/tmp/pip-build/uwsgi/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-lRnXgl-record/install-record.txt --single-version-externally-managed failed with error code 1 in /tmp/pip-build/uwsgi

Maybe something is missing?

Moreover, I don't like the fact that it tries to make a dist-upgrade

Regards, 

Paolo

[Richard Vézina]

@Paolo, I don't see dist-upgrade anymore you should have a wrong version look at the drop box of Simone Niphold : https://www.dropbox.com/s/n7chteos9sh6p2h/setup-web2py-nginx-uwsgi-ubuntu.sh

@Arnon, I don't know bitnami I heard about, but didn't have a look, but I guest it will not works because ubuntu nginx package doesn'st include passenger module support, so we have to compile nginx because it is not modular as apache. Actually I found a PPA, but I am not sure if it will be maintained for a wild, so I use a script include in ruby to automated the configuration, compilation and installation of an appropriate version of nginx... I am in the process of writing a tutorial about that. I will inform the list here when it done and publish. Notice, someone could just use FastCGI instead of uWSGI and Phussion Passenger in tamdem to serve the web2py apps and Redmine together with Nginx, but I prefer using uWSGI. Also, recently Phussion project report to be polyglot and say that Passenger can handle both WSGI (Python) and Ruby at the same time, but I don't know if it could work with uWSGI... Anyway. As soon as I get my tutorial ready, I get back here.

@Simone, an other improvement to the script could be to combine into a single server the 80, and 443 to avoid duplicating configuration, as stated here :

http://nginx.org/en/docs/http/configuring_https_servers.html#single_http_https_server

I don't had test this.

Also, here the code for permanent redirection 

server {

  server_name $hostname;

  listen              80;

  return 301 https://$hostname$request_uri; # NOTE: I am not sure for $hostname here, because I didn't set hostname for my VM until now, as in the example (URL below) we can use domainName.com instead if properly configure in nginx

...

}

In accordance with the pitfalls page suggested best practice : http://wiki.nginx.org/Pitfalls#Taxing_Rewrites

Richard

--
 
 
 

[Niphlod]

@Simone, an other improvement to the script could be to combine into a single server the 80, and 443 to avoid duplicating configuration, as stated here :

http://nginx.org/en/docs/http/configuring_https_servers.html#single_http_https_server

I don't had test this.

Also, here the code for permanent redirection 

server {

  server_name $hostname;

  listen              80;

  return 301 https://$hostname$request_uri; # NOTE: I am not sure for $hostname here, because I didn't set hostname for my VM until now, as in the example (URL below) we can use domainName.com instead if properly configure in nginx

...

}

People may want separate configs for http and https. The script objective is to have a working copy of web2py. If we start to follow such requests, we'd end up installing postgresql and redis too :P

@Paolo: try the script found at the dropbox link. If the same error happens I think we need the log of pip and a hand by Roberto on the specific error...

[Richard Vézina]

Agree...

Richard

[paolo....@gmail.com]

Hi

I was trying with the script that comes with the stable web2py. with the one on dropbox I problem has gone.

Paolo

2012/12/19 Niphlod <nip...@gmail.com>

--
 
 
 

--
 Paolo

[Niphlod]

perfect, Ccing Massimo on this final one.

[Massimo Di Pierro]

please email me the patch of latest file when ready for inclusion.

[Richard Vézina]

Hello,

I publish a new script that allow deployment of Redmine beside web2py.

Here : https://groups.google.com/forum/?fromgroups=#!searchin/web2py/redmine/web2py/ZqL7Si8Khbo/Es-wK1yXdgQJ

Notice : After some read, I choose Unicorn over Phussion Passenger.

Richard

--
 
 
 

[paolo....@gmail.com]

Hi all, I've just discovered that the regex used to retrieve the static files doesn't work if I use languages abbreviation in urls.

How can we fix this?

2012/12/28 Richard Vézina <ml.richa...@gmail.com>

--
 
 
 

--
 Paolo

[Massimo Di Pierro]

can you show an example?

[paolo....@gmail.com]

Hi Massimo, for example, if I visit this traffic.integreen-life.bz.it/default/wiki/about  (default is the controller and wiki is the function) it works while traffic.integreen-life.bz.it/it/default/wiki/about the application works very well but the static links do not.

Paolo

2013/1/22 Massimo Di Pierro <massimo....@gmail.com>

--
 
 
 

--
 Paolo

[Niphlod]

got it. can you post the routes.py you're using ?

The problem lies in the fact that routes.py is really flexible and adapting that logic using only rewrite or alias statements in nginx is cumbersome.
Standing on one feet (i.e. without tests) I'd say that the regex checking static files

        location ~* /(\w+)/static/ {
            root /home/www-data/web2py/applications/;
            #remove next comment on production
            #expires max;
        }

is catching it as the app name and not the language "trick" . so, for a request going to /it/app/static/whatever.css is looking into 
/home/www-data/web2py/applications/it/appname/static/whatever.css instead of  ..... (remove the bold part, "it/").

If you remove those lines web2py will take charge for static files serving, so the issue will be temporarily fixed.

I'll try to set up some additional rules to make nginx behave like routes.py, but it will take some time.

PS: please mind that the script it's a template.... it's not meant to fullfill every custom installation patterns out there without further tuning.
language routing is probably the most difficult out there because for a request
/it/appname/static/whateverfile.css
web2py looks into /web2py/appname/static/it/whateverfile.css and then if not found into /web2py/appname/static/whateverfile.css (a nice fallback, but hard to map with a simple nginx statement)

[paolo....@gmail.com]

Hi Niphlod, once I discovered that making a symbolic link for each language in the app directory pointing to the app directory worked well.

Namely something like that app_dir/it ---> app_dir . However  it would be better to find something cleaner but for that I have to look at nginx but right now I don't access to the virtual machine. 

I guess we can add a new directive for serving static files with language in url. will it works? 

Paolo

2013/1/22 Niphlod <nip...@gmail.com>

--
 
 
 

--
 Paolo

[Niphlod]

maybe I have a fix .
Can you please test it ?
assuming an example of routes.py like

myapp = dict(languages=['en', 'it', 'jp'], default_language='en')

this means that navigating to:
 /myapp/ --> the static files will be referenced as web2py_home/applications/myapp/en/static/whatever.css
 /myapp/it/ --> the static files will be referenced as web2py_home/applications/myapp/it/static/whatever.css

however, direct requests to /myapp/static/whatever.css should map to web2py_home/applications/myapp/en/static/whatever.css

additionally, if /myapp/it/static/whatever.css is requested, if not found web2py_home/applications/myapp/static/whatever.css should be served.

Now, I think that adding this section before the standard one will fix the issue

location ~* /(\w+)/(en|it|jp)/static/(.*)$ {
            alias /home/www-data/web2py/applications/$1/;
            try_files static/$2/$3 static/$3 =404;
        }

(en|it|jp) here is the regex matching the "languages" list of the routes app's dict .... This will set for the request /myapp/it/static/whatever.css   the alias to the web2py_home/applications/myapp/ and then try to serve static/it/whatever.css, if not found, it will fall back to a "normal" static/whatever.css, and if that one is not found will return a 404.

@Jonathan: please stop me if I did miss something in the routes logic
@all : seems a simple fix - maybe too simple... please help test it

[paolo....@gmail.com]

Just tested, it works, I've already upgraded the production server too :-P

I would suggest to add it (maybe as a comment) to the script.

Thanks, Paolo

2013/1/22 Niphlod <nip...@gmail.com>

--
 
 
 

--
 Paolo

[Niphlod]

ok, I'll test it more @home and then add a commented section to the script

PS: that site you posted ..... turn off response.optimize_js = 'inline' pleeease :P

[paolo....@gmail.com]

In the meanwhile I discovered that if you upload a file greater than 1M, I tried with a 1.5M, you will get a 413 error. In order to avoid this error for file not so big, I suggest you to add the directive client_max_body_size and to set it at something more 'common' for me few MB is already ok. see here: http://wiki.nginx.org/HttpCoreModule#client_max_body_size

paolo

2013/1/23 Niphlod <nip...@gmail.com>

--
 
 
 

[niphlod]

uhm. this is outside the scope of the location .... I mean... probably you'd like to turn on gzip too, but that is a feature to enable on nginx "main" conf, not in the "web2py" one ....

Anyway, did you put this setting inside the file /etc/nginx/sites-available/web2py and it just works ok ?

2013/1/23 paolo....@gmail.com <paolo....@gmail.com>

--
 
 
 

[paolo....@gmail.com]

yep, I've added it under the location / and it worked. You are right it might be placed in the main configuration, on the other hand today at work I was blocked because unable to upload files greaten than 1M. From my point of view we should at least mention it. For what concerns gzip is that on ubuntu server I can see that is already enabled by default.

paolo

2013/1/23 niphlod <nip...@gmail.com>

--
 
 
 

--
 Paolo

[Niphlod]

ok, made it more modular, and now web2py config for uwsgi is in ini-style (far more readable).
If you come from previous script, before executing this please delete /etc/uwsgi/web2py.xml . the new /etc/uwsgi/web2py.ini will be created with the exact same options.

thanks Paolo, patch attached.

[Arnon Marcus]

Great to see this coming along !

Few questions:

1. Would it be moduralized enough to separate the redmine and web2py parts?

2. if not, can I use it on 2 servers, and then remove redmine in one and web2py on the other?

3. Does this supports older web2py versions (say, 1.8) ?

4. Can somebody port this to CentOS ? Is there a CentOS expert in this group?

[Richard Vézina]

I will look at Niphold changes and update my script when I get time. I have no intention to port the w2p-redmine script to CentOS. I would not put redmine and web2py on 2 differents servers, I would put all the web app on one server (one html server) and the database on an other server. But that is fancy stuff and the script will not integrate that since it only help deploy faster with a basic config that you should adapt to your particular needs.

Old version of web2py are you kinding me or what?!

Richard

--
 
 
 

[Arnon Marcus]

I am deploying on VMs that are relatively easy to clone, so I opted for separate servers for redmine and web2py, for performance and security. I obviously also separated the database to be on its own server. Since it's all VMs on the same physical machine, the virtual-network is really fast, and since every VM uses its own set of hard-drive space, its own memory space, and its own CPU core, they can all run concurrently, even in single threaded cases. Security comes from each only having the services they each require and nothing more (so no python in the redmine or postgres servers, no ruby on the web2py or postgres servers, etc.)

As for old-version of web2py, I am actually not kidding.

We have a web2py application that we've been working on for 3 years now, and moving that to the new web2py version revealed itself to be non-trivial.

Backward compatibility is apparently not as well adhered to as we would expect. I am all for going for the latest and greatest, but we have a schedule to keep in development, and upgrading web2py is not too much a priority now... Unless it provides measurable performance benefits, which it doesn't in our case, from what I briefly tested.

--
 
 
 

[Arnon Marcus]

Is there a benefit for using nginx as rps for apache?
What are the pros/cons for such an arrangement over uwsgi ? 

[Richard Vézina]

The only benefit I see nginx over apache from my reading, is that nginx memory usage and cpu usage will not grow that much even if you add massive number of users since "Nginx is event-based it doesn't need to spawn new processes or threads for each request, so its memory usage is very low"

http://www.wikivs.com/wiki/Apache_vs_nginx

Richard

--
 
 
 

[Arnon Marcus]

Yeah, I already know that, I was asking something else:
You can have a hybrid setup of Apache +nginx, where nginx is hosting apache as a reverse-proxy. According to what I read, this kind of setup is very common, especially for php servers, as the php processing that apache   operates is single-threaded, and so apache becomes a bottleneck without a reverse-proxy hosting it. Once you add-in something like nginx in front of it, then now the php processing will not be blocking the process of the hosted session, and will run in parallel to it.
Now, I'm by far not a technical expert on these issues, but I am very good at conceptualized understanding of performance.
My question was referring to the way nginx would handle the web2py processes via uwsgi, and how it would compare to the php case, if/when an apache server is plugged-in between nginx and web2py.
I hope my question is clearer now.

[Arnon Marcus]

My second question, was basically to get a vibe for the pros/cons of this hybrid setup vs. the plain and pure vanilla setup that this script probably generates.

[Niphlod]

seems you missed a point.... uwsgi here is not a module, is an executable that does one job and it does it well (actually, very well, and there's a lot of it that can be used that is outside the scope of this script).
It could be used as a standalone highperformance webserver, but nginx is placed in front of it to serve static files and to take care of Ddos attacks.

If you want to use apache behind nginx instead of uwsgi behind nginx you're going basically to suffer wasted cpu, ram, a much harder to maintain config.
If you want to run python on apache because it's your default webserver, than mod_wsgi is the way to go. Have to install apache just to run python, it's only a waste of resources.

[Arnon Marcus]

10x for clearing things out - you're right, I didn't do too much resource on uwsgi, and just assumed that it is, for nginx. what mod_wsgi is for apache.

So I guess I had it wrong.

My current (soon to be "old") setup is running apache + mod_wsgi on windows 7, so I know all about the headaches that comes from setting this up...

I would be more than glad to put apache behind me for good, if it would offer not performance improvements to this script's setup the way it does for php...

On that note, how exactly is uwsgi handling web2py processes, as would be configured in this script? Is it easily customizable after the fact?

Are there any any pros/cons for different scenarios that one should be aware of?

--
 
 
 

[Arnon Marcus]

Oh, and what about memcache?

Can web2py benefit from it? Is there somewhere an explanation about this?

[Paolo valleri]

I am using memcached for caching cache.ram and cache.disk, the configuration is really easy, this is mine:

if not request.is_local:
    from gluon.contrib.memcache import MemcacheClient
    memcache_servers = ['127.0.0.1:11211']
    cache.memcache = MemcacheClient(request, memcache_servers)
    cache.ram = cache.disk = cache.memcache

but I don't know how to understand the gain of using it. Any idea?

paolo

[Arnon Marcus]

Well, the way I currently understand this, is as follows:

Web2py uses execfile for most of it's work (models, controllers and views), so no reload() is needed in production whenever a file get's modified.

The flip-side of this, is that there can be no cache in these module-files.

Then the ram/disk cache is for getting from these executed-modules to the python process running web2py. But this is still assuming that there is a single process. If nginx/uwsgi launches multiple processes of web2py, than this cache will no longer be helpful - each time a process is launched, the cache would have to be re-populated - this could mostly mean doing database queries. That's a very bad thing for performance.

I don't know about web2py's implementation of memcached or about memcache at all, but I guess its meant for solving that issue, right?

--
 
 
 

[Arnon Marcus]

Here is a cool episode about virtualisation architecture design:

https://www.youtube.com/watch?v=Kiftbm1L_eQ&list=PL3EFBFBCE1249ABC0

[Niphlod]

disk cache is shared among processes, so no problem there.

uwsgi should use os.fork() to spawn a new process, so modules are definitively cached.

Didn't test with cache.ram (but if you want I may test it) because I use redis as cache in production.

cache.ram will be "erased" when a "forced" shutdown of the process is required (this conf by default asks uwsgi to respawn a child process that fullfilled 2000 requests (the max-requests parameter)).

redis and memcache obviously "solve the theoretic issue" of course (they are totally independant process from web2py).

[dederocks]

Hello, 

I have setup nginx & uwsgi on lubuntu with the script and it works great (best server ever!), except for a part of my application that requires multi threads. I have changed the /etc/uwsgi/web2py.xml to enable threads, tried to set each worker with 2 threads, set the thread memory stack to 512, it doesn't work: I quickly get a 'resource not available (error 11)' message.

Anyone has a tip on what I could work on to make this work? Because otherwise the application works fine with rocket / gevent for example.

Thanks, 

Andre

Le mercredi 12 décembre 2012 00:06:16 UTC+1, Niphlod a écrit :

I run uwsgi using the emperor mode for some time now.... based on my config I tried to upgrade the existing one. Result is available at https://www.dropbox.com/s/n7chteos9sh6p2h/setup-web2py-nginx-uwsgi-ubuntu.sh

I'm looking forward to improve things both on uwsgi niceties and general bash customization (the moment I saw fabric I almost stopped writing bash :-P)

Improvements:
- no uwsgi from debian packages (overcomplicated configurations). uwsgi is installed from pip (so you can upgrade with a simple pip install --upgrade uwsgi)
- optional response.static_version friendly static directories configuration (added also cache headers as an option)
- improved syntax for web2py.xml (no app, no plugin)
- example of uwsgi cron facility (clean sessions script included)
- uwsgi emperor mode, managed with Upstart
- optional upstart pre-script command to fix permissions and compress static files

If interested, I can make a fabfile for that.

<offtopic on> I use Redis as cache but for the "I love embedded" guys uwsgi has a cache framework. I'm accepting votes to make a uwsgi's cache compatible module for web2py, but you will have to test it (no time for that, sorry)
<offtopic off>

[Roberto De Ioris]

> Hello,
>
> I have setup nginx & uwsgi on lubuntu with the script and it works great
> (best server ever!), except for a part of my application that requires
> multi threads. I have changed the /etc/uwsgi/web2py.xml to enable threads,
> tried to set each worker with 2 threads, set the thread memory stack to
> 512, it doesn't work: I quickly get a 'resource not available (error 11)'
> message.
>
> Anyone has a tip on what I could work on to make this work? Because
> otherwise the application works fine with rocket / gevent for example.
>
> Thanks,
>
> Andre

If your app is multithreads based you cannot spawn additional processes
(they do not share memory).

Just set processes to 1 and threads to a decent value


--
Roberto De Ioris
http://unbit.it

[dederocks]

AWSOME! Thank you so much for the fix and explanation!

Best regards, Andre