Setting up Tornado with Nginx on Ubuntu 10.04 for production use

[yupyup1234]

Hi all, I asked this question at serverfault.com and at
stackoverflow.com but arent getting any response.

So i guess I'll ask here:

I understand that there's an nginx configuration file at http://www.friendfeed.com
But i don't really know how to set up Tornada for production use on
Ubuntu 10.04 with Nginx.

Here's my situation and assumptions: 1) Assuming my Tornado project is
set up as such:

project/
src/
static/
templates/
project.py
And I have installed Tornado by downloading the repositary from Github
and than sudo python setup.py install

2) I've installed Nginx and started it based on the instructions
here : http://library.linode.com/web-servers/nginx/installation/ubuntu-10.04-lucid

My questions are: Where does my nginx configuration file go ? Within
the src/ folder? After configuring Nginx, how do I start my Tornado
project?

[Eugene Y.E]

Hi, 

Yes sorry for the confusion, project.py is originally in src/ folder, just a typo earlier on.

The project structure should be:

project/
    src/

    project.py
       static/
       templates/

So what you mean is that there's no need for me to create a seperate nginx.conf file within my project's folder?

What if I want to run multiple Tornado projects in a single server?
       

On Tue, Feb 8, 2011 at 12:34 AM, Andrew Zeneski <and...@andrewzeneski.com> wrote:

Your nginx.conf file either needs to be in the directory that ngnix is configured to look in (depending on how it is configured (default if compiled from source and you did not change any options /usr/local/nginx/conf/nginx.conf) or you can specify the config file when you start nginx (i.e. nginx -c /path/to/nginx.conf)

There are a number of sample init scripts for nginx look here:
http://wiki.nginx.org/Configuration#Init_Scripts

As for your tornado app, project.py probably should not be inside the templates directory; you should look at the demos that come with tornado. To answer your question, you would start your app the same way you would before you added nginx to the mix. (i.e. python myapp.py)

If you search the archives of this group you will find both example nginx configurations as well as configurations for supervisord (one of the recommended ways of starting your tornado app processes).

Andrew

--
Best Regards,

Liang Yuxian Eugene

http://www.liangeugene.com/about/

[Cliff Wells]

On Mon, 2011-02-07 at 07:33 -0800, yupyup1234 wrote:

> 2) I've installed Nginx and started it based on the instructions
> here : http://library.linode.com/web-servers/nginx/installation/ubuntu-10.04-lucid

I'd not follow those instructions. Instead you should use the Nginx
PPA documented on the Nginx wiki:

http://wiki.nginx.org/Install

basically this:

sudo aptitude purge nginx
sudo add-apt-repository ppa:nginx/stable
sudo aptitude update
sudo aptitude install nginx

This will give you the most recent, stable version of Nginx. The
official Ubuntu version is ancient and there's little reason to build
from source unless you need a customized build.

> My questions are: Where does my nginx configuration file go ?

It goes in /etc/nginx. The main file is nginx.conf, but it can include
other files. The key part you'll need is something like:

server {
root /path/to/static/files;

location / {
try_files $uri @tornado;
}

location @tornado {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:5000;
}
}

> Within
> the src/ folder? After configuring Nginx, how do I start my Tornado
> project?

Start Nginx, start your Tornado application

python --port=5000 myapp.py

and test. Next you'll want to look into using supervisord or similar
to manage your Tornado process, but you need to get Nginx and Tornado
tested first before you worry about that.

Regards,
Cliff

[Eugene Y.E]

Hello,

thank you for your reply.

Just a few more questions with regards to the configuration file:

server {
   root /path/to/static/files;

   location / {
       try_files $uri @tornado; 
   }

   location @tornado {
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_pass       http://127.0.0.1:5000;

   }
}

For the highlighted lines, does " location / " refer to the directory location of my Tornado project? As in the location of

project.py ?

What does try_files $uri @tornado do ?

What does  location @tornado do? 

Thank you once again!

[Benoit Hirbec]

Hi,

>>> Where does my nginx configuration file go ?

Your Nginx main configuration file should be in /etc/ngnix/ngnix.conf.
Within this file you can add the `include` directive that allows you
to "include any configuration files for what ever purpose you
want" (http://wiki.nginx.org/CoreModule#include). Some store site-
specific configuration files in /etc/nginx/conf.d/<mysite>.conf and
then include them in the nginx.conf file. I'd rather store site-
specific configuration files close to the source so I can commit both
at the same time. To be honest, I don't know which practice is the
best.

>>> how do I start my Tornado ?
I use supervisor to start, restart or stop the tornado instance. See
more here http://supervisord.org/index.html

Hope this will be helpfull,
Benoit

On Feb 7, 11:33 pm, yupyup1234 <neto...@gmail.com> wrote:
> Hi all, I asked this question at serverfault.com and at
> stackoverflow.com but arent getting any response.
>
> So i guess I'll ask here:
>

> I understand that there's an nginx configuration file athttp://www.friendfeed.com

> But i don't really know how to set up Tornada for production use on
> Ubuntu 10.04 with Nginx.
>
> Here's my situation and assumptions: 1) Assuming my Tornado project is
> set up as such:
>
> project/
>     src/
>        static/
>        templates/
>            project.py
> And I have installed Tornado by downloading the repositary from Github
> and than sudo python setup.py install
>
> 2) I've installed Nginx and started it based on the instructions

> here :http://library.linode.com/web-servers/nginx/installation/ubuntu-10.04...

[Cliff Wells]

On Tue, 2011-02-08 at 01:00 +0800, Eugene Y.E wrote:

>
> server {
> root /path/to/static/files;
>
> location / {
> try_files $uri @tornado;
> }
>
> location @tornado {
> proxy_set_header Host $host;
> proxy_set_header X-Real-IP $remote_addr;
> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
> proxy_pass http://127.0.0.1:5000;
> }
> }
>
>

> does " location / " refer to the directory location of my Tornado

> project? As in the location of
> project.py ?

No, it's a URI. http://mysite.com/ <-- that trailing slash is the "/"
in the location directive.

I should point out that Nginx does not care about the location of your
Tornado application. It will not be referencing any Tornado files
directly. Nginx and Tornado will be communicating via HTTP over your
loopback interface.

In fact, I recommend putting your static resources (images, css, etc) in
a directory completely separate from your Tornado application. This way
a misconfiguration of Nginx will be less likely to result in someone
gaining access to sensitive information.

> What does try_files $uri @tornado do ?

It tries first to locate a static file (such as css, js, jpg, etc),
using the $uri relative to the $document_root, and if that fails, passes
the request to the @tornado location, which in turn hands it over to
Tornado to handle.

> What does location @tornado do?

The @ symbol means a "named" location, that is, one that cannot be
referenced directly from the client's browser. This location is simply
to encapsulate the proxying directives into a tidy location.

Regards,
Cliff

[Andrew Zeneski]

Your nginx.conf file either needs to be in the directory that ngnix is configured to look in (depending on how it is configured (default if compiled from source and you did not change any options /usr/local/nginx/conf/nginx.conf) or you can specify the config file when you start nginx (i.e. nginx -c /path/to/nginx.conf)

There are a number of sample init scripts for nginx look here:
http://wiki.nginx.org/Configuration#Init_Scripts

As for your tornado app, project.py probably should not be inside the templates directory; you should look at the demos that come with tornado. To answer your question, you would start your app the same way you would before you added nginx to the mix. (i.e. python myapp.py)

If you search the archives of this group you will find both example nginx configurations as well as configurations for supervisord (one of the recommended ways of starting your tornado app processes).

Andrew

On Feb 7, 2011, at 10:33 AM, yupyup1234 wrote:

[Andrew Zeneski]

You would account for this in your nginx.conf file. You could bind nginx to different IPs or use virtual hosts, then configure each 'server' to proxy to a different set of tornado processes. 

Nginx has really good documentation: 

http://wiki.nginx.org/Configuration

Andrew

[TornadoRocks]

Hi all, thank you for your help.

I've got a few more questions after i've managed to set up Tornado with Nginx on Ubuntu 10.04 ( Server edition ) for production use.

I've managed to set up my nginx.conf file and it looks like this:

# Set another default user than root for security reasons

root       www www;

# As a thumb rule: One per CPU. If you are serving a large amount

# of static files, which requires blocking disk reads, you may want

# to increase this from the number of cpu_cores available on your

# system.

#

# The maximum number of connections for Nginx is calculated by:

# max_clients = worker_processes * worker_connections

worker_processes 1;

# Maximum file descriptors that can be opened per process

# This should be > worker_connections

worker_rlimit_nofile 8192;

events {

  # When you need > 8000 * cpu_cores connections, you start optimizing

  # your OS, and this is probably the point at where you hire people

  # who are smarter than you, this is *a lot* of requests.

  worker_connections  8000;

  # This sets up some smart queueing for accept(2)'ing requests

  # Set it to "on" if you have > worker_processes

  accept_mutex off;

  # These settings are OS specific, by defualt Nginx uses select(2),

  # however, for a large number of requests epoll(2) and kqueue(2)

  # are generally faster than the default (select(2))

  # use epoll; # enable for Linux 2.6+

  # use kqueue; # enable for *BSD (FreeBSD, OS X, ..)

}

# Change these paths to somewhere that suits you!

error_log  logs/error.log;

pid        logs/nginx.pid;

http {

  # Set the mime-types

  include       mime.types;

  # And the fallback mime-type

  default_type  application/octet-stream;

  # Format for our log files

  log_format   main '$remote_addr - $remote_user [$time_local]  $status '

    '"$request" $body_bytes_sent "$http_referer" '

    '"$http_user_agent" "$http_x_forwarded_for"';

  # Click tracking!

  access_log   logs/access.log  main;

  # ~2 seconds is often enough for HTML/CSS, but connections in

  # Nginx are cheap, so generally it's safe to increase it

  keepalive_timeout  5;

  # You usually want to serve static files with Nginx

  sendfile on;

  tcp_nopush on; # off may be better for Comet/long-poll stuff

  tcp_nodelay off; # on may be better for Comet/long-poll stuff

  # Enable Gzip

  gzip  on;

  gzip_http_version 1.0;

  gzip_comp_level 2;

  gzip_min_length 1100;

  gzip_buffers     4 8k;

  gzip_proxied any;

  gzip_types text/html text/plain text/xml application/xml application/xml+rss text/css text/javascript application/javascript application/json;

  gzip_static on;

  gzip_proxied        expired no-cache no-store private auth;

  gzip_disable        "MSIE [1-6]\.";

  gzip_vary           on;

  server {

    # listen 80 default deferred; # for Linux

    # listen 80 default accept_filter=httpready; # for FreeBSD

    listen 80 default;

    # e.g. "localhost" to accept all connections, or "www.example.com"

    # to handle the requests for "example.com" (and www.example.com)

    server_name _;

    # Path for static files

    root /srv/www/example.com/public_html/src/static;

    expires 1M;

    # Static assets

    location ~* ^.+\.(manifest)$ {

      expires -1D;

      root   /srv/www/example.com/public_html/src/;

      access_log /srv/www/example.com/logs/static.logs;

    }

    location ~* ^.+\.(ico|ogg|ogv|svg|svgz|eot|otf|woff|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)$ {

      # Only set expires max IFF the file is a static file and exists

      if (-f $request_filename) {

        expires max;

        root   /srv/www/example.come/public_html/src/static;

        access_log /srv/www/example.com/logs/static.logs;

      }

    }

  }

}

My questions for nginx file is:

1) I would want to host my application with the domain name : www.example.com, which line of the nginx.conf file do i change?

2) When I was trying to start nginx by typing /etc/init.d/nginx start, I received the following error message:

Error Message:

Starting nginx: [emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)

[emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)

[emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)

[emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)

[emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)

[emerg]: still could not bind()

nginx.

I have not done any configuration to the nginx.conf file found at /etc/nginx

My questions for this part are:

3)  How do i fix the error? 

4) How do i start nginx automatically?

I've tried running my Tornado app by typing

python app.py

Than I navigate to http://xxx.xx.xxx.xxx:8888 , and my app works correctly.

However, should I close my terminal ( killing the process ) , my tornado app is no longer active.

My question here is:

5) how do i start tornado app automatically?

Thank you all for your kind patience.

Best Regards.

[TornadoRocks]

bump.

anyone help please?

[Jeremy Kelley]

I'll try to quickly answer inline below but many of your questions are
nginx specific and you may find more help on an nginx forum.

-j

On Fri, Mar 11, 2011 at 4:40 AM, TornadoRocks <ye.e...@gmail.com> wrote:
> My questions for nginx file is:
> 1) I would want to host my application with the domain name :
> www.example.com, which line of the nginx.conf file do i change?

server_name

> 2) When I was trying to start nginx by typing /etc/init.d/nginx start, I
> received the following error message:
> Error Message:
> Starting nginx: [emerg]: bind() to 0.0.0.0:80 failed (98: Address already in
> use)
> [emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)
> [emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)
> [emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)
> [emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)
> [emerg]: still could not bind()
> nginx.
> I have not done any configuration to the nginx.conf file found at /etc/nginx
> My questions for this part are:
> 3)  How do i fix the error?

It's already running. kill it.

> 4) How do i start nginx automatically?

This is more of an ubuntu question. Google for "start service at boot
ubuntu". I can't remember which command it is to do this but it's
very simple.

> I've tried running my Tornado app by typing
> python app.py
> Than I navigate to http://xxx.xx.xxx.xxx:8888 , and my app works correctly.
> However, should I close my terminal ( killing the process ) , my tornado app
> is no longer active.
> My question here is:
> 5) how do i start tornado app automatically?

use supervisord to start it.

-j

--
The Christian ideal has not been tried and found wanting;
it has been found difficult and left untried – G. K. Chesterton

[TornadoRocks]

Hi, 

thank you.

I forgot to mention that the nginx.conf is in the same folder as my app.py :

src/

     /static

     /templates

     app.py

     nginx.conf

So when nginx is started, am I starting the config file within src/ ?

Best Regards.

[Cliff Wells]

On Fri, 2011-03-11 at 18:43 -0800, TornadoRocks wrote:
> Hi,
> thank you.
>
>
> I forgot to mention that the nginx.conf is in the same folder as my
> app.py :

This is wrong. Nginx is a completely separate program unrelated to
Tornado. You need to configure /etc/nginx/nginx.conf.

Cliff

[TornadoRocks]

hi, 

thank you for your reply.

I intend to server multiple tornado apps ( each with a domain name ) on a single server.

I suppose a seperate nginx file is required at /etc/nginx/sites-enabled ? If so, what are the contents required for the config file

in sites-enabled ?

Best Regards.

[Cliff Wells]

On Fri, 2011-03-11 at 22:08 -0800, TornadoRocks wrote:
> hi,
> thank you for your reply.
>
>
> I intend to server multiple tornado apps ( each with a domain name )
> on a single server.
>
>
> I suppose a seperate nginx file is required
> at /etc/nginx/sites-enabled ?

Not required (you could put all the config in a single file), but
usually recommended.

> If so, what are the contents required for the config file
> in sites-enabled ?

It depends. There's no "recipe" for a generic website. At the very
least you'll need a server section with directives to proxy to your
Tornado app, but if I were you, I'd at least become somewhat familiar
with Nginx (try serving some static files before moving on to more
complex tasks). There's lots of examples and documentation here:

http://wiki.nginx.org/

Cliff

[Ben Darnell]

It's actually not a bad strategy to keep nginx.conf in the same place as your code (or at least to deploy it in the same way).  Personally I like to disable the nginx that is run by the ubuntu startup scripts and then run "nginx -c my/directory/nginx.conf" from supervisord.  (note that running nginx under supervisord requires "daemon off" in nginx.conf)

-Ben

[Jeremy Kelley]

We run multiple tornado sites behind one nginx instance so storing the
configurations locally to the project would get a little tricky.

Instead we configure a separate config for each tornado project
isntance and then have the /etc/nginx config include those.

-j

--