HTML content truncated using Tornado + nginx

[Jorge Escobar]

Hi all,

I've been debugging this issue for the last 4 days with no luck, and I'm posting here with the hope that someone has seen something similar.

I have a chat application running on Tornado (http://geochat.io) with nginx as the proxy server. I have the configuration exactly like the one recommended here: http://www.tornadoweb.org/documentation/overview.html?highlight=nginx#running-tornado-in-production

For some reason, users are seeing a truncated version of the page, where there's only a partial load of it. Literally the view source shows the stream stopping mid way, sometimes a little longer, sometimes a little shorter.

I had the users test hitting tornado directly (I exposed port 8000 for them) and they are seeing the content load fine, so it's definitely nginx. I also saw the behavior in at least 3 different computers.

I'm happy to offer more details as needed.

Thanks in advance for all your help!

[Phil Whelan]

Hi Jorge,

If "it's definitely nginx", then this is probably not the mailing list your looking for :)

Cheers,
Phil

Sent from my iPhone

[Ben Darnell]

Make sure that all of nginx's temporary directories exist and are
writeable by the nginx process. If one of them (I think
proxy_temp_path) doesn't exist, then small pages will work but larger
ones will be truncated.

-Ben

[Jorge Escobar]

Hi Ben,

Thanks for the response. I had suspicions this was the case, even though nginx error log, tornado error log or system error log weren't saying anything about this. I enabled gzip (I didn't have it enabled):

http {

    # Enumerate all the Tornado servers here

    upstream frontends {

        server 127.0.0.1:8000;

    }

    include       /etc/nginx/mime.types;

    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

                      '$status $body_bytes_sent "$http_referer" '

                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    proxy_read_timeout 200;

    tcp_nopush on;

    tcp_nodelay on;

    gzip  on;

    gzip_min_length 1000;

    gzip_proxied any;

    gzip_types text/plain text/html text/css text/xml

               application/x-javascript application/xml

               application/atom+xml text/javascript;

    # Only retry if there was a communication error, not a timeout

    # on the Tornado server (to avoid propagating "queries of death"

    # to all frontends)

    proxy_next_upstream error;

    # Load config files from the /etc/nginx/conf.d directory

    include /etc/nginx/conf.d/*.conf;

}

...and added the proxy_temp_path on the location / module (and of course created the path on /tmp/nginx with proper permissions):

    location / {

        proxy_pass_header Server;

        proxy_set_header Host $http_host;

        proxy_redirect off;

        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header X-Scheme $scheme;

        proxy_pass http://frontends;

        proxy_temp_path /tmp/nginx 1 2;

    }

The page is now loading fine for the users, even though I don't see any files being written on /tmp/nginx. My guess at this point is that gzip is taking care of the file limit for most of the requests, and that I will see pages written in /tmp/nginx if they're much bigger.

One minor thing I'm noticing is that nginx complains about a duplicate MIME type upon startup:

2012/08/15 16:56:06 [warn] 29250#0: duplicate MIME type "text/html" in /etc/nginx/nginx.conf:73

Not sure what that is about.

Thanks!

[Jorge Escobar]

I just found this flag that prints out all the configure arguments (in yellow the temp paths):

[jorge@jg-www01 ~]$ nginx -V

nginx version: nginx/0.8.54

built by gcc 4.5.1 20100924 (Red Hat 4.5.1-4) (GCC) 

TLS SNI support enabled

configure arguments: --user=nginx --group=nginx --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-mail --with-file-aio --with-mail_ssl_module --with-ipv6 --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=-Wl,-E

So I went to look at the paths and they were all created properly (albeit no nginx group write access):

[jorge@jg-www01 ~]$ ls -al /var/lib/nginx/tmp

total 28

drwxr-xr-x  7 nginx nginx 4096 Jul  2 19:49 .

drwxr-xr-x  3 nginx nginx 4096 Jul  2 19:46 ..

drwx------  2 nginx root  4096 Aug 15 13:23 client_body

drwx------ 12 nginx root  4096 Jul  2 22:28 fastcgi

drwx------ 12 nginx root  4096 Jul  4 00:06 proxy

drwx------  2 nginx root  4096 Jul  2 19:49 scgi

drwx------  2 nginx root  4096 Jul  2 19:49 uwsgi

So I'm taking out the /tmp/nginx path from the conf file, turning off gzip and making these directories publicly writeable to see what happens.