Serving Static File with Permissions using Nginx, Uwsgi

[Boyi]

HI,
I use pyramid, uwsgi with nginx. Before this, I serve my static pdf
files using nginx. The files can be download fast. But now, because I
set the permission for the files to a directory called '/resources'
for a certain group of users by using add_static_view, my
understanding is the file will now be serve by uwsgi instead of
directly by nginx server. The '/static' directory will now only serve
js, css and template files.

The problem that I faced now is that, when users try to download the
pdf file now, the download process will stuck at certain time, and
sometimes, will stop prematurely. Even if the download complete, it
will take a long time even when the file is small.

my nginx.conf and uwsgi parameter are as below. There should be
something wrong with my configurations. Can somebody bring some light
to solve my probs?

TQ.

----------------------------------------------------
#nginx.conf
user user user;
worker_processes 2;

events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;


sendfile on;
#tcp_nopush on;

keepalive_timeout 65;

server {
listen 80;
server_name myweb.com;
access_log /path/to/directory/file.log;
error_log /path/to/log/debug/file.log debug;
charset utf-8;
root /path/to/my/app;
#index Home index.html index.htm;

location /static {
root /path/to/my/app;
expires max;
}

#for robot.txt and favicon
location ~* \.(txt|ico)$ {
root /path/to/my/app/static/directory;
expires max;
}

location / {
nclude uwsgi_params;
uwsgi_pass unix:/tmp/uwsgi.sock;
#uwsgi_param SCRIPT_NAME /;
}
}

}

----------------------------------------------
my uwsgi parameter:
socket = /tmp/uwsgi.sock
home = /my/home/path
daemonize = /path/to/log/uwsgi.log

[Graham Higgins]

Dunno if it's relevant but you have a typo in the posted config at least:

location / { 
            nclude     uwsgi_params; 

[Sharil Shafie]

Not relevant unfortunately. That's pasting error.

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" gro...

To view this discussion on the web visit https://groups.google.com/d/msg/pylons-discuss/-/ID2ey966l28J.

To post to this group, send email to pylons-...@googlegroups.com.
To unsubscribe from this grou...

[Michael Merickel]

You can still have nginx serve your static files after pyramid has checked the permissions by having pyramid return a response containing the x-accel-redirect header. For more info about it look into nginx's version of X-Sendfile.

http://wiki.nginx.org/X-accel

http://wiki.nginx.org/XSendfile

--

Michael

[Sharil Shafie]

I did follow your suggestions. The files in the directory and its sub directories will be served. I did go along well with the permission stuff. But the problem still persist. If I try to open the pdf through google chrome, if the download stuck, it will stuck forever even if i reload. It would stuck at the same byte size. The pdf file wont be displayed. I think it's something wrong with  uwsgi or nginx config but cant figure out why. I might be some something to do with the my system resources. I didnt get this problem if I use paster. 

        

location /resources/~*  {

                #internal;

                root /root/to/app;

                expires max;

            }

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.

To post to this group, send email to pylons-...@googlegroups.com.

To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.

[Sharil Shafie]

ops. The #internal was actually be uncommented. I commented it as temporary measure.

[Sharil Shafie]

Finally able to directly serve static files with permission using nginx. Thanks Michael for your suggestion. 

Just want to share my approach below:

#location configuration in nginx.conf

        

        location ^~ /resources/  {

                internal;

                root /path/to/myapp;

            }

--------------------------------------

# __init__.py

    #add route for static files with permissions

    config.add_route('resource_file', '/MyFiles/{filename}')

    config.add_view(view='myapp.views.resource_file',

    route_name='resource_file',

    permission='special')

----------------------------------------------

# myapp.views

import os

from pyramid.response import Response

def resource_file(request):

_here = os.path.dirname(__file__)

file_name = request.matchdict['filename']

pdf_file = open(os.path.join(_here, 'resources',file_name)).read()

response = Response(content_type='application/pdf',

headers={'X-Accel-Redirect':'/resources/'+file_name},

body=pdf_file)

return response

[Michael Merickel]

Your solution actually isn't any more efficient than what you had before, so I'd expect you'd see a similar performance issue.

The point of X-Accel-Redirect is that you do not have to open the file and read it in Python. Your view should return a simple Response object with no body, and just the appropriate headers (possibly content-type, as well as status_code). nginx will see that header and add the body to the response for you outside of Python, leaving your app free to service other requests while nginx handles the file I/O.

If you're able to make these improvements and it works, I'd like to add this to the Pyramid cookbook. So hopefully either you can issue a pull request to the cookbook or just paste your changes here and I'll write it up. :-)

--

Michael

[Sharil Shafie]

OK.  That means that the response doesnt contain that read file code. I'll think about it. Btw, I am quite slow in this kind of thing. I am not really a programmer. Just a math lecturer with a hobby. Hopefully I'll find a way to improve it as suggested.

 

--

Michael

--

[Michael Merickel]

On Mon, Oct 10, 2011 at 2:34 PM, Sharil Shafie <sish...@gmail.com> wrote:

OK.  That means that the response doesnt contain that read file code.

That's the idea. The response body would be filled in by nginx.

Regardless this was all just a suggestion, there are other reasons why your original response may be having issues when served directly by Pyramid. For example, there could be misconfigured timeouts or size limitations, but it's hard to say. Pyramid should have no trouble serving up static files under light-moderate load. It's just that serving a file takes time, and during that time your application won't be able to service many other requests because it'll occupy one of the threads in your WSGI server's threadpool.

--

Michael

[Sharil Shafie]

It seem that I would be just fine to just remove the read file code. Then nginx will take over the static file by serving the file provided in the X-Accel-Redirect header. 

#location configuration in nginx.conf

        

        location ^~ /resources/  {

                internal;

                root /path/to/myapp;

            }

--------------------------------------

# __init__.py

    #add route for static files with permissions

    config.add_route('resource_file', '/MyFiles/{filename}')

    config.add_view(view='myapp.views.resource_file',

     route_name='resource_file',

     permission='special')

----------------------------------------------

# myapp.views

import os

from pyramid.response import Response

def resource_file(request):

_here = os.path.dirname(__file__)

file_name = request.matchdict['filename']

[Sharil Shafie]

Anyway, below are the codes. Feel free to take and modify it if you think it can be include in the cookbook. By the way, I put status code as 200. It can be changed anyway. As has been put just now, provide the header of  X-Accel-Redirect header containing the static file location,  nginx will take over to serve the static file. It was the ignorance of me to put the body before.  TQ.

#file are located in /resources

#eg to open /resources/sample.pdf -> http://www.mysite.com/MyFiles/sample.pdf

#location configuration in nginx.conf

        location ^~ /resources/  {

        internal;

        root /path/to/myapp;

     }

       

--------------------------------------

# __init__.py

    #add route for static files with permissions

    config.add_route('resource_file', '/MyFiles/{filename}')

    config.add_view(view='myapp.views.resource_file',

    route_name='resource_file',

    permission='special')

----------------------------------------------

# myapp.views

import os

from pyramid.response import Response

def resource_file(request):

_here = os.path.dirname(__file__)

filename = request.matchdict['filename']

response= Response(content_type='application/pdf', headers={'X-Accel-Redirect':'/resources/'+filename})

response.status='200 OK'

return response