Luigi tornado server not working with Nginx reverse_proxy

96 views
Skip to first unread message

Tashrif

unread,
Aug 11, 2020, 3:18:33 PM8/11/20
to Tornado Web Server
Hi all,

I launched my luigid server as:
luigid --address 127.0.0.1 --port 8082

# For the following Nginx proxy_pass, it works fine:

location / {

            proxy_pass http://127.0.0.1:8082;

} 

curl -L http://`hostname`/ shows Luigi server content with the following Tornado log:
2020-08-11 INFO: 302 GET / (127.0.0.1) 0.47ms
2020-08-11 INFO: 200 GET /static/visualiser/index.html (127.0.0.1) 0.77ms



# However, if I have `/luigi/` in the location, it does not:

location /luigi/ {

            proxy_pass http://127.0.0.1:8082/$1;

} 
curl -L http://`hostname`/luigi/ yields error 404 with the following Tornado log:
2020-08-11 INFO: 302 GET / (127.0.0.1) 0.47ms 
Note that, the second line from before is missing.

Here is how Luigi launches the Tornado server.

Can anybody please help to configure a Luigi server behind Nginx reverse proxy for a non-root URL?

Thanks,
Tashrif

Ben Darnell

unread,
Aug 15, 2020, 12:13:27 PM8/15/20
to Tornado Mailing List
On Tue, Aug 11, 2020 at 3:18 PM Tashrif <tashri...@gmail.com> wrote:
Note that, the second line from before is missing.


That's because `/static/visualizer/image.html` no longer matches your location rule, so nginx isn't sending it to Tornado. (it should show up in the nginx logs)

Unfortunately, the URL rewriting functionality in nginx that you're using here isn't fully transparent - it won't work if the application behind the rewrites uses absolute links (which most Tornado applications do, thanks at least to the `static_url` template function). I think you have four options:
1. Make Luigi add a prefix `/luigi/` to all of its url routes (including `static_url_prefix`), perhaps by adding a new configuration parameter. This is the approach I always take; so my backend applications are always aware of the URL at which they are being served.  
2. Make Luigi use only relative links (replacing all uses of `static_url`, among other things)
3. Use nginx's `sub_filter` feature to rewrite links
4. Give up on non-root locations and give each application it's own port or hostname

-Ben

Tashrif

unread,
Aug 15, 2020, 12:38:28 PM8/15/20
to Tornado Web Server
Hi Ben,

Thanks for your insightful reply. By this time, I figured the following (from luigi/server.py):

300     handlers = [
301         (r'/api/(.*)', RPCHandler, {"scheduler": scheduler}),
302         (r'/luigi/', RootPathHandler, {'scheduler': scheduler}),
303         (r'/tasklist', AllRunHandler, {'scheduler': scheduler}),
304         (r'/tasklist/(.*?)', SelectedRunHandler, {'scheduler': scheduler}),
305         (r'/history', RecentRunHandler, {'scheduler': scheduler}),
306         (r'/history/by_name/(.*?)', ByNameHandler, {'scheduler': scheduler}),
307         (r'/history/by_id/(.*?)', ByIdHandler, {'scheduler': scheduler}),
308         (r'/history/by_params/(.*?)', ByParamsHandler, {'scheduler': scheduler}),
309         (r'/metrics', MetricsHandler, {'scheduler': scheduler})
310     ]

And then curl -L http://`hostname`/luigi/ works fine.

However, keeping line 302 pristine (r'/', RootPathHandler, {'scheduler': scheduler}) and including `static_url_prefix` as follows didn't solve the problem:

295     settings = {"static_path": os.path.join(os.path.dirname(__file__), "static"),
296                 "static_url_prefix":"/luigi/",
297                 "unescape": tornado.escape.xhtml_unescape,
298                 "compress_response": True,
299                 }

What do you think about the latter?

Best,
Tashrif

Tashrif B.

unread,
Aug 15, 2020, 1:43:05 PM8/15/20
to python-...@googlegroups.com
Let's keep our discussion limited to native access first. Introduction of `static_url_prefix` like above doesn't make the following work, let alone Nginx reverse_proxy:

<html><title>403: Forbidden</title><body>403: Forbidden</body></html>

Tornado log:
2020-08-15 13:40:05,716 tornado.general[6527] WARNING: 403 GET /luigi/ (127.0.0.1):  is not a file
2020-08-15 13:40:05,717 tornado.access[6527] WARNING: 403 GET /luigi/ (127.0.0.1) 1.30ms

Thanks again,
Tashrif

--
You received this message because you are subscribed to a topic in the Google Groups "Tornado Web Server" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-tornado/FAcEpSBHxZQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-tornad...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-tornado/06341ea7-f9f8-40c8-9f90-b18c433eeecdn%40googlegroups.com.

Ben Darnell

unread,
Aug 29, 2020, 2:56:48 PM8/29/20
to Tornado Mailing List
On Sat, Aug 15, 2020 at 12:38 PM Tashrif <tashri...@gmail.com> wrote:
Hi Ben,

Thanks for your insightful reply. By this time, I figured the following (from luigi/server.py):

300     handlers = [
301         (r'/api/(.*)', RPCHandler, {"scheduler": scheduler}),
302         (r'/luigi/', RootPathHandler, {'scheduler': scheduler}),
303         (r'/tasklist', AllRunHandler, {'scheduler': scheduler}),
304         (r'/tasklist/(.*?)', SelectedRunHandler, {'scheduler': scheduler}),
305         (r'/history', RecentRunHandler, {'scheduler': scheduler}),
306         (r'/history/by_name/(.*?)', ByNameHandler, {'scheduler': scheduler}),
307         (r'/history/by_id/(.*?)', ByIdHandler, {'scheduler': scheduler}),
308         (r'/history/by_params/(.*?)', ByParamsHandler, {'scheduler': scheduler}),
309         (r'/metrics', MetricsHandler, {'scheduler': scheduler})
310     ]

And then curl -L http://`hostname`/luigi/ works fine.

However, keeping line 302 pristine (r'/', RootPathHandler, {'scheduler': scheduler}) and including `static_url_prefix` as follows didn't solve the problem:

295     settings = {"static_path": os.path.join(os.path.dirname(__file__), "static"),
296                 "static_url_prefix":"/luigi/",
297                 "unescape": tornado.escape.xhtml_unescape,
298                 "compress_response": True,
299                 }

What do you think about the latter?

To use my "option 1", you have to change *all* the URLs: add `/luigi/` to all of lines 301-309, and set `"static_url_prefix": "/luigi/static/"`.

-Ben
 

Best,
Tashrif

On Saturday, August 15, 2020 at 12:13:27 PM UTC-4 Ben Darnell wrote:
On Tue, Aug 11, 2020 at 3:18 PM Tashrif <tashri...@gmail.com> wrote:
Note that, the second line from before is missing.


That's because `/static/visualizer/image.html` no longer matches your location rule, so nginx isn't sending it to Tornado. (it should show up in the nginx logs)

Unfortunately, the URL rewriting functionality in nginx that you're using here isn't fully transparent - it won't work if the application behind the rewrites uses absolute links (which most Tornado applications do, thanks at least to the `static_url` template function). I think you have four options:
1. Make Luigi add a prefix `/luigi/` to all of its url routes (including `static_url_prefix`), perhaps by adding a new configuration parameter. This is the approach I always take; so my backend applications are always aware of the URL at which they are being served.  
2. Make Luigi use only relative links (replacing all uses of `static_url`, among other things)
3. Use nginx's `sub_filter` feature to rewrite links
4. Give up on non-root locations and give each application it's own port or hostname

-Ben

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages