Routing problems behind nginx

[Jim Steil]

Hi

Testing deployment on Linode Ubuntu 20.04 server.  Site is mostly working except for the only form I have.  It displays fine, but when I click on submit it tries routing me to http://127.0.0.1:8000/hiring_board/profile instead of http://45.79.24.85/hiring_board/profile which is where I came from.

nginx sites-available/py4web

server {
        listen          80;
        server_name     $hostname;
        client_max_body_size 20M;
        proxy_connect_timeout 10;
        proxy_send_timeout 15;
        proxy_read_timeout 20;
        location / {
            proxy_pass http://127.0.0.1:8000;
        }
        location ~* ^/(\w+)/static(?:/_[\d]+\.[\d]+\.[\d]+)?/(.*)$ {
            alias /home/www-data/py4web/apps/$1/static/$2;
            expires max;
        }
}

/etc/systemd/system/py4web.service

[Unit]
Description=py4web service

[Service]
ExecStart=/usr/bin/python3 /home/www-data/py4web/py4web.py run --password_file /home/www-data/py4web/password.txt /home/www-data/py4web/apps
Type=simple

[Install]
WantedBy=multi-user.target

Any suggestions?

-Jim

[Jim Steil]

I received the following from Val

Hi Jim!

It seems that google deleted my messages on the forum  - can't understand why

so it works for me (before proxy_pass):

   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

   proxy_set_header Host $http_host;

   proxy_set_header   X-Real-IP $remote_addr;

   proxy_set_header X-Forwarded-Proto $scheme;

   proxy_set_header   X-Forwarded-Host $server_name;

   #  proxy_set_header X-Script-Name /sub_location/;  # - works like subdomain

   #  proxy_redirect off;       # - turn off  inherited proxy_redirect settings, if any

[Jim Steil]

Thanks Val

I added that so now my sites-available/py4web reads

server {
        listen          80;
        server_name     $hostname;
        client_max_body_size 20M;
        proxy_connect_timeout 10;
        proxy_send_timeout 15;
        proxy_read_timeout 20;
        location / {

                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header   X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header   X-Forwarded-Host $server_name;

                proxy_pass      http://127.0.0.1:8000;
        }
        location ~* ^/(\w+)/static(?:/_[\d]+\.[\d]+\.[\d]+)?/(.*)$ {
            alias /home/www-data/py4web/apps/$1/static/$2;
            expires max;
        }
}

Now it is redirecting me to http://localhost/hiring_board/profile instead of 127.0.0.1

Any idea what I have messed up?

-Jim

[Jim Steil]

Val and I have continued this conversation throughout the afternoon, and a transcript of it is below.

TLDR;

In py4web, when using Form(), the action is set to request.url.  In my situation (ubuntu instance on linode) my nginx config is not passing the proper 'hostname' to build the url.  On linode, by default the hostname is set to localhost.  

request.url generates an absolute url in the <form action=""> therefore, the hostname being passed from nginx proxy_set_header config is used to build the URL.

I can get it to work properly if I change the server_name attribute in my nginx config to either the IP address or the dns name used to access the site (and then access it using the ip or dns name accordingly).

I can also get it to work if I change form.py to use request.path instead of request.url, or if we just use _action="#".  It appears to me that this is what web2py does.

Anyone else have thoughts on this?

-Jim

Val K

1:09 PM (3 hours ago)

Hi Jim! It seems that google deleted my messages on the forum - can't understand why so it works for me (before proxy_pass): proxy_set_header X-Forwarded-For $p

Jim Steil

1:21 PM (3 hours ago)

I received the following from Val

Jim Steil

1:25 PM (3 hours ago)

Thanks Val I added that so now my sites-available/py4web reads Now it is redirecting me to http://localhost/hiring_board/profile instead of 127.0.0.1 Any idea w

Val K

1:31 PM (3 hours ago)

try to uncomment proxy_redirect off; ( I have proxy_redirect off on my server) ср, 30 сент. 2020 г. в 21:25, Jim Steil <ato....@gmail.com>: To unsubscribe fro

Jim Steil

1:37 PM (3 hours ago)

Val I've not got this in my nginx sites-available proxy_redirect off; still redirecting to localhost on form submit. all other links working fine.

Val K

1:43 PM (3 hours ago)

What does url/python code for this button look like? ср, 30 сент. 2020 г. в 21:37, Jim Steil <ato....@gmail.com>:

Val K

1:45 PM (3 hours ago)

I mean it seems that python code generates wrong URL for submit-button ср, 30 сент. 2020 г. в 21:41, Val K 

Val K

1:49 PM (2 hours ago)

also it maybe py4web Vue-based forms issue, if you use one ср, 30 сент. 2020 г. в 21:43, Val K

Jim Steil

1:50 PM (2 hours ago)

It is just the submit button that is generated for Form() <input class="button" type="submit" value="Submit"> But, the <form> tag is generating as <form action=

Jim Steil

1:52 PM (2 hours ago)

I'm using html forms. Here is the relevant code in forms.py form = FORM(_method="POST", _action=request.url, _enctype="multipart/form-data")

Val K

1:55 PM (2 hours ago)

_action=request.url -> _action=URL() #- should solve the problem ср, 30 сент. 2020 г. в 21:53, Jim Steil <ato....@gmail.com>:

Val K

1:57 PM (2 hours ago)

I think request.url is local url, but URL() uses proxy_headers (if any) to generate url ср, 30 сент. 2020 г. в 21:54, Val K

Jim Steil

2:01 PM (2 hours ago)

I will try it

Jim Steil

2:11 PM (2 hours ago)

If I use just URL() it routes me to /hiring_board/ and post isn't an allowed method. Somehow I need to get it to put the rest of the current URL in there too.

Val K

2:18 PM (2 hours ago)

yes, sorry, it should be URL( scheme= True) - to get full url ср, 30 сент. 2020 г. в 22:11, Jim Steil <ato....@gmail.com>:

Jim Steil

2:19 PM (2 hours ago)

Ok, I got it working with form = FORM(_method="POST", _action=request.path, _enctype="multipart/form-data") using request.path instead of request.url. Does that

Jim Steil

2:19 PM (2 hours ago)

Just got your message about sheme=True Will try that if you think it is safer than request.path

Val K

2:20 PM (2 hours ago)

URL('profile', scheme= True) ср, 30 сент. 2020 г. в 22:17, Val K

Jim Steil

2:22 PM (2 hours ago)

Just tested URL(scheme=True) also returned localhost. request.path is working though

Jim Steil

2:23 PM (2 hours ago)

scheme=True is inserting localhost, not the public URL.

Val K

2:24 PM (2 hours ago)

I made on my server - request.url - should work too! also from the bottle.py doc: so did you restart nginx after fix config? ср, 30 сент. 2020 г. в 22:19, Val K

Jim Steil

2:25 PM (2 hours ago)

yes /etc/init.d/nginx restart Will test again.

Jim Steil

2:29 PM (2 hours ago)

Still getting the error with request.url /etc/nginx/sites-available/py4web

Val K

2:45 PM (2 hours ago)

It is strange, could you make a simple test - just include in controller return dict( ..., rurl = request.url) and insert it in the template [[=rurl]]? - if it

Jim Steil

2:48 PM (1 hour ago)

I will do more testing and post a summary of this back to the group. Thanks for all your help, you seem to have a lot more knowledge in this area than I do. It

Val K

3:09 PM (1 hour ago)

final suggestion: server_name 45.79.24.85 # instead of $hostname - it seems that $hostname is `localhost` ср, 30 сент. 2020 г. в 22:48, Jim Steil <ato.steil@gma

Jim Steil

3:14 PM (1 hour ago)

That definitely made a difference. So, once I have a DNS name for that IP I should switch it back to $hostname?

Val K

3:17 PM (1 hour ago)

usually server_name is a list of domains: ср, 30 сент. 2020 г. в 23:14, Jim Steil <ato....@gmail.com>:

Val K

3:18 PM (1 hour ago)

It works for me ( before proxy_pass): # proxy_set_header X-Script-Name /sub_location/; # - like subdomain proxy_redirect off; среда, 30 сентября 2020 г. в 19:37

Val K

3:18 PM (1 hour ago)

works for me (before proxy_pass): # proxy_set_header X-Script-Name /sub_loctaion/; # - works like subdomain # proxy_redirect off; # - off previous proxy_redirec

Val K

3:18 PM (1 hour ago)

Hi, Jim! posted by email To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/f99841e7-f937-4362-9960-f502a594bfcfn%40googlegroups.

Val K

3:18 PM (1 hour ago)

среда, 30 сентября 2020 г. в 21:25:00 UTC+3, ato....@gmail.com: To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/1e72b9ba-9de9-

Val K

3:19 PM (1 hour ago)

but also could be IP-address https://nginx.org/en/docs/http/server_names.html ср, 30 сент. 2020 г. в 23:16, Val K

Jim Steil

3:40 PM (1 hour ago)

And, can you access it via IP address or by server name?

Val K

3:51 PM (56 minutes ago)

yes, tested on my server just now - I added ip of my server to domain (like server_name example.com 1.2.3.4) - works fine through domain and ip access! ср, 30 с

Val K

4:03 PM (44 minutes ago)

just in case - I registered domain at mooo.com (freeDNS provider)Without domain it is impossible to use Let's Encrypt ср, 30 сент. 2020 г. в 23:50, Val K <valq7

Jim Steil

4:05 PM (42 minutes ago)

Val I looked at a form generated by web2py and the action in it is '#' Seems to me that the better way to go is to use a relative url (request.path) or to use j

Jim Steil

4:05 PM (42 minutes ago)

I think it would be nice if the form action wasn't dependent on the nginx config.

Val K

4:28 PM (19 minutes ago)

to me

It does not depend  - your py4web is behind nginx,  the problem is that nginx passes to py4web (i.e. WSGI-server - gunicorn/gevent/tornado  - doesnt matter) wrong X-Forwarded-Host which is  $server_name which is set to $hostname which is `localhost`, $hostname is the name of the machine on the network, so as far as I understand, to get  your config working, your machine should be exposed directly in the internet (no router, internet cable inserted directly into PC), or machine local-network-name should matches the external IP address

чт, 1 окт. 2020 г. в 00:06, Jim Steil <ato....@gmail.com>:

Val K

4:38 PM (9 minutes ago)

to me

if you use linux-like OS  - you can just type in the console `hostname` to get machine name

чт, 1 окт. 2020 г. в 00:26, Val K 

Jim Steil

4:47 PM (0 minutes ago)

to Val

FWIW, this is an ubuntu instance running on linode.

...and the hostname from the terminal is indeed 'localhost'

But, if I change my hostname to 'webserver1' then it is still going to be delivering the wrong url as there is no dns setup for 'webserver1'.

I'm going to move some web2py apps to the same box and mess with the nginx config.  Not sure how much time I'll get working on it the next couple of days but hoping to learn some more.

I'll post a summary of this to the google group.  Please respond to it if you feel I'm misrepresenting anything...

-Jim

[Val K]

I think relative urls maybe a problem if you use parametric routes (like action('/abc/<some:path>'))

четверг, 1 октября 2020 г. в 01:00:52 UTC+3, ato....@gmail.com:

[Jim Steil]

Val - I don't know anything about that. 

-Jim