More bizarre URL behaviour after deploying

[Bernd Wechner]

OK, I've more or less solved my original problems in deploying in this space. Well I've deployed and am happy.

BUT, I am still bamboozled by something and wonder if anyone has any clue or insights to share?

I have these two urlpatterns defined:

    url(r'^list/(?P<model>\w+)', views.view_List.as_view(), name='list')
    url(r'^view/(?P<model>\w+)/(?P<pk>\d+)$', views.view_Detail.as_view(), name='view'),

Works a charm with these template tags:

    <a href="{% url 'list' 'Player' %}">List Players</a>
    <a href="{% url 'view' model_name o.pk %}">View Player {{ o }} </a>

That url tag produces these URLs:

    http://127.0.0.1:8000/list/Player
    http://127.0.0.1:8000/view/Player/1

And it works. I click the links and they work. Well I've been developing this way for  a couple of years and it's always worked ;-). it's great!

BUT, after deploying, this resolves on the home page to:

    http://leaderboard.space/list/Player

And when I click it it works! Awesome. Happy as. Great stuff. Gotta love Django!

BUT, if is open this page:

    http://leaderboard.space/view/Player/1

It renders fine, I see a great view of player 1.

BUT, on that page is the good old list link from above, namely:

    <a href="{% url 'list' 'Player' %}">List Players</a>

and on the dev site this resolves to:

    http://127.0.0.1:8000/list/Player

and here is the thing I'm stuck on! On the deployed site it renders as:

    http://leaderboard.space/view/list/Player

What? Where did that 'view' come from? Of course I click it I get a 404 error:

Page not found (404)
Request Method:    GET
Request URL:    http://leaderboard.space/view/list/Player
Using the URLconf defined in CoGs.urls, Django tried these URL patterns, in this order:

1. ^$ [name='home']
2. ^admin/
3. ^login/$ [name='login']
4. ^logout/$ [name='logout']
5. ^fix [name='fix']
6. ^unwind [name='unwind']
7. ^check [name='check']
8. ^rebuild [name='rebuild']
9. ^kill/(?P<model>\w+)/(?P<pk>\d+)$ [name='kill']
10. ^list/(?P<model>\w+) [name='list']
11. ^view/(?P<model>\w+)/(?P<pk>\d+)$ [name='view']
12. ^add/(?P<model>\w+)$ [name='add']
13. ^edit/(?P<model>\w+)/(?P<pk>\d+)$ [name='edit']
14. ^delete/(?P<model>\w+)/(?P<pk>\d+)$ [name='delete']
15. ^leaderboards [name='leaderboards']
16. ^json/game/(?P<pk>\d+)$ [name='get_game_props']
17. ^json/leaderboards [name='json_leaderboards']
18. ^static\/(?P<path>.*)$

The current path, view/list/Player, didn't match any of these.

clearly. Because I don't look for this bizarre and newly introduced 'view'.

Now to be clear, I have a way around this, but I am not looking for a workaround here, I'm fishing for any clues that help me understand how the url template tag generates its output and when and why it inserts a string like this that seems to be the name of the view that the link is on!

Mystery++ and PITA.

Regards,

Bernd.

[Melvyn Sopacua]

Humor me and call it list_view in the pattern and template references.


--
Melvyn Sopacua

[Bernd Wechner]

Well, not a lot of insight from the broader community on this one, but a warm thank you to Melvyn Sopacua who took the time to think about it a bit and apply his experience which led me to:

    /usr/local/lib/python3.5/dist-packages/django/urls/base.py

where I could instrument Django a bit and find out how and why it was adding this bizarre prefix to the urls.

Alas it's a tad complicated and I'll try and publish a module with the fix some time soon (it relates to lighttpd, uwsgi, django interactions and the management of SCRIPT_NAME and PATH_INFO environment variables which lighttpd doesn't do to Django's satisfaction and so it needs tweaking and I have both a uwsgi application tweak and a middleware tweak for the job now).

Regards,

Bernd.


Bernd Wechner wrote:

[Antonis Christofides]

Alas it's a tad complicated and I'll try and publish a module with the fix some time soon (it relates to lighttpd, uwsgi, django interactions and the management of SCRIPT_NAME and PATH_INFO environment variables which lighttpd doesn't do to Django's satisfaction and so it needs tweaking and I have both a uwsgi application tweak and a middleware tweak for the job now).

Thanks for writing this.

It reminds me of a time, almost two years long, when I was using uwsgi. Every now and then I had similar obscure issues that needed a hell of debugging. After I migrated to Gunicorn all these issues stopped. So I'm wondering: why are so many people using uwsgi? What am I missing?

Antonis Christofides
http://djangodeployment.com

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d3a95675-8137-ab90-54ea-b3e44ae6f53d%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Melvyn Sopacua]

You're missing the fact that it's the scgi module of lighttpd is what
is causing it. uwsgi shouldn't fix it, as it has no idea if this is
intentional, for example when using mounts [1].
As can be seen in the WSGI spec, the WSGI client (uwsgi / gunicorn /
whathaveyou) should trust what is provided here [2]. If gunicorn does
it differently, it will prevent you from mounting an app in a virtual
location, say "/django/".

[1] http://uwsgi-docs.readthedocs.io/en/latest/Nginx.html#hosting-multiple-apps-in-the-same-process-aka-managing-script-name-and-path-info
[2] https://www.python.org/dev/peps/pep-0333/#environ-variables

> https://groups.google.com/d/msgid/django-users/f1ca6e9e-4948-dcd5-cf28-1b95b40eabce%40djangodeployment.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
Melvyn Sopacua

[Antonis Christofides]

That particular issue may be a lighttpd issue, but my question is more general.
Unfortunately I don't remember the kind of trouble I was having with uwsgi.

Are there any people who have used both uwsgi and gunicorn and have a preference
for the former? What benefits have they seen?

Antonis Christofides
http://djangodeployment.com

[Bernd Wechner]

Antonis,

Well truth be told I'm with uwsgi only because of the snowball effect you're alluding to, namely we have so many options I'm like looking on-line for what the most popular and recommended one is, and uwsgi has a huge fan base.  As to server, I opted for lighttpd only because I already have it installed (on an OpenWRT router the LuCI interface is driven by lighttpd by default) and so I wanted to keep to one system and installed it on the webserver too (so I've got one set of configs/quirks to keep my head wrapped around).

That said in the stack lighttpd->uwsgi->django for the issue I just fixed the least culpable and probably off scot-free is uwsgi. It's arguably a django bug.

The technical issue is that a URL like this:

        http://domain.tld/script/path

is split into these two variables by Lighttpd:

    SCRIPT_NAME: /script
    PATH_INFO: /path

and Django wants this:

    SCRIPT_NAME:
    PATH_INFO: /script/path

Now if you override get_wsgi_application you can pull the switch simply inside of the Django app you're writing with something like this in the WSGIHandler:

        environ['PATH_INFO'] = environ.get('SCRIPT_NAME','') + environ.get('PATH_INFO','')
        environ['SCRIPT_NAME'] = ''

and that works a charm. Everything rocks from there on in. It's lighttpd that does this split and Django that has the expectations, uwsgi is just providing the evironent lighttpd wants to pass on, and giving to Django.  It's not really at fault here in any way.

The specific issue I was hung on, was if you move this switch to middleware, where I think it is more at home because middleware is a well established and welcome way to inject code between Django and the outside world (hence middle ware) then the Django url template tag starts writing URLs like:

        http://domain.tld/script/script/path

What? And it turns out that IMO it's a Django bug of sorts, because Django squirrels SCRIPT_NAME away before calling middleware, so if you tweak SCRIPT_NAME in middleware, Django don't care no more. Which defeats the purpose of middleware. So IMO Django should squirrel SCRIPT_NAME away after middleware is run!

As it happens I explicitly squirrel it away for Django in the middleware and all comes good:

from django.core.urlresolvers import set_script_prefix
set_script_prefix(.request.environ['SCRIPT_NAME'])

And now the fiddle works in middleware too where I feel it's more at home than with a UWSGIHandler override (i.e. theoretically is more robust against Django evolutions).

Regards,

Bernd.

On 14/09/2017 1:27 AM, Antonis Christofides wrote:

[James Schneider]

The technical issue is that a URL like this:

        http://domain.tld/script/path

is split into these two variables by Lighttpd:

    SCRIPT_NAME: /script
    PATH_INFO: /path

and Django wants this:

    SCRIPT_NAME:
    PATH_INFO: /script/path

Your example is a bit misleading/lacking. Both of these scenarios can be correct (or incorrect) based on the web server process configuration.

I did a bit of digging and it looks like lighttpd >= 1.4.23 supports a "fix-root-scriptname" directive that appears to exactly solve your problem. Have you tried that?

https://redmine.lighttpd.net/issues/729#note-11

https://xn--tigreray-i1a.org/Using%20MoinMoin%20with%20lighttpd%20for%20a%20root%20wiki

That directive doesn't appear to be documented very well anywhere else.

-James