Bizarre URL behaviour after deploying

[Bernd Wechner]

This has bamboozled me some. And the best thing I've found on-line doesn't seem to apply:

    https://stackoverflow.com/questions/26944908/django-url-mapping-how-to-remove-app-name-from-url-paths

Let me summarise.

I have a site that I've been building and testing with djangos development server of course. And it implements URLS like:

http://127.0.0.1:8000/list/Model
http://127.0.0.1:8000/add/Model
http://127.0.0.1:8000/edit/Model/nn
http://127.0.0.1:8000/view/Model/nn
http://127.0.0.1:8000/delete/Model/nn

Works like a dream, so I'm deploying, locally first, under lighttpd and uwsgi. That is working fine too. Sort of,

These are all reached with links in my templates like yo:

{% url 'list' model %}
{% url 'add' model %}
{% url 'edit' model pk %}
{% url 'view' model pk %}
{% url 'delete' model pk %}

because in urls.py they all have "name"s defined like that.

All honky dory.

Now after deploying everything works nicely, but those same links point to:

http://mysite.tld/app/list/Model
http://mysite.tld/app/add/Model
http://mysite.tld/app/edit/Model/nn
http://mysite.tld/app/view/Model/nn
http://mysite.tld/app/delete/Model/nn

That is the app name is inserted. Odd. And undesired if not a crisis.

But here's what bamboozles me. I can replace "app" in the url with any string at all, "x" say and the site continues to work but those URLs now point to :

http://mysite.tld/x/list/Model
http://mysite.tld/x/add/Model
http://mysite.tld/x/edit/Model/nn
http://mysite.tld/x/view/Model/nn
http://mysite.tld/x/delete/Model/nn

For what it's worth I don't want to quote urls.py and every other bit of possible config here of course, I am mainly interested to know if someone has insights that aren't shared in that SO link above as none of what is shared there seems to apply.

For anyone really keen on code explorations urls.py is here:

https://github.com/bernd-wechner/CoGs/blob/master/CoGs/urls.py

and the whole site is there too.

What I want to understand is why this "app" suddenly appears in my URLs and how I can control it (remove it ideally). But I have looked at urls.py long enough and scratched my head and not found it. settings.py is right next to if you need to inspect it.

Kind regards,

Bernd.

[Daniel Roseman]

How have you deployed this project? Can you show your deployment configuration?

--

DR. 

[mohammad k]

from django.conf.urls import url

from . import views

app_name = 'polls'

urlpatterns = [

    url(r'^$', views.IndexView.as_view(), name='index'),

    url(r'^(?P<pk>[0-9]+)/$', views.DetailView.as_view(), name='detail'),

    url(r'^(?P<pk>[0-9]+)/results/$', views.ResultView.as_view(), name='results'),

    url(r'^(?P<question_id>[0-9]+)/vote/$', views.vote, name='vote'),

]

you have to create a url.py for each app in them folder

On Wed, Aug 30, 2017 at 4:27 PM, mohammad k <k252...@gmail.com> wrote:

like that : 

app_name = 'polls' in polls folder and in url.py

On Wed, Aug 30, 2017 at 4:26 PM, mohammad k <k252...@gmail.com> wrote:

you have Different app in your project yes ?

you have to use app_name in url.py in each app 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/c2b971d4-61be-e74f-e6bc-6f3ed8ce2a65%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[mohammad k]

i can't do that sorry man

Tell me more clearly, so I may help you better

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/8142e4c2-8a1b-4e6a-8bea-7f39416b1e61%40googlegroups.com.

[mohammad k]

and in url.py beside settings.py use code like that :

url('^polls/',include('polls.urls')),

[mohammad k]

you have Different app in your project yes ?

you have to use app_name in url.py in each app 

On Wed, Aug 30, 2017 at 4:11 PM, Bernd Wechner <bernd....@gmail.com> wrote:

[mohammad k]

like that : 

app_name = 'polls' in polls folder and in url.py

On Wed, Aug 30, 2017 at 4:26 PM, mohammad k <k252...@gmail.com> wrote:

[Bernd Wechner]

Daniel,

Yes, I have deployed, that is the problem in a sense. URLs are clean in dev and suddenly contain an app_name when deployed.

Not sure what you mean by configuration? The Django settings are here:

    https://github.com/bernd-wechner/CoGs/blob/master/CoGs/settings.py

The rest of the config is uwsgi under lighttpd, but none of that is likely to impact the appearance of an app_name in my URLs all of a sudden. I don't mind sharing the config files, but it's a distraction I fear. As the site runs fine, only the URLs are modified by Django post deployment for some reason.

I have found more folk wanting same and yet none of the posted solution are useful to me:

https://stackoverflow.com/questions/40322042/remove-django-app-name-from-path-with-custom-domain-heroku
https://stackoverflow.com/questions/18764967/django-admin-removing-app-name-from-url-for-single-app-projects

as my puzzle is why the app name is used post deployment but not in development. I have nowhere asked for it to be used and am not making sense of why it is being used alas.

Kind regards,

Bernd.

--

DR. 

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/8142e4c2-8a1b-4e6a-8bea-7f39416b1e61%40googlegroups.com.

[Bernd Wechner]

Mohammad,

> you have to use app_name in url.py in each app

I think you have misunderstood or misread my issue below. Clearly I do not have to use app_name in url.py, because I don't and it works fine. The problem is the complete inverse, namely I have not asked for the app_name to appear in the URL, and it does not appear in the URL with the development server (manage.py runserver)  and suddenly when deployed under lighttpd and uwsgi Django inserts the app_name into the URL without my ever having asked it to!

I want that to go away, I don't want an app_name in my URLs as I have a single app project so to speak.

Here are two people wanting to do the same (remove that from URL):

https://stackoverflow.com/questions/40322042/remove-django-app-name-from-path-with-custom-domain-heroku
https://stackoverflow.com/questions/18764967/django-admin-removing-app-name-from-url-for-single-app-projects

except none of the answers apply to my situation, because I have not ever defined app_name and it has never appeared in URLS so far (when running under  development). They only appeared, out of the blue, when deploying.

The whole project is here:

https://github.com/bernd-wechner/CoGs

and I want to lose app name form my URLs (and ideally understand why they are there to begin with) not add them!

Kind regards,

Bernd

mohammad k wrote:

you have Different app in your project yes ?

you have to use app_name in url.py in each app

like that : 

app_name = 'polls' in polls folder and in url.py

from django.conf.urls import url

from . import views

app_name = 'polls'

urlpatterns = [

    url(r'^$', views.IndexView.as_view(), name='index'),

    url(r'^(?P<pk>[0-9]+)/$', views.DetailView.as_view(), name='detail'),

    url(r'^(?P<pk>[0-9]+)/results/$', views.ResultView.as_view(), name='results'),

    url(r'^(?P<question_id>[0-9]+)/vote/$', views.vote, name='vote'),

]

you have to create a url.py for each app in them folder

and in url.py beside settings.py use code like that :url('^polls/',include('polls.urls')),

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CACOk0Tz1oBdnusZLeyHpcYSsVhSynKNOSsd%3DQdNoHuzQmkWdJQ%40mail.gmail.com.

[James Schneider]

What I want to understand is why this "app" suddenly appears in my URLs and how I can control it (remove it ideally). But I have looked at urls.py long enough and scratched my head and not found it. settings.py is right next to if you need to inspect it.

Out of sheer curiosity, what is the actual href= value in the source code (ie the actual result of the {% url %} tag resolution)? 

I also notice that you are generating a bunch of links via JS, have you verified that those values are not being polluted somehow?

Do URL resolutions for the admin work correctly?

What about for 'login' and 'logout'? Those two are a few URL's you've defined with trailing slashes. It shouldn't make a difference, but I'm shooting in the dark here.

-James

[Tom Evans]

On Thu, Aug 31, 2017 at 2:09 AM, Bernd Wechner <bernd....@gmail.com> wrote:
> Daniel,
>
> Yes, I have deployed, that is the problem in a sense. URLs are clean in dev
> and suddenly contain an app_name when deployed.
>
> Not sure what you mean by configuration? The Django settings are here:
>
> https://github.com/bernd-wechner/CoGs/blob/master/CoGs/settings.py
>
> The rest of the config is uwsgi under lighttpd, but none of that is likely
> to impact the appearance of an app_name in my URLs all of a sudden. I don't
> mind sharing the config files, but it's a distraction I fear.

I think you will be surprised.

I'm surprised your diagnosis doesn't point you at this straight away,
if the URLs are correct on one site but incorrect on another site, and
both sites run the exact same python/django code, then the error is
certainly in the bits that are different.

Cheers

Tom

[Daniel Roseman]

Yes. Especially as the problem doesn't seem to be "an app name in my urls" but "a random prefix in my urls". We do need to see the lighttpd and uwsgi configs.

--

DR.

[Bernd Wechner]

Tom,

I will credit you there with stating the obvious in a sense, that I was reluctant to accept. Because it seems frankly impossible. But when nothing seems possible, we do need to revisit the "apparently" impossible, so I pulled a sweet trick and tried identical code in devlopment and deployment.

That required an upgrade to Django on development as it had 1.11.2 and the webserver deployed on had 1.11.4 so now both have 1.11.4.

Then I tweaked settings.py so it runs the Development settings on the Webserver.  Essentially the only bit of varied code between the two instances in in settings.py and reads:

ALLOWED_HOSTS = ["127.0.0.1", "leaderboard.space"]

import platform
HOSTNAME = platform.node()

if HOSTNAME == WEBSERVER:
    print("Django settings: Web Server")
else:
    print("Django settings: Development Server")
    DEBUG = True

And I see "Django settings: Development Server" echoed by runserver when it starts and by uwsgi on the web server.

And so now the only difference remaining is that the dev server uses:

    http://127.0.0.1:8000/

and the webserver:

    http://leaderboard.space/

Now when I test on both sites, the problem goes away! So you are on the money my friend! Now there's a menu link say for listing players and it looks respectively like:

http://127.0.0.1:8000/list/Player
http://leaderboard.space/list/Player

Where the latter (without DEBUG = True) links to:

http://leaderboard.space/Leaderboards/list/Player

and that problem goes away just by using DEBUG = True.

Which does not explain it, but pinpoints the cause. I would still love to understand how that is not a Django bug and how to lose that URL snippet without being in DEBUG mode.

BUT, ain't there always a but, although the menu link lost the app_name from the URL It now works on the Development server (i.e. lists players in the database) and on the web server dumps with 404:

Page not found (404)

Request Method:	GET
Request URL:	http://leaderboard.space/list/Player
Raised by:	Leaderboards.views.view_List

Using the URLconf defined in CoGs.urls, Django tried these URL patterns, in this order:

1. ^$ [name='home']
2. ^admin/
3. ^login/$ [name='login']
4. ^logout/$ [name='logout']
5. ^fix [name='fix']
6. ^unwind [name='unwind']
7. ^check [name='check']
8. ^rebuild [name='rebuild']
9. ^kill/(?P<model>\w+)/(?P<pk>\d+)$ [name='kill']
10. ^list/(?P<model>\w+) [name='list']
11. ^view/(?P<model>\w+)/(?P<pk>\d+)$ [name='view']
12. ^add/(?P<model>\w+)$ [name='add']
13. ^edit/(?P<model>\w+)/(?P<pk>\d+)$ [name='edit']
14. ^delete/(?P<model>\w+)/(?P<pk>\d+)$ [name='delete']
15. ^leaderboards [name='leaderboards']
16. ^json/game/(?P<pk>\d+)$ [name='get_game_props']
17. ^json/leaderboards [name='json_leaderboards']
18. ^static\/(?P<path>.*)$

The current path, Player, didn't match any of these.

You're seeing this error because you have DEBUG = True in your Django settings file. Change that to False, and Django will display a standard 404 page.

again I would love to know how that is not a Django bug, because it says "http://leaderboard.space/list/Player" does not match "^list/(?P<model>\w+) [name='list']" which it does on the Development server (understandably, as it does visually here).

Hmmm ... a little closer to seeing what's happening, a little, but not much closer to understanding what is happening alas. And now, code is identical, only the URL is different, and

So trying to converge even that difference, I access it on the webserver (which is a server, and hence lacks GUI, but lynx to the rescue) and I can navigate to http://127.0.0.1 and I see the site and click on the Players link and get:

 Request Method: GET
    Request URL:   http://127.0.0.1/list/Player
     Raised by:    Leaderboards.views.view_List

   Using the URLconf defined in CoGs.urls, Django tried these URL patterns, in this order:
    1. ^$ [name='home']
    2. ^admin/
    3. ^login/$ [name='login']
    4. ^logout/$ [name='logout']
    5. ^fix [name='fix']
    6. ^unwind [name='unwind']
    7. ^check [name='check']
    8. ^rebuild [name='rebuild']
    9. ^kill/(?P<model>\w+)/(?P<pk>\d+)$ [name='kill']
   10. ^list/(?P<model>\w+) [name='list']
   11. ^view/(?P<model>\w+)/(?P<pk>\d+)$ [name='view']
   12. ^add/(?P<model>\w+)$ [name='add']
   13. ^edit/(?P<model>\w+)/(?P<pk>\d+)$ [name='edit']
   14. ^delete/(?P<model>\w+)/(?P<pk>\d+)$ [name='delete']
   15. ^leaderboards [name='leaderboards']
   16. ^json/game/(?P<pk>\d+)$ [name='get_game_props']
   17. ^json/leaderboards [name='json_leaderboards']
   18. ^static\/(?P<path>.*)$

   The current path, Player, didn't match any of these.

   You're seeing this error because you have DEBUG = True in your Django settings file. Change that to False,
   and Django will display a standard 404 page.

Now runserver is using port 8000 and lighttpd is using port 80, but to converge those will have to wait till the morrow I think (I have to lighttpd on port 8000 or runserver on port 80) and my guess is no change.

I note that Django knows it's running under runserver because when running under runserver it runs fine without ALLOWED_HOSTS set and on the web server if that is missing it bombs. To wit, Django knows its context and can be and probably is it seems following different code paths internally based on that context.

Hmmm ...

The bother is, on the deployment server it's not easy to debug (uwsgi is loading and running django).

Regards,

Bernd.

[Bernd Wechner]

Daniel,

it's a red herring. I agree, that's where I'd suspect the issue lies, but I'm still confident it's a red herring. But because I'm stuck even the seemingly impossible needs to be revisited I guess. It's hard to share everything about a webservers config of course in an email, but there are two files of relevance attached if it helps.

lighttp.conf is the lighttp configuration which talks to uwsgi, and uwsgi.ini is the uwsgi config that loads the django app and serves it (over a unix domain socket to lighttpd). The former is incomplete as it has includes (it gets complicated fast I guess), but the relevant section is in fact repeated to serve the domain name and the 127.0.0.1 URL as I was just testing a convergence plan after reading Tom's suggestion below (and rather than read up on "or" conditional in lighttpd confs I just duplicated the block verbatim for now.

Regards,

Bernd.

Daniel Roseman wrote:

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/92ba18a5-4332-493b-a268-928e1f5feabc%40googlegroups.com.

[Bernd Wechner]

Tom,

So I did a quick test now before retiring after all. And I am pulling my hair out now. If I restart lighttpd on port 8000 (the last step to apparent convergence) it works. I access both development and webserve sites on:

    http://127.0.0.1:8000/

with server local browsers (Chrome on the dev server and Lynx on the webserver for lack of a desktop). And they both now have links to:

    http://127.0.0.1:8000/list/Player

and the link serves properly on both sites!

Aaargh. Simply moving the web server from port 80 to port 8000 and the problem went away. I could tear my hair out here. Again, a small step closer to pinning a cause, but not much close to understanding the reason. The site wants to run on 80 not 8000.

So I guess more empirical tests on the morrow or weekend will include moving runserver onto port 80 (after checking my dev box doesn't have something listing there already ;-) to see if it develops the problem) and a close look at the code to see if I can find any port dependencies?

I call Twilight Zone of this so far.

Remember, this:

    http://127.0.0.1/Leaderboards/list/Player

also works consistently on port 80, that is the menu presents that URL from

    <a href="{% url 'list' 'Player' %}">Players</a>

and it serves that URL albeit it serves all URLS

    http://127.0.0.1/whatever/list/Player

identically.

And on port 8000 that menu presents the URL:

    http://127.0.0.1:8000/list/Player

and serves it.

The only difference is one config line in lighttpd.conf:

    server.port                 = 80

vs

    server.port                 = 8000

with a a lighttpd restart ("sudo /etc/init.d/lighttpd restart") between. Some code paths differ based on port it seems,

I'll go find a wall to bang my head on now ;-).

Regards,

Bernd.


'Tom Evans' via Django users wrote:

[Melvyn Sopacua]

You may have a reverse proxy on port 80 that incorrectly rewrites
/(.*) to / both ways.

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/862e89bd-19ab-1a20-9fcc-8f5152eed92d%40gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
Melvyn Sopacua

[mohammad k]

when you use the url tage in templates 

you have to used like that {% url 'polls:edit' parameters %}: polls is the app or controller and the edit is view or method

> email to django-users+unsubscribe@googlegroups.com.

> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/862e89bd-19ab-1a20-9fcc-8f5152eed92d%40gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
Melvyn Sopacua

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2Bgw1GUnbnTcqA4TjQTHyOEuy_bZKSDTSsLWfhs8vymPFYneBA%40mail.gmail.com.

[Melvyn Sopacua]

On Fri, Sep 1, 2017 at 5:44 PM, mohammad k <k252...@gmail.com> wrote:
> when you use the url tage in templates
> you have to used like that {% url 'polls:edit' parameters %}: polls is the
> app or controller and the edit is view or method

No you don't *have* to. And polls isn't the app or controller. Please review:
https://docs.djangoproject.com/en/1.11/topics/http/urls/#url-namespaces

Also, there's a reason you won't find many reusable apps with URL
namespace support: it's one of the more counter-intuitive features of
Django and solves only very few cases where you think it might provide
a solution.
--
Melvyn Sopacua

[James Schneider]

#stillcurious

I'd also be interested to see if request.path and request.path_info are the same.

https://docs.djangoproject.com/en/1.11/ref/request-response/#django.http.HttpRequest.path

-James

[Bernd Wechner]

James, you da man! Fixed it at last.

The problem in the end was indeed lighttpd and is explained here:

    https://searchcode.com/codesearch/view/12783344/

Essentially lighttpd splits REQUEST_URI into SCRIPT_NAME and PATH_INFO and passes those three along with a host of other values in the environment to uwsgi, which passes them on to Django.

Now I'm not sure of what was at play all the time here, because I couldn't reproduce some of the behaviours I had last week (alas). But it was essentially gobbling the first part of the URI after the domain name and hence Django was seeing:

SCRIPT_NAME = '/list'
PATH_INFO =  '/Player'

and what Django wants to see, what the dev server sees is:

SCRIPT_NAME = ''
PATH_INFO =  '/list/Player'

Now lighttpd also know this as it happens:

    https://redmine.lighttpd.net/projects/lighttpd2/wiki/Howto_WSGI

But that is in lighttpd2 and I'm using 1.45 (2 is not available yet it seems) and even then the fix there seems a tad kludgy to say the least.

So the fix I have for now is I write my own wsgi handler:

from django.core.handlers.wsgi import WSGIHandler
import os, django

class MyWSGIHandler(WSGIHandler):
    def __call__(self, environ, start_response):
        environ['PATH_INFO'] = environ['SCRIPT_NAME'] + environ['PATH_INFO']
        environ['SCRIPT_NAME'] = ''       
        response = super().__call__(environ, start_response)
        return response

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "CoGs.settings")
django.setup()
application = MyWSGIHandler()

And it (mostly) works. Mostly as the admin site doesn't work. But I'm working on it, and have some insights (thanks for that) on where to look, namely the environment that is flowing from web server to wsgi server to django.

What I cannot for love or money reproduce or understand is behavioural differences based on the port or domain name, which I had a week and bit ago, and took good notes on and recorded well. But alas, can't see that now, I get consistent behaviour regardless of the URL or port I'm using.

Nor can I see Django inserting the app_name in the URLs at all, which it most definitely was doing. Grrrr.  So while that bugs me at some level I can't justify singing heaps of effort or worry into trying to reproduce it and understand it, and alas will leave it in the twilight zone.

What I clearly was seeing is the first path element gobbled up as the app_name was. I have a gut feel there's a django setting somewhere I have changed that explains that, i.e. a django setting that sees django put the app_name into the URLs.

I'll move onto an inspection of the admin site next, this coming weekend.

Apologies for the slow response, I only get time on weekends and some evenings to look at this and the weekend passed was Fathers Day here and so family time absorbed my free time ;-).

Regards,

Bernd.

James Schneider wrote:

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CA%2Be%2BciXMNwCimngxQ9Ouq6f9gy79zLCeC_UJaCrtnRwZ311q0g%40mail.gmail.com.