Re: Relative path support for TEMPLATE_DIRS and others in settings.py (django ticket 694)

[Florian Apolloner]

Hi,

On Saturday, December 22, 2012 10:35:59 PM UTC+1, Ben Porter wrote:

I would like to see support for relative paths.  It seems the solution is simple, but I wonder if there is some compelling reason to require absolute paths?

It would seem so but it is everything but simple: First of, for a relative path one needs a base path to join with, what is that? In your example it's the project root, Django doesn't have such thing as a project root. All that Django requires is a settings module which is importable (and not even that if you call settings.configure() yourself).

Secondly if you were to suggest the cwd (current working directory) this won't work either since many deployment solutions change that.

That said I am in agreement with the answers of the other core devs on the ticket and won't support this feature.

Best regards,
Florian

[Shai Berger]

Hi,

On Saturday 22 December 2012, Florian Apolloner wrote:
> On Saturday, December 22, 2012 10:35:59 PM UTC+1, Ben Porter wrote:
> > I would like to see support for relative paths. It seems the solution is
> > simple, but I wonder if there is some compelling reason to require
> > absolute paths?
>
> It would seem so but it is everything but simple: First of, for a relative
> path one needs a base path to join with, what is that? In your example it's
> the project root, Django doesn't have such thing as a project root.

You are right, but that might be the root of the problem (no pun intended).
Django doesn't have a concept of a project root, and (partly as a result)
requires paths to almost anything that isn't a python module to be given as
absolute path.

I think adding an optional "PROJECT_ROOT" or "PROJECT_PATH_BASE" setting,
specifying that other paths can be made relative to it, will remove a
significant amount of boilerplate in settings files, and will not have any of
the problems you name.

Personally, I have yet to see a settings file in a non-trivial Django project
that doesn't do the os.path.join(os.path.dir(__file__), ...) dance. When done
properly, you end up with a function in_proj_dir that is applied to many
settings values. It would be much cleaner, IMO, to have this code defined in
the framework (rather than settings files) and limit the use of code to a
single setting.

My 2 cents,
Shai.

[Daniel Sokolowski]

PROJECT_ROOT is what I have been using myself and seen it done by others so
to me it makes sense to introduce an official setting for this purpose.

--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to
django-develop...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.

--

Daniel Sokolowski
http://webdesign.danols.com/

[Luke Plant]

On 28/12/12 16:42, Daniel Sokolowski wrote:
> PROJECT_ROOT is what I have been using myself and seen it done by others
> so to me it makes sense to introduce an official setting for this purpose.

PROJECT_ROOT would still need to be defined inside the settings.py
module, using the os.path.dir(__file__) etc. dance, because it can't be
defined within django code (which doesn't know where your project
lives), and a utility function to do it isn't worth it (it's only one
line, and you don't want your settings file to be doing imports to
Django code).

So I agree with the original WONTFIX here.

I also think that app directories for the template loader is a better
solution, and making a 'project' app if necessary answers the objection.

However, I do see a case for putting the PROJECT_ROOT code into the
settings.py generated by creating a new template, and updating the help
text for TEMPLATE_DIRS and STATICFILES_DIRS to mention it.


Luke


--
The fashion wears out more apparel than the man.
-- William Shakespeare

Luke Plant || http://lukeplant.me.uk/

[Cal Leeming [Simplicity Media Ltd]]

Since the day I started using Django, I have always used a relative path for TEMPLATES_DIR.

import os

CURRENT_DIR = os.path.realpath(os.path.dirname(__file__))

TEMPLATE_DIRS  =  "%s/templates/" % ( CURRENT_DIR, )

Imho, the idea of having to hard code your PROJECT_ROOT is ludicrous.

I understand the arguments being made about relative paths on settings, but I am yet to come across a project where the above code has not worked.

Could we not have something like this in the settings.py, which in turn enabled the code pasted above?

TEMPLATE_PATH_RELATIVE=True

Cal

[Yo-Yo Ma]

-1 FWIW

Computing the root of your project with os.path works just fine and doesn't require another setting... which, btw, could have no sane default.

[Shai Berger]

On Saturday 29 December 2012, Cal Leeming [Simplicity Media Ltd] wrote:
> Since the day I started using Django, I have always used a relative path
> for TEMPLATES_DIR.
>
> import os
> CURRENT_DIR = os.path.realpath(os.path.dirname(__file__))
> TEMPLATE_DIRS = "%s/templates/" % ( CURRENT_DIR, )
>

The main point of the idea was to enable relative paths without the string
manipulations, that is,

import os
PROJECT_ROOT = os.path.realpath(os.path.dirname(__file__))
TEMPLATE_DIRS = ["templates/"]

It has some merit because TEMPLATE_DIRS is not the only such setting, so
adding the project root to a path is a pattern repeated several times in the
settings file.

> Imho, the idea of having to hard code your PROJECT_ROOT is ludicrous.
>

Nobody suggested any such thing (well, except, maybe, the comments in the
current default project template).

Shai.

[Luke Plant]

On 29/12/12 04:08, Cal Leeming [Simplicity Media Ltd] wrote:

> Could we not have something like this in the settings.py, which in turn
> enabled the code pasted above?
> TEMPLATE_PATH_RELATIVE=True

For consistency, we'd need STATICFILES_PATH_RELATIVE,
STATIC_ROOT_PATH_RELATIVE, MEDIA_ROOT_PATH_RELATIVE etc. which is
craziness. Having just one extra setting is a big deal.

There are use cases for all of these being absolute paths (or at least
some of them), and use cases for all of them being relative. We've
already chosen absolute paths, and you can generate absolute from
relative using os.path.realpath etc.

The only option I can is whether we put that snippet of code (e.g.
PROJECT_ROOT=os.path.realpath(os.path.dirname(__file__)) ) into the
settings file generated when starting a new project.

Luke


--
"If we could just get everyone to close their eyes and visualise
world peace for an hour, imagine how serene and quiet it would be
until the looting started" -- Anon

Luke Plant || http://lukeplant.me.uk/

[Cal Leeming [Simplicity Media Ltd]]

On Sun, Dec 30, 2012 at 11:01 PM, Luke Plant <L.Pla...@cantab.net> wrote:

On 29/12/12 04:08, Cal Leeming [Simplicity Media Ltd] wrote:

> Could we not have something like this in the settings.py, which in turn
> enabled the code pasted above?
> TEMPLATE_PATH_RELATIVE=True

For consistency, we'd need STATICFILES_PATH_RELATIVE,
STATIC_ROOT_PATH_RELATIVE, MEDIA_ROOT_PATH_RELATIVE etc. which is
craziness. Having just one extra setting is a big deal.

There are use cases for all of these being absolute paths (or at least
some of them), and use cases for all of them being relative. We've
already chosen absolute paths, and you can generate absolute from
relative using os.path.realpath etc.

The only option I can is whether we put that snippet of code (e.g.
PROJECT_ROOT=os.path.realpath(os.path.dirname(__file__)) ) into the
settings file generated when starting a new project.

 

+1

 

Luke


--
"If we could just get everyone to close their eyes and visualise
world peace for an hour, imagine how serene and quiet it would be
until the looting started" -- Anon

Luke Plant || http://lukeplant.me.uk/

[Aymeric Augustin]

A modern Django project is a collection of apps. Files are looked up under conventional paths within apps. Modules (especially the settings module) can live anywhere on $PYTHONPATH. Actually, there's not such thing as a project root.

For instance, instead of using TEMPLATE_DIRS, project-wide templates can go into an "myproject" application. You need one anyway as soon as you write a custom template tag or filter.

There are two special cases that don't fit into apps: STATIC_ROOT and MEDIA_ROOT. (Technically, there's ALLOWED_INCLUDE_ROOTS too, but it isn't commonly used.)

Static files are collected from the apps into STATIC_ROOT that is then served by the web server. To avoid accidentally leaking Python code over the web, it's a good idea to keep this directory outside of your source tree.

Media files are written by the application server into MEDIA_ROOT. It's a good idea to put them on a separate partition (DoS by filling up / isn't fun), or at least in a directory outside of your source tree, which must be read-only for the web server.

For local development, it's certainly fine to store the code in /home/myproject, compile the static files in /home/myproject/static, and uploading the media files in /home/myproject/media, and it's convenient to express that with relative paths. But it's hardly a best practice in production — at least, not until you've spent 30 seconds thinking about it.

Django leaves it up to the developer to structure the settings for different environments. This means we cannot provide a "development only" settings template.

For these reasons, I'm against codifying PROJECT_ROOT and relative paths in Django's default settings files. No amount of "+1 because I'm doing it already" or blog posts will make it a best practice in production.

And developers will keep doing it until sysadmins tell them not to.

--
Aymeric.

[Cal Leeming [Simplicity Media Ltd]]

For the record, the only time I'd suggested using relative paths is for 'TEMPLATE_DIRS' only, I do not use the other two.

Rather than saying "spend 30 seconds thinking about it", could you perhaps spend 30 seconds explaining why using relative paths for TEMPLATE_DIRS would be considered a bad thing to do?

Your post makes sense for STATIC_ROOT and MEDIA_ROOT, but it makes no sense for TEMPLATE_DIRS, imo.

Cal

--
You received this message because you are subscribed to the Google Groups "Django developers" group.

[Alex Gaynor]

I would like to *strongly* object to adding any notion of root path (and relative paths with it) to Django. This would be furthering the notion that a django project is a thing. It's not. You don't hear twisted people talking about the twisted project root, or any other python package. The notion of a django project is a bizzare historical relic, and should not be the basis for future design decisions.

Alex

--
"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero

[Cal Leeming [Simplicity Media Ltd]]

Just re-read my post, and realised it may have come across a bit loaded (was replying very quickly)

No insult was intended, and it's a genuine question

Cal

[Florian Apolloner]

Hi,

On Tuesday, January 1, 2013 7:48:59 PM UTC+1, Cal Leeming [Simplicity Media Ltd] wrote:

Rather than saying "spend 30 seconds thinking about it", could you perhaps spend 30 seconds explaining why using relative paths for TEMPLATE_DIRS would be considered a bad thing to do?

I don't think that per se it would be a bad thing to do, but then you could also just dump the project name (from startproject) into INSTALLED_APPS and have the same effect (plus you can have project templatetags and filters, yay :)) as Aymeric already tried to explain. I guess your point for relative TEMPLATE_DIRS is only that you can have PROJECT_DIR + templates which is already fullfiller by the app template-loader in a much cleaner way imo.

I hope that clears things up a bit.

Cheers,
Florian

[Cal Leeming [Simplicity Media Ltd]]

Just realised why this has been so heavily debated, and just discovered something new.

https://docs.djangoproject.com/en/dev/ref/templates/api/#loader-types
django.template.loaders.app_directories.Loader

If 'myproject.polls' is placed into INSTALLED_APPS then Django will look at '/path/to/myproject/polls/templates/' for the templates.

I didn't even know about app_directories loader, and using it sounds much cleaner than having to use relative paths.

In regards to the other two vars (media root etc), as stated before, those should really be deployment specific and not relative, this is a devops problem not a code problem.

As such, I'm now -9000 on supporting relative paths.

Thanks for explaining this Florian - makes a lot more sense now.

Cal

--

You received this message because you are subscribed to the Google Groups "Django developers" group.

To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/xuVeOmwI_8AJ.

[Cal Leeming [Simplicity Media Ltd]]

Also the stupid thing as well, we are already using 'templatetags' and 'filters'.. I just had no idea it supported 'appname/templates' as well.

*doh*

Cal

[Shai Berger]

Hi,

I have to take back my support of PROJECT_ROOT, in view of Aymeric's
arguments. However, now I think he isn't pursuing the conclusions of the
argument far enough:

On Tuesday 01 January 2013, Aymeric Augustin wrote:
> For instance, instead of using TEMPLATE_DIRS, project-wide templates can go
> into an "myproject" application.

"There should be one-- and preferably only one --obvious way to do it". If a
project app is the way to do it, TEMPLATE_DIRS should be deprecated.
Conversely, at the moment, a project app is the non-obvious way to do it (and
it carries with it further non-obviousness, like the importance of the order
of INSTALLED_APPS).

Also, settings with essentially the same behavior:
FIXTURE_DIRS
LOCALE_PATHS
STATICFILES_DIRS (mentioned in contrib.staticfiles docs, but not main settings
doc)

> There are two special cases that don't fit into apps: STATIC_ROOT and
> MEDIA_ROOT. (Technically, there's ALLOWED_INCLUDE_ROOTS too, but it isn't
> commonly used.)
>

Also:
In CACHES: LOCATION when BACKEND is FileBasedCache
In DATABASES: NAME when ENGINE is sqlite3 (relative path accepted here;
granted, sqlite3 is not recommended for production)
FILE_UPLOAD_TEMP_DIR
SESSION_FILE_PATH (for file-based sessions)

Third-party apps may add further such paths, where working data gets stored.

[Good arguments, about best practices for production and how paths relative to
the sources' root are only acceptable for development, snipped]

> For these reasons, I'm against codifying PROJECT_ROOT and relative paths in
> Django's default settings files.

But what if we called it "WORK_FILES_ROOT" (or something similar), and made it
clear that it's expected to lie outside the sources?

> No amount of "+1 because I'm doing it already" or blog posts will make it a
> best practice in production.

One point to take from those posts and reported practices, is that your
current position does not promote the cause of keeping working-data (and work-
copies of static files) out of the source tree. instead, it promotes
boilerplate coding in settings files, and a dangerous mixing of non-python
source files with work-files.

I think we can (and should) do better -- deprecate the global *_DIRS settings,
include a project app in the default settings (at least as a comment), clarify
the importance of order in INSTALLED_APPS, and make putting the work-files out
of the sources obvious.

My 2 cents,
Shai.

[Victor Hooi]

Hi,

I may have missed it, but has been a fundamental shift in how Django looks at projects versus applications, and how they should be laid out?

I get the impression from Alex Gaynor's comments above that the concept of "projects" is on it's way out?

I know there was a change in project layout with Django 1.4, but I thought things were still more or less the same as before - and projects were still in.

James Bennett wrote a blog post way back in 2006 about projects versus applications:

http://www.b-list.org/weblog/2006/sep/10/django-tips-laying-out-application/

I'm wondering if anybody from core is interested in updating that to reflect current best practices, and the future direction of Django?

Cheers,

Victor

[Daniel Sokolowski]

+1 on the idea of just adding PROJECT_ROOT official setting and using that
as needed - this is what I do in my projects already.

-----Original Message-----
From: Luke Plant
Sent: Sunday, December 30, 2012 6:01 PM
To: django-d...@googlegroups.com
Subject: Re: Relative path support for TEMPLATE_DIRS and others in
settings.py (django ticket 694)

--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to
django-develop...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.

Daniel Sokolowski
http://klinsight.com/

[Waylan Limberg]

On Wed, Jan 2, 2013 at 1:36 AM, Victor Hooi <victo...@gmail.com> wrote:
> Hi,
>
> I may have missed it, but has been a fundamental shift in how Django looks
> at projects versus applications, and how they should be laid out?
>
> I get the impression from Alex Gaynor's comments above that the concept of
> "projects" is on it's way out?
>
> I know there was a change in project layout with Django 1.4, but I thought
> things were still more or less the same as before - and projects were still
> in.
>
> James Bennett wrote a blog post way back in 2006 about projects versus
> applications:
>
> http://www.b-list.org/weblog/2006/sep/10/django-tips-laying-out-application/
>
> I'm wondering if anybody from core is interested in updating that to reflect
> current best practices, and the future direction of Django?
>

James updated his position on that almost exactly one year and one month later:

http://www.b-list.org/weblog/2007/nov/09/projects/

My recollection is that the basic premise discussed in that post has
been the general view of all core developers since that time (if not
before). Search the archives of this list and you'll find various
discussions about this - mostly proposals to update the tutorials to
not use the "project" concept. Unfortunately, no one has stepped up to
do the work.

--
----
\X/ /-\ `/ |_ /-\ |\|
Waylan Limberg

[ted]

A modern Django project is a collection of apps. Files are looked up under conventional paths within apps. Modules (especially the settings module) can live anywhere on $PYTHONPATH. Actually, there's not such thing as a project root.

For instance, instead of using TEMPLATE_DIRS, project-wide templates can go into an "myproject" application. You need one anyway as soon as you write a custom template tag or filter.

There are two special cases that don't fit into apps: STATIC_ROOT and MEDIA_ROOT. (Technically, there's ALLOWED_INCLUDE_ROOTS too, but it isn't commonly used.)

You can't just say "there is a solution I use, that isn't codified or widely used but solves the problem" and then waive your hand and have the problem go away.  TEMPLATE_DIRS, FIXTURES_DIRS, and STATIC_FILES_DIRS are equally special cases unless and until a "project app" is the codified way to do it.  

Best practice or not, PROJECT_PATH/DIR/ROOT/ETC is common practice to avoid boilerplate.  Relative paths for TEMPLATE_DIRS (& others) via Os.Path.Join is a straightforward solution that provides a clean conceptual runway from development to production.  Sure the project as a directory analogy isn't a perfect one, but it is a useful first order approximation.

For local development, it's certainly fine to store the code in /home/myproject, compile the static files in /home/myproject/static, and uploading the media files in /home/myproject/media, and it's convenient to express that with relative paths. But it's hardly a best practice in production — at least, not until you've spent 30 seconds thinking about it.

Django leaves it up to the developer to structure the settings for different environments. This means we cannot provide a "development only" settings template.

 

Django doesn't provide an easy way to distinguish between Dev and Produciton environments (which I think is a separate discussion), Agreed.    

However, a default that works for development with the information that "you shouldn't do this in production, here's a link to how you should" is far better than nothing.  Especially when the nothing is going to lead to people searching for a solution to usually find the default that works for development without the caveat.  The current setup gets people to the same "ok for dev" state with more work and less knowledge.

I could get behind a proposal to have a "project app" but I'd want to flush out the implications -- right now I'm still +1 on PROJECT_PATH and +0 on "project app".

T

FWIW, this is a working draft of a default settings file I like that addresses most of these issues:  https://github.com/tedtieken/django-project-skel/blob/master/project_name/settings.py  (would appreciate constructive criticism offline)

 

[Florian Apolloner]

Hi,

On Wednesday, January 2, 2013 8:19:36 PM UTC+1, ted wrote:

FWIW, this is a working draft of a default settings file I like that addresses most of these issues:  https://github.com/tedtieken/django-project-skel/blob/master/project_name/settings.py  (would appreciate constructive criticism offline)

 

Luckily you can already use that due to --template support for startproject, but here a few comments (Aside from still being violently -1 on it):

 * local_settings is imo a bad pattern as they can't easily override anything without copying it completely into the local_settings (think of all the settings which are dicts like DATABASES and CACHES)
 * Where does LOCAL_SETTINGS come from, are you supposed to set that in your settings file to get locale_settings imported? Why, if local_settings are there I want them imported, that is the point of local_settings.
 * All the env configuration stuff is just scary and way to specific for a useful default settings file
 * You have way to much stuff in INSTALLED_APPS, most of it is completely out of scope for a default settings file.
 * that applib sys.path magic just makes me cry.

That all said, it's nice if that settings file fits you, but I surely couldn't do anything with it. This is probably how I would do it (Talking about a big monolithic project):
project/default_settings.py
test_settings.py
dev_settings.py
production_settings.py

then each of the settings files would do: "from project.default_settings import *" and then customize stuff as needed (eg: INSTALLED_APPS += ('debug_toolbar',)). This way all the settings files are clearly separated and stay readable. Might not be best practices but that's how it works quite well for me.

Cheers,
Florian

[ted]

To be clear, I'm not proposing the settings.py file from my github as a django default.  It is what works for me, provided for context.

Florian, the method you use for environment specific settings is one of the two most common I saw in my survey:  local_settings.py being the most common.  Again, best practice vs common practice I don't know.  

But again, I'm not proposing my settings.py file as the default, just provided for context.

T

[Florian Apolloner]

Hi Ted,

On Wednesday, January 2, 2013 10:26:10 PM UTC+1, ted wrote:

Florian, the method you use for environment specific settings is one of the two most common I saw in my survey:  local_settings.py being the most common.  Again, best practice vs common practice I don't know. 

In my surveys it turned out to be this: people often use local_settings but once I showed them the benefits of "my" approach they quickly converted.

But again, I'm not proposing my settings.py file as the default, just provided for context.

Oh, I misunderstood you there. Sorry!

Regards,
Florian

 

[Luke Plant]

On 01/01/13 18:28, Aymeric Augustin wrote:

> There are two special cases that don't fit into apps: STATIC_ROOT and
> MEDIA_ROOT. (Technically, there's ALLOWED_INCLUDE_ROOTS too, but it
> isn't commonly used.)
>
> Static files are collected from the apps into STATIC_ROOT that is
> then served by the web server. To avoid accidentally leaking Python
> code over the web, it's a good idea to keep this directory outside of
> your source tree.
>
> Media files are written by the application server into MEDIA_ROOT.
> It's a good idea to put them on a separate partition (DoS by filling
> up / isn't fun), or at least in a directory outside of your source
> tree, which must be read-only for the web server.
>
> For local development, it's certainly fine to store the code in
> /home/myproject, compile the static files in /home/myproject/static,
> and uploading the media files in /home/myproject/media, and it's
> convenient to express that with relative paths. But it's hardly a
> best practice in production — at least, not until you've spent 30
> seconds thinking about it.

You are assuming here that use of relative paths and PROJECT_ROOT for
media implies that you are putting your MEDIA_ROOT/STATIC_ROOT *inside*
PROJECT_ROOT. But you don't have to.

I often have the situation where I have multiple copies of a project on
a machine - often 'staging' and 'production' on the same machine (e.g.
shared hosting), and also for other reasons. I have a layout something
like:


/home
/myuser
/apps
/foo_staging
/src
/static
/media
/foo_production
/src
/media
/static

/src is PROJECT_ROOT, and STATIC_ROOT and MEDIA_ROOT will be calculated
relative to that. This system makes projects completely relocatable, and
is a good use case for PROJECT_ROOT IMO.

However, this isn't something that you could put into a default
settings.py, because it assumes things about directory structure outside
the project sources.

Common opinion amongst core devs seems to be against PROJECT_ROOT. If we
are consistent about that, we ought to be thinking about deprecating
TEMPLATE_DIRS, STATICFILES_DIRS,
django.template.loaders.filesystem.Loader,
django.contrib.staticfiles.finders.FileSystemFinder
etc., as mentioned by Shai.


Luke

--
A mosquito cried out in pain:
"A chemist has poisoned my brain!"
The cause of his sorrow
was para-dichloro-
diphenyltrichloroethane

Luke Plant || http://lukeplant.me.uk/

[Luciano Pacheco]

On Thu, Jan 3, 2013 at 8:17 AM, Florian Apolloner <f.apo...@gmail.com> wrote:

 * local_settings is imo a bad pattern as they can't easily override anything without copying it completely into the local_settings (think of all the settings which are dicts like DATABASES and CACHES)

I use this way to allow variable changes:

    try:

        execfile('local_settings.py', globals(), locals())

    except IOError:

        pass

Mostly because I want in development: new apps, middlewares, so these are possible:

INSTALLED_APPS += ('debug_toolbar', 'django_extensions', 'devserver')

MIDDLEWARE_CLASSES += ('debug_toolbar.middleware.DebugToolbarMiddleware',)

On the main topic, I'm in favor of PROJECT_ROOT settings, it's a common practice, and isn't that bad, once in production it can be a mount point or overwritten on local_settings.py

[],

--
Luciano Pacheco
blog.lucmult.com.br