Make template caching a feature of the Django template engine

[Jaap Roes]

It’s considered a good practice to enable template caching in Django. It saves on the overhead of loading and parsing templates and turning them into template objects, which can give a nice performance boost when repeatedly rendering the same templates.

Currently it’s possible to enable template caching by using the cached loader. To do so you’ll need to add the loaders option to the engine settings and set it to a nesting of lists/tuples (I’ll wager no one can get this correct on the first try without first consulting the docs). This makes enabling template caching needlessly hard, and the discoverability of this feature lower than necessary. To this day I run into developers who have worked with Django for years but are unaware of the cached loader.

One of the downsides of enabling the cached loader is that it will cache templates until the server is reloaded. This makes it ill suited for development. I have, on many occasions, seen developers (including myself) forget to turn off template caching in their local settings, resulting in lost time trying to figure out why their changes are not showing.

I think Django can, and should do better. The bare minimum is to make template caching a simple toggle option that can be passed to the engine (https://code.djangoproject.com/ticket/25788 / https://github.com/jaap3/django/commit/21e9d3aa0eb0a9d2728f8a35318d49d528f3a60f (an admittedly naive patch)). This way cache_templates could simply mirror DEBUG and (new) projects can have a sane template caching configuration by default.

It would also be useful to have caching enabled in development so the template loading behaviour is (mostly) the same in development as it is in production.

One reason for this is that custom template tags need to be thread safe (https://docs.djangoproject.com/en/1.8/howto/custom-template-tags/#thread-safety-considerations). Because cached templates always reuse the nodes it’s easier to notice when a template tag misbehaves.

Another reason is that the performance of template rendering will be roughly the same development as in production, which can make a great difference if you’re working with complex templates and/or custom tags that perform expensive operations in their __init__.

Off course, having caching on in development is impractical if templates stay cached forever. So some sort of template invalidation and reloading system needs to be added. It’s actually fairly easy to add this to the cached loader https://code.djangoproject.com/ticket/25791 / https://github.com/jaap3/django/commit/20e1caa9f923d82ce60dff155304de5289242186 (an earlier/alternative implementation that moves caching to the engine can be found here: https://github.com/prestontimmons/django/commit/4683300c60033ba0db472c81244f01ff932c6fb3).

With caching and auto reloading as a core feature of Django’s engine it will be (somewhat) on par with the alternative Jinja2 engine shipped by Django. Jinja2’s Engine lets you configure an Environment (http://jinja.pocoo.org/docs/dev/api/#jinja2.Environment) using the engine's options. This includes setting a bytecode_cache and enabling auto_reload (this last option mirrors DEBUG by default in Django).

So in short, I propose making template caching a true feature of Django’s template engine. Which can be configured in a way similarl to the Jinja2 engine (and could be on by default for new projects):

TEMPLATES = [{

    'BACKEND': 'django.template.backends.django.DjangoTemplates',

    'DIRS': [os.path.join(BASE_DIR, 'templates')],

    'APP_DIRS': True

    'OPTIONS': {

        'cache_templates': True, 

        'auto_reload': DEBUG  // this could be the default set by the engine

    }

}]

[Marc Tamlyn]

So. If I understand correctly, this would deprecate the existing cached template loader, replace it with a more powerful, debug ready version (on by default?). Assuming this is what you mean, I wholeheartedly support this. There's no reason not to be performant by default.

Marc

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c2897e2b-c506-44e3-93e4-cb077bb8f4ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jaap Roes]

Yes, more or less. Although it's not strictly required to deprecate the cached template loader.

However if we can deprecate the cached template loader I'd also like to deprecate the weird way of passing arguments to template loaders https://github.com/django/django/blob/master/django/template/engine.py#L116. The issue with that is that the locmem loader also uses it.

It would also be interesting to look at the possibility to set a cache backend for the template cache, i.e. don't use process memory but memcached instead.

[Tim Graham]

I like the functionality this provides, but the API doesn't seem quite right to me -- namely passing configuration for template loaders through OPTIONS.

Since you mentioned "the weird way of passing arguments to template loaders" and the fact that the locmem loader uses it, I'd pose the question -- is it appropriate to pass the locmem templates_dict in OPTIONS too? If we go down this path for "auto_reload" and "cache_templates" in OPTIONS, where do you draw the line for what template loader configuration can go in there? Any third-party template loader cannot get arbitrary configuration options from OPTIONS without patching Django, so it doesn't seem like this API has a clear separation of concerns.

[Aymeric Augustin]

Hello,

I would suggest to enable the cached template loader by default when DEBUG = False and no ‘loaders’ options is specified.

This is technically backwards-incompatible, but:

- people who don’t usually configure it will get a free performance boost in production

- people who do will be able to simplify their configurations

Currently many projects have a variant of this awful pattern:

# settings/prod.py

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',

        'OPTIONS': {
            'loaders': [
                ('django.template.loaders.cached.Loader', [
                    'django.template.loaders.app_directories.Loader',
                ]),
            ],
        },
    },
]

# settings/dev.py

from .prod import *

TEMPLATES[0]['OPTIONS'].update({
    'loaders': [
        'django.template.loaders.app_directories.Loader',
    ],
})

My suggestion makes it possible to remove the configuration of ‘loaders’ and to rely only on DIRS and APP_DIRS instead.

Best regards,

-- 

Aymeric.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/476510b8-97c7-4a08-b7e3-a9cf3864ba17%40googlegroups.com.

[Tim Graham]

Seems okay to me, but what about the point "It would also be useful to have caching enabled in development so the template loading behaviour is (mostly) the same in development as it is in production." Would it be feasible to always enable the cached template loader and control the proposed "auto_reload" behavior using the value of DEBUG (i.e. no control over it in OPTIONS)?. Would a requirement (after a deprecation cycle) to use the cached template loader (without an opt-out) be problematic for some projects?

-- 

Aymeric.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsub...@googlegroups.com.

[Preston Timmons]

When I created my initial branch for this, it was easy enough to auto reload a file once it changed, but it was difficult to deal with template directory precedence. For instance, if `admin/base.html` was cached, but a custom `admin/base.html` was added in a loader or directory with higher precedence, the cached version was still used. My concern is that it's not clear to the user they need to restart the process to see the new template. A similar situation happens when templates are deleted.

I can see why some people would still want the ability to turn this feature on anyway, but without a solution for the above cases it seems better to me as an opt-in rather than a default.

[Aymeric Augustin]

On 18 déc. 2015, at 16:21, Tim Graham <timog...@gmail.com> wrote:

> Seems okay to me, but what about the point "It would also be useful to have caching enabled in development so the template loading behaviour is (mostly) the same in development as it is in production.” Would it be feasible to always enable the cached template loader and control the proposed "auto_reload" behavior using the value of DEBUG (i.e. no control over it in OPTIONS)?

I was proposing to make the default more useful while keeping the option to configure template loaders explicitly because it seemed to give almost the same benefits with less effort.

The proposed patch for auto-reloading doesn’t seem too bad in terms of complexity. Its main drawback is that it add a new auto-reload mechanism, distinct from django.utils.autoreload.

I would prefer to trigger the autoreloader on template file changes to make it easier to switch e.g. to watchman in the future. That said, I swore I’d never touch Django’s autoreloader again except under duress ;-)

--
Aymeric.

[Sven R. Kunze]

Hi everybody,

Tim pointed me to this discussion here and I support the overall goal. It accelerates development, especially when designing somewhat prettier Web pages. Quoting my quite naive approach from the issue tracker (https://code.djangoproject.com/ticket/26507#comment:2):

"""

I don't know if I would agree with the new title. Actually, I do not want to reload/replace the already loaded template.

Looking at the source, I would say, we could make 'Loader.cache_key' respect the mtime of the first found template.

So, they are two versions of the template in the cache but a cache does not need to be perfect after all and the increased memory footprint would not even matter in development.

"""

Do you think that's a feasible solution?

Sven