Multiple template engines for Django - week 1

[Aymeric Augustin]

Hello,

I just posted the first update on this project:
https://myks.org/en/multiple-template-engines-for-django/#2014-10-12

My work in progress on the DEP is available here:
https://myks.org/en/multiple-template-engines-for-django/dep/

Feedback is welcome, especially on sections I’ve described as “done” in the update.

Thanks,

--
Aymeric.

[Fred Stluka]

Excellent!  Thanks for the update.  I donated to the IndieGoGo
campaign.

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---

[Anssi Kääriäinen]

On Sun, 2014-10-12 at 20:31 +0200, Aymeric Augustin wrote:
> Feedback is welcome, especially on sections I've described as "done" in the update.

The FAQ section says that template context processors isn't going to be
supported for context processors.

The context processors return a dictionary to be added to the context,
so there is nothing template language specific in this definition.

What is the rationale of dropping context processors out of the
proposal? It seems supplying different template engines with different
data isn't going to work well for any project in which one wants to
replace DTL templates.

- Anssi

[Aymeric Augustin]

Hi Anssi,

On 17 oct. 2014, at 14:49, Anssi Kääriäinen <anssi.ka...@thl.fi> wrote:

> The FAQ section says that template context processors isn't going to be
> supported for context processors.

Indeed, at this time, my theoretical position is not to enforce such an API
and leave it up to template engines.

But I may settle for a pragmatic position if this results in every template
engine copy-pasting 50 lines of code to provide that feature.

As shown by the `render` shortcut, APIs in this area must be pragmatic,
even if that requires some compromise with the general design.

> The context processors return a dictionary to be added to the context,
> so there is nothing template language specific in this definition.

It’s implemented in RequestContext which used to be called DjangoContext
until magic-removal (2006) and lives in django.template.context. That’s why
I considered it part of the Django template language itself.

> What is the rationale of dropping context processors out of the
> proposal? It seems supplying different template engines with different
> data isn't going to work well for any project in which one wants to
> replace DTL templates.

Yes, providing all templates with a common set of request-specific values
is a useful feature. It was on my todo list :-)

--
Aymeric.

[Aymeric Augustin]

Hello,

Here’s the second update:
https://myks.org/en/multiple-template-engines-for-django/#2014-10-19

Best,

--
Aymeric.

[Aymeric Augustin]

Hello,

I just published the third update:
https://myks.org/en/multiple-template-engines-for-django/#2014-10-26

--
Aymeric.

[Aymeric Augustin]

Hello,

I’m happy to annonce that the DEP is ready for public review!

Direct link, HTML:
https://myks.org/en/multiple-template-engines-for-django/dep/

Direct link, ReStructured Text:
https://raw.githubusercontent.com/aaugustin/mtefd/master/multiple-template-engines.rst

(I’m not reproducing it here because I don’t think email is a very good medium for communicating 50kB of markup.)

I’ve written a bit more about what “ready” means:
https://myks.org/en/multiple-template-engines-for-django/#2014-11-01

Of course, your regularly scheduled update will also be available (in half an hour):
https://myks.org/en/multiple-template-engines-for-django/#2014-11-02

I’m looking forward to your feedback!

--
Aymeric.

[Michael Manfre]

Overall the DEP looks really good.

It's currently assumed that BaseEngine.select_template() will scan the list in order and return the first one it can load, but it might make sense to explicitly state that in the DEP.

To avoid having 3rd party template engines suffering some of the same disparity that 3rd party database backends faced, what are your thoughts on having the jinga2 engine maintained outside of core? This would leave only the string template reference implementation and the DTL in the core.

Regards,

Michael Manfre

>>>
>>> --
>>> Aymeric.
>>>
>>
>

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/AF442429-4EFF-4C78-804F-F47ADF453245%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

[Aymeric Augustin]

> On 2 nov. 2014, at 00:53, Michael Manfre <mma...@gmail.com> wrote:
>
> Overall the DEP looks really good.

Thanks!

> It's currently assumed that BaseEngine.select_template() will scan the list in order and return the first one it can load, but it might make sense to explicitly state that in the DEP.

I’ve clarified that in the docstring.

> To avoid having 3rd party template engines suffering some of the same disparity that 3rd party database backends faced, what are your thoughts on having the jinga2 engine maintained outside of core? This would leave only the string template reference implementation and the DTL in the core.

String template don’t have a concept of “loading” a template; they only support “rendering”. I will have to implement a custom loading infrastructure. That doesn’t make it a very good reference implementation. I even considered scraping it for this reason. So I think we need at least another one in core.

I was planning to build and maintain a Mako backend externally so as to check how things looks like from the other side of the fence.

Hopefully the well-defined a public API will make the lives of maintainers of third-party template engines easier. Also templates are simpler than databases :-)

--
Aymeric.

[Collin Anderson]

Hi Aymeric,

Thanks for all of your work on this. Am I too late to discuss the settings?

I don't see much advantage to the OPTIONS dict. It is consistent with DATABASES, and it separates the engine specific settings from the common settings. However, it doesn't seem like that helpful of a distinction to the user, especially considering there's only 3(?) non-OPTIONS settings. It seems like it only opens up the door to misconfiguration. Could we just pop-off the 3 common settings when configuring the template engine?

Thanks,
Collin

[Carl Meyer]

Hi Aymeric,

Great work!

I'm afraid that my review comments are lengthy. Feel free to suggest
that I provide them in some other format (issues and/or PRs on
https://github.com/aaugustin/mtefd ?) if that would be more useful.

* In the rationale section, it may be worth mentioning the specific case
of template-driven form widgets, since this is a pathologically bad case
for DTL performance (e.g. when rendering select-boxes with hundreds of
choices) and (at least as far as I'm concerned) a strong motivator for a
faster built-in templating option.

* In the "selecting an engine" section, I think option 1, if the only
API, has an additional downside beyond just boilerplate: it places
control over engine selection at the rendering site, which may be
outside the control of the project integrator. I still think it's good
to provide option 1 as an option for tight low-level control in atypical
cases.

* I don't agree with the characterization of option 1 as "ugly" in the
description of option 2: it's not ugly, it's a perfectly clean and
sensible API. It's just low-level, less convenient, and doesn't offer
override hooks for project integrators. (I would say option 2 is
actually uglier than option 1, because it is conceptually incoherent
given that template engines are responsible for their own template
finding, loading, and parsing.)

* The description for option 4 implies that ordering of template engines
is only useful if their directories overlap. This is not the case. I
would see ordering as perhaps most useful in cases where a project wants
to selectively (or wholly) override some templates provided by a
third-party app with replacements using a different engine. In this
case, there would be no overlapping directories in the configuration,
just the same template name provided in two different engines' directories.

* In the current DTL configuration, the relative priority of
`TEMPLATE_DIRS` vs app directories can be controlled via ordering of the
`TEMPLATE_LOADERS` setting. In your proposed configuration scheme, how
can one control the relative priority of `DIRS` vs `APP_DIRS`? This can
be quite important for overriding templates provided by third-party
apps. If that's the primary/only use case, perhaps it's sufficient to
just have `DIRS` always take priority, since that's the more
explicitly-configured option? (I see that this is what your sample
`BaseEngine` implementation does - if that's your proposal, it should be
more explicit in the DEP.)

* Similarly, since it's left up to the user to configure other loading
mechanisms that an engine might offer (presumably via OPTIONS), that
raises the question of how a user could control the priority of those
alternative loading mechanisms vs `DIRS` and `APP_DIRS`. Perhaps the
best choice here is to punt that issue to the individual template engine
and its OPTIONS.

* I assume it is permitted to omit the `DIRS` and `APP_DIRS`
configuration keys (with default values of `[]` and `False`,
respectively) if a template engine is being configured to load templates
only from a non-filesystem source?

* Along a similar vein, I think it should be permissible (though not
encouraged) for a template engine to refuse to define an app-dirs
subdirectory name. In this case it would simply be an error to attempt
to configure that template engine with `APP_DIRS: True`.

* Is there any meaning to the top-level keys in the `TEMPLATES` setting?
None is mentioned; the only one I can think of would be as a string that
one could use in the option-1 API to explicitly select a specific
engine. I think the ability to configure template engine priority will
be important, and a dictionary is problematic for that. Importing
OrderedDict is ugly boilerplate. So I wonder if `TEMPLATES` should be a
list of dictionaries rather than a dictionary of dictionaries; perhaps
with a `NAME` key if we need a string identifier for each configured engine?

* Related to the above, I am glad to see that the proposed configuration
scheme would allow configuring more than one instance of the same
template engine. I don't have any use case for this off the top of my
head, but my instinct says it's a useful capability to maintain. (This
would imply that we shouldn't re-use the BACKEND string itself as an
identifier for a particular configured engine.)

* Re CSRF compatibility: by what convention does a template engine know
if a `request` has been provided? (This is clarified by the template
object interface code below, but is left unclear at the first mention of
CSRF compatibility).

* I'm not convinced that rejecting the "switch to Babel" option for
makemessages actually reduces the scope / difficulty / disruptiveness of
this DEP. I think improving makemessages to handle multiple template
engines will likely involve an undesirable level of reimplementing
things Babel already does well (whereas since Babel already has support
for DTL syntax, makemessages' hacky handling of DTL could be removed
with a switch to Babel). There is already precedent in Django for
required dependencies for subsystems which are not enabled by default
and do not block the initial getting-started flow. I think a more
flexible makemessages will likely be best (and simplest) implemented as
a wrapper to Babel, perhaps settings some configuration defaults
according to what is known about the Django project structure.

* Re Django backend: I think it should be an error to attempt to
configure the DTL backend with `APP_DIRS: True` and `OPTIONS['LOADERS']`
set. Allowing self-contradictory configurations to pass silently,
ignoring one aspect of the user's expressed desires, is bad.

* It's not mentioned explicitly, but I presume the DTL backend's
`render` method will take an ordinary dict (not a `Context` instance) as
its `context` argument, and will automatically use `RequestContext` (and
thus context processors) if the `request` argument is provided? This
seems like the best approach to me (though for backwards-compatibility
it will also need to accept a `Context` instance, and even perhaps pull
the request out of a `RequestContext` instance if provided, at least for
a deprecation period).

* That gets into the question of how the public API and implementation
of `django.shortcuts.render` and `render_to_response` will change. This
isn't fully outlined in the DEP; perhaps it should be, since for most
people it's the primary entry point for templates?

* I'm confused about how you plan to organize the Python namespaces for
Django's template support. I see use of both `django.template` and
`django.templates` in various code examples, but if there is a
consistency to the usage I'm missing it (and I don't think
differentiating modules by a single letter would be a good plan,
anyway). Ideally it seems to me that the DTL (currently in
`django.template` and must probably stay there for
backwards-compatibility) and the new engine-configuration system would
reside in totally separate namespaces; having the latter reside in a
submodule of the former seems logically backward. The issue is finding a
good name for the latter, since `django.template` would be the obvious
choice. Perhaps `django.templating`? Or maybe your apparent choice of
`django.template.backends` is the best option in practice, even if it
implies a relationship that is inverted from the reality
(`django.template` is one of the backends supported by
`django.template.backends`; `django.template.backends` is not a feature
of `django.template`).

* Re jinja2 backend's 'env' option: can we remove the option to have
that point directly to an instance, and require that it point to a
callable accepting the other options and returning an instance? I don't
see a benefit to this option that outweighs the additional complexity it
introduces. In the worst case, if someone has a module-level instance
already, it's a trivial one-liner (or two-liner if you don't like
lambda) to create a function returning that instance. I don't agree with
the assertion that `project_name.jinja2.env` will be the most convenient
solution in general, because it means you have to manually handle the
default values for `autoescape`, `loader`, `auto_reload`, and
`undefined` that Django would otherwise provide. This is demonstrated by
your example jinja2.py module, most of which is boilerplate which could
be eliminated by instead defining a function that takes the options from
settings, with defaults already handled.

* You mention `SimpleTemplateResponse` and `TemplateResponse` in
Appendix A and note that they aren't really features of DTL (which I
agree with), but you don't provide anywhere a migration plan for them. I
would like to see them usable with any template engine, which shouldn't
be too hard. It does raise the question of whether they should be moved
to a different location.

* Re FAQ "Is it possible to use Django template filters or tags with
other engines?" - I'm not sure I agree with the strong assertion that
this "certainly will" be implemented as a third-party module. As you
mention in the previous FAQ answer, in general it will be much less work
and a better implementation for providers of custom DTL filters/tags to
provide the core functionality via simple Python functions (which can
then be trivially added to the context for most non-DTL template
languages, including Jinja2), and the DTL integration as thin wrappers
around the core function.

Whew! Thanks again for all your work on this project, and for reading
through all my comments :-)

Carl

[Aymeric Augustin]

Hi Collin,

It’s exactly the right time to discuss APIs :-)

After pondering your proposal, I'm still +0 on consistency with DATABASES and CACHES, but I'll make that change if other people agree with you. Does anyone else have an opinion on this?

Thanks,

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/adf99a14-e993-490f-b713-7ca68fd201f8%40googlegroups.com.

[Carl Meyer]

Hi Collin and Aymeric,

On 11/03/2014 03:26 PM, Aymeric Augustin wrote:
> Hi Collin,
>
> It’s exactly the right time to discuss APIs :-)
>
> After pondering your proposal, I'm still +0 on consistency with
> DATABASES and CACHES, but I'll make that change if other people agree
> with you. Does anyone else have an opinion on this?
>
> Thanks,
>
> --
> Aymeric.
>
>
>> On 3 nov. 2014, at 01:50, Collin Anderson <cmawe...@gmail.com

>> <mailto:cmawe...@gmail.com>> wrote:
>>
>> Hi Aymeric,
>>
>> Thanks for all of your work on this. Am I too late to discuss the
>> settings?
>>
>> I don't see much advantage to the OPTIONS dict. It is consistent with
>> DATABASES, and it separates the engine specific settings from the
>> common settings. However, it doesn't seem like that helpful of a
>> distinction to the user, especially considering there's only 3(?)
>> non-OPTIONS settings. It seems like it only opens up the door to
>> misconfiguration. Could we just pop-off the 3 common settings when
>> configuring the template engine?

I favor keeping OPTIONS. I don't think OPTIONS will be significantly
confusing to beginners (it may even provide a useful hedge between "the
basics" and "the advanced knobs").

Once you are doing anything beyond the basics, the distinction between a
setting that is handled specially by Django vs one that is just passed
as-is to the template backend is an important distinction to keep clear.

OPTIONS provides clear indication (without referencing the docs) which
settings you can expect to provide to any template backend with
equivalent effects, and which ones are engine-specific. I think losing
this clear distinction is likely to result in much more
mis-configuration frustration than OPTIONS will.

Carl

[Marc Tamlyn]

A few thoughts:

I like OPTIONS, I think it's a good idea for the reasons Carl suggested.

It would be preferable to have the backend configuration outside of `django.template`. `django.templating` seems reasonable, but even `django.template_backends` might be appropriate as it only contains backends.

Assuming we go for the "subdirectory" based convention for app directories (option 4?) it would seem appropriate to make this configurable. In particular I can imagine a possibility where you may want multiple instances of the same backend with different configuration (e.g. different jinja extensions which conflict to support multiple pluggable apps). The loading problem is the only obvious flaw I can see here. It's a weird use case though.

I agree the DEP needs to specify more clearly what happens with `render` and friends. I also think there's potential merit to Option 1 (engine=) support on these APIs even with Option 4 - there may be times you need to force the use of a particular engine.

Marc

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/5458079E.70104%40oddbird.net.

[Collin Anderson]

Hi All,

On Monday, November 3, 2014 5:54:38 PM UTC-5, Carl Meyer wrote:

I favor keeping OPTIONS. I don't think OPTIONS will be significantly
confusing to beginners (it may even provide a useful hedge between "the
basics" and "the advanced knobs").

Once you are doing anything beyond the basics, the distinction between a
setting that is handled specially by Django vs one that is just passed
as-is to the template backend is an important distinction to keep clear.

 That makes sense, though, is it just me or is CONTEXT_PROCESSORS a fairly frequently configured setting? My main use case is to enable the request context processor.

Collin

[Preston Timmons]

Hi Aymeric,

With the new template update, do you foresee django.utils.setup_test_template_loader and django.utils.restore_template_loaders still working?

Those aren't officially public api's, but they're useful nonetheless. Perhaps the way to do it in the future would be with override_settings and a dict loader?

Preston

[Aymeric Augustin]

Hi Preston,

2014-11-04 20:58 GMT+01:00 Preston Timmons <preston...@gmail.com>:

With the new template update, do you foresee django.utils.setup_test_template_loader and django.utils.restore_template_loaders still working?

This is a good question. To be honest, I had filed it under "implementation details I'll figure out when I get there" :-)

If I end up having to rewrite them, which is quite possible, expect a new API. I don't like changing and restoring state with a pair of functions. I prefer context managers / decorators for such use cases.

Since you find them useful, and even though they're private APIs, I'm making a note to document changes to these functions.

-- 

Aymeric.

[Aymeric Augustin]

Hi Carl,

Thank you very much for the thorough review!

Most of your comments are in line with my thinking and only require

clarifications in the DEP. Tomorrow I'll proof-read and push what I've

written tonight.

I'm answering on a few items below. Everything else I agree with :-)

2014-11-03 19:47 GMT+01:00 Carl Meyer <ca...@oddbird.net>:

* The description for option 4 implies that ordering of template engines
is only useful if their directories overlap. This is not the case. I
would see ordering as perhaps most useful in cases where a project wants
to selectively (or wholly) override some templates provided by a
third-party app with replacements using a different engine. In this
case, there would be no overlapping directories in the configuration,
just the same template name provided in two different engines' directories.

In version you read, I didn't find that implication; my reasoning was:

1 - If different engines look for files in the same directory, and you aren't

    careful, one may attempt to render a file intended for another and bad

    things will happen.

2 - Within different directories, it will usually be easier to avoid conflicts, if

    only to make sure the developer can tell what template in rendered. In

    hindsight, this isn't good advice.

The setup you're proposing is valid. It's the kind of idea I was describing

as a fallback scheme ("try to find an override version, fall back to the app's

default templates") in the last paragraph. I've revised it to remove idea 2

and be neutral about such setups.

* In the current DTL configuration, the relative priority of
`TEMPLATE_DIRS` vs app directories can be controlled via ordering of the
`TEMPLATE_LOADERS` setting. In your proposed configuration scheme, how
can one control the relative priority of `DIRS` vs `APP_DIRS`? This can
be quite important for overriding templates provided by third-party
apps. If that's the primary/only use case, perhaps it's sufficient to
just have `DIRS` always take priority, since that's the more
explicitly-configured option? (I see that this is what your sample
`BaseEngine` implementation does - if that's your proposal, it should be
more explicit in the DEP.)

Yes, `DIRS` always takes priority. I don't see an use case for the opposite

behavior (and there's an escape hatch described a few paragraphs below).

I've made that explicit in the DEP.

 

* Similarly, since it's left up to the user to configure other loading
mechanisms that an engine might offer (presumably via OPTIONS), that
raises the question of how a user could control the priority of those
alternative loading mechanisms vs `DIRS` and `APP_DIRS`. Perhaps the
best choice here is to punt that issue to the individual template engine
and its OPTIONS.

Yes, that's my choice. I've also made that explicit.

* Along a similar vein, I think it should be permissible (though not
encouraged) for a template engine to refuse to define an app-dirs
subdirectory name. In this case it would simply be an error to attempt
to configure that template engine with `APP_DIRS: True`.

I've chosen to make it mandatory for consistency across engines and

because every engine should support loading templates from directories.

Your suggestion would be easy to implement, but to me it looks like it's

adding something that doesn't have a use case. Is there a good reason?

 

* Is there any meaning to the top-level keys in the `TEMPLATES` setting?
None is mentioned; the only one I can think of would be as a string that
one could use in the option-1 API to explicitly select a specific
engine.

Yes, that's the use case. I've mentioned it.

 

I think the ability to configure template engine priority will
be important, and a dictionary is problematic for that. Importing
OrderedDict is ugly boilerplate. So I wonder if `TEMPLATES` should be a
list of dictionaries rather than a dictionary of dictionaries; perhaps
with a `NAME` key if we need a string identifier for each configured engine?

I'm torn between consistency with DATABASES and CACHES on one side

and the ugliness of OrderedDict on the other side... I chose the former

because I value consistency a lot and because I discouraged relying on the

order (but that's no longer true...)

I find the idea of extracting a special key called "NAME" and using it as the

identifier rather confusing.

The datastructure is really an ordered mapping of engine identifier => engine

configuration. There's no syntax for that in Python, so... `import collections`.

 

* Related to the above, I am glad to see that the proposed configuration
scheme would allow configuring more than one instance of the same
template engine. I don't have any use case for this off the top of my
head, but my instinct says it's a useful capability to maintain. (This
would imply that we shouldn't re-use the BACKEND string itself as an
identifier for a particular configured engine.)

Yes, it feels like a sane property even without a use case.

You could use it to have APP_DIRS take precedence over DIRS.

* I'm not convinced that rejecting the "switch to Babel" option for
makemessages actually reduces the scope / difficulty / disruptiveness of
this DEP. I think improving makemessages to handle multiple template
engines will likely involve an undesirable level of reimplementing
things Babel already does well (whereas since Babel already has support
for DTL syntax, makemessages' hacky handling of DTL could be removed
with a switch to Babel). There is already precedent in Django for
required dependencies for subsystems which are not enabled by default
and do not block the initial getting-started flow. I think a more
flexible makemessages will likely be best (and simplest) implemented as
a wrapper to Babel, perhaps settings some configuration defaults
according to what is known about the Django project structure.

I don't think I have enough information at this point to make a decision nor

that a decision is absolutely required right now. I've changed the DEP to

keep this option under consideration.

 

* It's not mentioned explicitly, but I presume the DTL backend's
`render` method will take an ordinary dict (not a `Context` instance) as
its `context` argument, and will automatically use `RequestContext` (and
thus context processors) if the `request` argument is provided? This
seems like the best approach to me (though for backwards-compatibility
it will also need to accept a `Context` instance, and even perhaps pull
the request out of a `RequestContext` instance if provided, at least for
a deprecation period).

Yes.

 

* That gets into the question of how the public API and implementation
of `django.shortcuts.render` and `render_to_response` will change. This
isn't fully outlined in the DEP; perhaps it should be, since for most
people it's the primary entry point for templates?

They don't change — well, almost :-)

They get a new keyword argument, `engine`.

I think I'll have to deprecate the current_app keyword argument and make it

an attribute of the request instead. That's an implementation contingency for

which I'll implement a deprecation path.

 

* I'm confused about how you plan to organize the Python namespaces for
Django's template support. I see use of both `django.template` and
`django.templates` in various code examples, but if there is a
consistency to the usage I'm missing it (and I don't think
differentiating modules by a single letter would be a good plan,
anyway).

It's a typo. Read `django.template` everywhere.

 

Ideally it seems to me that the DTL (currently in
`django.template` and must probably stay there for
backwards-compatibility) and the new engine-configuration system would
reside in totally separate namespaces; having the latter reside in a
submodule of the former seems logically backward.

Indeed. For lack of a better solution, I plan to let them live in the same

namespace. That gives:

django.template.backends.* - built-in backends

django.template.utils - utilities for supporting backends

django.template.* - implementation of the DTL 

I shall rewrite the docstring of django.template to explain that.

 

The issue is finding a
good name for the latter, since `django.template` would be the obvious
choice. Perhaps `django.templating`? Or maybe your apparent choice of
`django.template.backends` is the best option in practice, even if it
implies a relationship that is inverted from the reality
(`django.template` is one of the backends supported by
`django.template.backends`; `django.template.backends` is not a feature
of `django.template`).

To me the alternative would be to move the implementation of the DTL to

e.g. django.template.engine. I judged that it would create gratuitous code

churn and I chose not do to it but I can if you think it's useful. 

* You mention `SimpleTemplateResponse` and `TemplateResponse` in
Appendix A and note that they aren't really features of DTL (which I
agree with), but you don't provide anywhere a migration plan for them. I
would like to see them usable with any template engine, which shouldn't
be too hard. It does raise the question of whether they should be moved
to a different location.

My answer is the same as for `render` and `render_to_response`. Besides

their private method `resolve_context` won't be needed any more since

`Template.render` will accept a plain dict.

Since template responses couple request handling and template rendering,

they belong to django.http, django.template, or django.shortcuts. I don't

believe the other places are enough of an improvement to warrant moving

them.

 

* Re FAQ "Is it possible to use Django template filters or tags with
other engines?" - I'm not sure I agree with the strong assertion that
this "certainly will" be implemented as a third-party module. As you
mention in the previous FAQ answer, in general it will be much less work
and a better implementation for providers of custom DTL filters/tags to
provide the core functionality via simple Python functions (which can
then be trivially added to the context for most non-DTL template
languages, including Jinja2), and the DTL integration as thin wrappers
around the core function.

I've changed the language but I'm still convinced that someone will do it :-)

-- 

Aymeric.

[Aymeric Augustin]

2014-11-04 14:45 GMT+01:00 Collin Anderson <cmawe...@gmail.com>:

That makes sense, though, is it just me or is CONTEXT_PROCESSORS a fairly frequently configured setting? My main use case is to enable the request context processor.

In my opinion the request context processor should be enabled by default.

I suspect the only reason why it isn't is to avoid changing the default settings (django.conf.global_settings).

--

Aymeric.

[Aymeric Augustin]

Hi Marc,

2014-11-04 13:09 GMT+01:00 Marc Tamlyn <marc....@gmail.com>:

It would be preferable to have the backend configuration outside of `django.template`. `django.templating` seems reasonable, but even `django.template_backends` might be appropriate as it only contains backends.

See my answer to Carl's email. 

Assuming we go for the "subdirectory" based convention for app directories (option 4?) it would seem appropriate to make this configurable. In particular I can imagine a possibility where you may want multiple instances of the same backend with different configuration (e.g. different jinja extensions which conflict to support multiple pluggable apps). The loading problem is the only obvious flaw I can see here. It's a weird use case though.

Configuring the subdirectory doesn't save you if two conflicting apps ship their templates in identically named subdirectories.

In that situation you could configure a backend that has the subdirectory containing the templates for a conflicting app in DIRS and another backend that deals with the other conflicting app. It's messy, but so is the use case :-)

I agree the DEP needs to specify more clearly what happens with `render` and friends.

Yes, I'll add a paragraph about shortcuts and another about template responses in the implementation section.

 

I also think there's potential merit to Option 1 (engine=) support on these APIs even with Option 4 - there may be times you need to force the use of a particular engine.

Yes, option 1 is included in my proposal.

 

--

Aymeric.

[Carl Meyer]

On 11/04/2014 03:27 PM, Aymeric Augustin wrote:
>
> 2014-11-03 19:47 GMT+01:00 Carl Meyer <ca...@oddbird.net

> <mailto:ca...@oddbird.net>>:

>
> * The description for option 4 implies that ordering of template engines
> is only useful if their directories overlap. This is not the case. I
> would see ordering as perhaps most useful in cases where a project wants
> to selectively (or wholly) override some templates provided by a
> third-party app with replacements using a different engine. In this
> case, there would be no overlapping directories in the configuration,
> just the same template name provided in two different engines'
> directories.
>
>
> In version you read, I didn't find that implication; my reasoning was:
>
> 1 - If different engines look for files in the same directory, and you
> aren't
> careful, one may attempt to render a file intended for another and bad
> things will happen.
>
> 2 - Within different directories, it will usually be easier to avoid
> conflicts, if
> only to make sure the developer can tell what template in rendered. In
> hindsight, this isn't good advice.
>
> The setup you're proposing is valid. It's the kind of idea I was describing
> as a fallback scheme ("try to find an override version, fall back to the
> app's
> default templates") in the last paragraph. I've revised it to remove idea 2
> and be neutral about such setups.

Yes, you're right. On second reading, the implication I asserted isn't
actually there. I was really objecting to the phrase "the intent of this
design..." which it sounds like you will remove.

> * In the current DTL configuration, the relative priority of
> `TEMPLATE_DIRS` vs app directories can be controlled via ordering of the
> `TEMPLATE_LOADERS` setting. In your proposed configuration scheme, how
> can one control the relative priority of `DIRS` vs `APP_DIRS`? This can
> be quite important for overriding templates provided by third-party
> apps. If that's the primary/only use case, perhaps it's sufficient to
> just have `DIRS` always take priority, since that's the more
> explicitly-configured option? (I see that this is what your sample
> `BaseEngine` implementation does - if that's your proposal, it should be
> more explicit in the DEP.)
>
> Yes, `DIRS` always takes priority. I don't see an use case for the opposite
> behavior (and there's an escape hatch described a few paragraphs below).
> I've made that explicit in the DEP.

I can't think of a use case either; I think that's the right choice.

> * Along a similar vein, I think it should be permissible (though not
> encouraged) for a template engine to refuse to define an app-dirs
> subdirectory name. In this case it would simply be an error to attempt
> to configure that template engine with `APP_DIRS: True`.
>
> I've chosen to make it mandatory for consistency across engines and
> because every engine should support loading templates from directories.
>
> Your suggestion would be easy to implement, but to me it looks like it's
> adding something that doesn't have a use case. Is there a good reason?

I don't feel strongly. The "use case" (that's a strong term for it) is
this: I don't usually put templates in apps, only in a single
project-wide templates directory. I can imagine a scenario where I am
implementing a specialized project-specific template backend (for some
reason - let's hand-wave past this), and it would feel extraneous to be
forced to name an app sub-directory that I plan to never use.

Clearly, this isn't a strong case - I can easily just pick some string
and move on. My suggestion is really motivated by the general principle
that configuration which is not strictly required for a good reason
should be optional, and that I don't see the harm in allowing it. (I
actually would guess that a naive implementation would work precisely as
I outlined, and you'd have to go out of your way in the implementation
to "validate" that a template backend defines the app-dirs name even
when APP_DIRS is `False`.)

> * Is there any meaning to the top-level keys in the `TEMPLATES` setting?
> None is mentioned; the only one I can think of would be as a string that
> one could use in the option-1 API to explicitly select a specific
> engine.
>
>
> Yes, that's the use case. I've mentioned it.
>
>
> I think the ability to configure template engine priority will
> be important, and a dictionary is problematic for that. Importing
> OrderedDict is ugly boilerplate. So I wonder if `TEMPLATES` should be a
> list of dictionaries rather than a dictionary of dictionaries; perhaps
> with a `NAME` key if we need a string identifier for each configured
> engine?
>
>
> I'm torn between consistency with DATABASES and CACHES on one side
> and the ugliness of OrderedDict on the other side... I chose the former
> because I value consistency a lot and because I discouraged relying on the
> order (but that's no longer true...)
>
> I find the idea of extracting a special key called "NAME" and using it
> as the
> identifier rather confusing.
>
> The datastructure is really an ordered mapping of engine identifier =>
> engine
> configuration. There's no syntax for that in Python, so... `import
> collections`.

Yes, I see this reasoning. It's probably the best of bad alternatives :/
My main concern is that if anyone forgets to use OrderedDict, their
settings file will easily give the visual impression of a "priority"
that may or may not actually be correct.

I'm almost tempted to propose that `django.conf.BaseSettings` should
warn if `TEMPLATES` has length > 1 and is an unordered dict, but I won't
go that far. I do think the DEP should be updated to use OrderedDict in
all examples with more than one template engine configured, and the
default startproject settings file should also use OrderedDict.

> * Related to the above, I am glad to see that the proposed configuration
> scheme would allow configuring more than one instance of the same
> template engine. I don't have any use case for this off the top of my
> head, but my instinct says it's a useful capability to maintain. (This
> would imply that we shouldn't re-use the BACKEND string itself as an
> identifier for a particular configured engine.)
>
>
> Yes, it feels like a sane property even without a use case.
>
> You could use it to have APP_DIRS take precedence over DIRS.
>
> * I'm not convinced that rejecting the "switch to Babel" option for
> makemessages actually reduces the scope / difficulty / disruptiveness of
> this DEP. I think improving makemessages to handle multiple template
> engines will likely involve an undesirable level of reimplementing
> things Babel already does well (whereas since Babel already has support
> for DTL syntax, makemessages' hacky handling of DTL could be removed
> with a switch to Babel). There is already precedent in Django for
> required dependencies for subsystems which are not enabled by default
> and do not block the initial getting-started flow. I think a more
> flexible makemessages will likely be best (and simplest) implemented as
> a wrapper to Babel, perhaps settings some configuration defaults
> according to what is known about the Django project structure.
>
>
> I don't think I have enough information at this point to make a decision nor
> that a decision is absolutely required right now. I've changed the DEP to
> keep this option under consideration.

That's reasonable.

> * It's not mentioned explicitly, but I presume the DTL backend's
> `render` method will take an ordinary dict (not a `Context` instance) as
> its `context` argument, and will automatically use `RequestContext` (and
> thus context processors) if the `request` argument is provided? This
> seems like the best approach to me (though for backwards-compatibility
> it will also need to accept a `Context` instance, and even perhaps pull
> the request out of a `RequestContext` instance if provided, at least for
> a deprecation period).
>
>
> Yes.
>
>
> * That gets into the question of how the public API and implementation
> of `django.shortcuts.render` and `render_to_response` will change. This
> isn't fully outlined in the DEP; perhaps it should be, since for most
> people it's the primary entry point for templates?
>
>
> They don't change — well, almost :-)
>
> They get a new keyword argument, `engine`.
>
> I think I'll have to deprecate the current_app keyword argument and make it
> an attribute of the request instead. That's an implementation
> contingency for
> which I'll implement a deprecation path.

Right.

I think the `context_instance` and `dirs` arguments should also be
deprecated (both `render` and `render_to_response` have both, since they
pass along *args/**kwargs to `render_to_string`)?

And `render` should no longer wrap the given context dict in a
RequestContext, but just pass it along to the backend's `render` as a dict.

And both `render` and `render_to_response` can probably gain explicit
signatures, rather than *args/**kwargs?

> * I'm confused about how you plan to organize the Python namespaces for
> Django's template support. I see use of both `django.template` and
> `django.templates` in various code examples, but if there is a
> consistency to the usage I'm missing it (and I don't think
> differentiating modules by a single letter would be a good plan,
> anyway).
>
>
> It's a typo. Read `django.template` everywhere.
>
>
> Ideally it seems to me that the DTL (currently in
> `django.template` and must probably stay there for
> backwards-compatibility) and the new engine-configuration system would
> reside in totally separate namespaces; having the latter reside in a
> submodule of the former seems logically backward.
>
>
> Indeed. For lack of a better solution, I plan to let them live in the same
> namespace. That gives:
>
> django.template.backends.* - built-in backends
> django.template.utils - utilities for supporting backends
> django.template.* - implementation of the DTL
>
> I shall rewrite the docstring of django.template to explain that.

I think that's probably again the best of poor alternatives, since I
can't think of a new name for the backends system that isn't forced.

> The issue is finding a
> good name for the latter, since `django.template` would be the obvious
> choice. Perhaps `django.templating`? Or maybe your apparent choice of
> `django.template.backends` is the best option in practice, even if it
> implies a relationship that is inverted from the reality
> (`django.template` is one of the backends supported by
> `django.template.backends`; `django.template.backends` is not a feature
> of `django.template`).
>
>
> To me the alternative would be to move the implementation of the DTL to
> e.g. django.template.engine. I judged that it would create gratuitous code
> churn and I chose not do to it but I can if you think it's useful.

No, I agree that that option would be too much churn for too little value.

> * You mention `SimpleTemplateResponse` and `TemplateResponse` in
> Appendix A and note that they aren't really features of DTL (which I
> agree with), but you don't provide anywhere a migration plan for them. I
> would like to see them usable with any template engine, which shouldn't
> be too hard. It does raise the question of whether they should be moved
> to a different location.
>
>
> My answer is the same as for `render` and `render_to_response`. Besides
> their private method `resolve_context` won't be needed any more since
> `Template.render` will accept a plain dict.
>
> Since template responses couple request handling and template rendering,
> they belong to django.http, django.template, or django.shortcuts. I don't
> believe the other places are enough of an improvement to warrant moving
> them.

If the template-backends system is staying in `django.template`, and
we're accepting that `django.template` contains both the backends layer,
and the DTL backend, then it's fine for them to stay there too.

> * Re FAQ "Is it possible to use Django template filters or tags with
> other engines?" - I'm not sure I agree with the strong assertion that
> this "certainly will" be implemented as a third-party module. As you
> mention in the previous FAQ answer, in general it will be much less work
> and a better implementation for providers of custom DTL filters/tags to
> provide the core functionality via simple Python functions (which can
> then be trivially added to the context for most non-DTL template
> languages, including Jinja2), and the DTL integration as thin wrappers
> around the core function.
>
>
> I've changed the language but I'm still convinced that someone will do
> it :-)

Thanks for the response! Looking forward to seeing this in Django,

Carl

[Collin Anderson]

On Tuesday, 4 November 2014 17:30:27 UTC-5, Aymeric Augustin wrote:

In my opinion the request context processor should be enabled by default.

+1 

I suspect the only reason why it isn't is to avoid changing the default settings (django.conf.global_settings).

Right. Can we uncomment it anyway? I can't imagine that there are many people are using the "request" name for anything other than the django request.

Or, I would also be happy with a render('template.html', {'request': request, 'myvar': 3}) convention and then stop using context processors completely.

Thanks,

Collin

[Aymeric Augustin]

Hi Carl,

2014-11-05 0:02 GMT+01:00 Carl Meyer <ca...@oddbird.net>:

I can imagine a scenario where I am
implementing a specialized project-specific template backend (for some
reason - let's hand-wave past this), and it would feel extraneous to be
forced to name an app sub-directory that I plan to never use.

That makes sense. I've allowed templates engines not to provide the

basic loading features but in that case I've required them to raise an

exception if DIRS isn't empty or APP_DIRS isn't False.

 

I do think the DEP should be updated to use OrderedDict in
all examples with more than one template engine configured,

I've made this change in the only example that had two engines.

 

and the default startproject settings file should also use OrderedDict.

Well, the good news is that we don't need to put TEMPLATES

in the auto-generated settings file. The defaults just work if you're

putting templates in apps.

If we decided to add it for easier customisation, you're right, we

should include the OrderedDict, or it would be a Gun Pointed at

User's Feet, waiting for them to add a second engine.

 

I think the `context_instance` and `dirs` arguments should also be
deprecated (both `render` and `render_to_response` have both, since they
pass along *args/**kwargs to `render_to_string`)?

And `render` should no longer wrap the given context dict in a
RequestContext, but just pass it along to the backend's `render` as a dict.

And both `render` and `render_to_response` can probably gain explicit
signatures, rather than *args/**kwargs?

OK, I won't get away with hand-vawing :-)

I'll let you know when I have a specification ready for review.

--

Aymeric.

[Aymeric Augustin]

Hi Collin,

2014-11-05 3:21 GMT+01:00 Collin Anderson <collinm...@gmail.com>:

On Tuesday, 4 November 2014 17:30:27 UTC-5, Aymeric Augustin wrote: 

I suspect the only reason why it isn't is to avoid changing the default settings (django.conf.global_settings).

Right. Can we uncomment it anyway?

The problem with changing global_settings is that we don't have any way to provide a deprecation path. However, since we have the checks framework, we've started making such changes and attempting to warn users with checks. Would you like to explore this direction?

I consider this idea to be independent from my refactoring.

Or, I would also be happy with a render('template.html', {'request': request, 'myvar': 3}) convention and then stop using context processors completely.

render(request, template_name, context) is such a strong convention at this point that I prefer keeping it.

Of course, I expect many template engines to simply push the request into the context and move on :-)

--

Aymeric.

[Aymeric Augustin]

> On 5 nov. 2014, at 09:42, Aymeric Augustin <aymeric....@polytechnique.org> wrote:
>
> I'll let you know when I have a specification ready for review.

I just pushed an implementation plan for shortcuts and template responses.

Search for `render(request, template_name` in the DEP or look at the history in
https://github.com/aaugustin/mtefd/commits/master/multiple-template-engines.rst

--
Aymeric.

[Tim Graham]

Hi Aymeric,

Nice work on the DEP. I tend to agree with Carl that I like a 'NAME' key in TEMPLATES rather than requiring the use of OrderedDict, but I can also see why you don't. I think it might simply the implementation if TEMPLATES was always a list of dictionaries rather than allowing it to be a dict or an OrderedDict. But if you decide not to go that route: if Django iterates over a plain dict to select a template, it seems that could result in some weird performance issues, e.g. if a page that expects template loader 'A' and sometimes tries 'A' as the first loader, sometimes as the second, etc (assuming there is some performance penalty for trying a loader and not finding the template). If it were me, I think I'd always enforce ordering if len(TEMPLATES) > 1 as Carl suggested. If the only downside is the ugliness of OrderedDict, well I think it might save some headaches and support queries.

Tim

[Carl Meyer]

On 11/05/2014 05:16 PM, Tim Graham wrote:
> Nice work on the DEP. I tend to agree with Carl that I like a 'NAME' key
> in TEMPLATES rather than requiring the use of OrderedDict, but I can
> also see why you don't. I think it might simply the implementation if
> TEMPLATES was always a list of dictionaries rather than allowing it to
> be a dict or an OrderedDict. But if you decide not to go that route: if
> Django iterates over a plain dict to select a template, it seems that
> could result in some weird performance issues, e.g. if a page that
> expects template loader 'A' and sometimes tries 'A' as the first loader,
> sometimes as the second, etc (assuming there is some performance penalty
> for trying a loader and not finding the template). If it were me, I
> think I'd always enforce ordering if len(TEMPLATES) > 1 as Carl
> suggested. If the only downside is the ugliness of OrderedDict, well I
> think it might save some headaches and support queries.

Yes, I'm still a bit uneasy with the idea of even allowing a
multi-engine `TEMPLATES` to be an unordered dict as well. I hadn't
thought about the fact that it is relevant for performance reasons even
if you aren't doing any template overrides. It just seems like a bad
idea to allow it at all.

And like Tim, I do feel that a `NAME` key would also still be a
reasonable alternative, even though it breaks the parallel with
`DATABASES` and `CACHES` (there is not really a parallel anyway, since
those are unordered mappings, which is a different data structure).

On the other hand, I do recognize that the name is by nature a referent
(and should be unique), and thus having it be the key in a mapping makes
a lot of sense.

Seeing the `OrderedDict` syntax in the example in the latest PEP update
made me sad. So much more verbose than dictionary syntax; I think people
will really be tempted to skip it if we don't force ordered-ness, but I
think unordered will cause them even more problems.

Sure wish Python had builtin syntax for ordered mappings :/

Carl

[Collin Anderson]

Could we do a list of 2-tuples instead of OrderdDict?

[Aymeric Augustin]

> On 6 nov. 2014, at 01:31, Carl Meyer <ca...@oddbird.net> wrote:
>
> Seeing the `OrderedDict` syntax in the example in the latest PEP update
> made me sad.

I’m convinced.

I had an interesting idea for NAME. It can default to the name of the engine.
Then only people who configure more than one instance of a given engine
need to configure it. I updated the DEP accordingly. TEMPLATES becomes
quite elegant.

This “smart” idea that turns out to be stupid for an obvious reason. Let me
know if I missed something.

--
Aymeric.

[Carl Meyer]

Looks great to me, as long as the error message for non-unique `NAME` is
clear about what needs to be done. Much easier to read and write the
`TEMPLATES` setting now.

Carl

[Preston Timmons]

Aymeric,

Great work on this. I have a few more questions:

First, if multiple engines are configured, how do you see the error being displayed if a template isn't found in any of them? Currently, this originates from the template loaders. For instance, the filesystem loader raises a TemplateDoesNotExist with a list of templates it tried.

Second, if I want to get a template from within a script, do I have to manually loop through the engines? Or is there a higher-level get_template that encapsulates this?

Third, if I understand right, extending templates would only happen within the template engine that found the template. That means if multiple Django template engines were specified, the extends tag would not extend templates from other Django engines, even when specified. Is this correct?

Preston

[Aymeric Augustin]

Hi Preston,

On 7 nov. 2014, at 18:51, Preston Timmons <preston...@gmail.com> wrote:

> First, if multiple engines are configured, how do you see the error being displayed if a template isn't found in any of them? Currently, this originates from the template loaders. For instance, the filesystem loader raises a TemplateDoesNotExist with a list of templates it tried.

Django will still raise the same TemplateDoesNotExist exception. I think I’ll build an error message by concatenating exceptions raised by each engine.

> Second, if I want to get a template from within a script, do I have to manually loop through the engines? Or is there a higher-level get_template that encapsulates this?

I will preserve public APIs, including:

django.template.loader.get_template(template_name[, dirs])
django.template.loader.select_template(template_name_list[, dirs])

> Third, if I understand right, extending templates would only happen within the template engine that found the template. That means if multiple Django template engines were specified, the extends tag would not extend templates from other Django engines, even when specified. Is this correct?

Yes, that’s correct.

--
Aymeric.

[Aymeric Augustin]

Hello,

Here’s my fifth update:
https://myks.org/en/multiple-template-engines-for-django/#2014-11-09

The first step of my project, as outlined in my original proposal, is complete.

--
Aymeric.

[Aymeric Augustin]

On 4 nov. 2014, at 21:38, Aymeric Augustin <aymeric....@polytechnique.org> wrote:

2014-11-04 20:58 GMT+01:00 Preston Timmons <preston...@gmail.com>: