[ANN] django-synth, a bridge to Synth's C++ template engines

[Alvaro J. Genial]

Howdy,

I'd like to announce a beta release of django-synth, a Python package that enables the seamless use of Synth in Django projects. Synth is a template parser and renderer written in C++ that features a complete, standalone implementation of Django's template system, with experimental support for custom tags and filters.

You can install it as follows:

    pip install pip --upgrade # Ensure wheel support

    pip install django-synth

Then require it (e.g. django-synth>=0.7.0) and add it to your settings:

    TEMPLATE_LOADERS = (

        'django_synth.loaders.FilesystemLoader',

        'django_synth.loaders.AppDirectoriesLoader',

    )

(Note that installation can take a while if there are no pre-built binary wheels for the platform; also, note that the Synth bindings are designed to work predominantly with Python 2.7 and 3.4.)

Preliminary testing suggests that the Synth implementation can be 10x to 30x faster, though take those figures with a grain of salt because benchmarking is difficult to get right in the general case. If you're curious, this is the relevant timing code--feel free to help improve it.

At this stage the output may not always match byte for byte, especially when using existing custom libraries (which require a good deal of black magic to interoperate.) For example, the blocktrans tag from Django's i18n library is known to give some grief; regardless, the plan is to implement all the common custom libraries (i18n, l10n, static, staticfiles, tz, humanize and webdesign) in order to minimize use of the custom-loading mechanism and crossing the Python/C++ boundary.

For what it's worth, besides the Django template language, django-synth also lets you use server-side includes (SSI) and Perl's HTML::Templates "for free" from Django, since Synth implements those too. There's a setting that controls the default engine used, plus a few others to tweak Synth's behavior.

Anyway, is there interest in developing django-synth further?

If there is, the roadmap would include:

  - Reaching a stable, fully-compatible first release

  - Implementing all common custom libraries natively

  - Writing a full set of documentation

  - Enhancing performance: parallel parsing, speculative rendering, aggressive caching, etc.

However, first I'd like to get a sense of whether such work would be useful. As always, all feedback is appreciated.

Thank you!

Alvaro

[Russell Keith-Magee]

Hi Alvaro,

Firstly - congratulations - sounds like an interesting package. However, you may find that django-users is a better forum to announce something like that. Django-developers is the forum for discussing the development of Django itself, and while most of the people on this list also use django, django-users is a bigger audience.

Secondly - the "would I use this" question... maybe. Faster templates are obviously a good thing; I'd be interested to see exactly how much faster (and how much faster compared to a "known fast" alternative template language like Jinja2). I'm especially interested to see how it's handled custom template tags, because that has always been the sticking point for faster *native* Django templates.

Lastly - you've implemented a binding layer to a new template language. Is there anything in Django's core that you would like to see modified that would make your life easier? Using a custom template language is an area where many people claim it's impossible to do (despite evidence to the contrary), so we'd like to make it as easy as possible to do. Any suggestions/requests?

Yours,

Russ Magee %-)

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a052bfe4-4106-410d-8bb4-bbf8082f7f8a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alvaro J. Genial]

On Wednesday, August 6, 2014 7:51:09 PM UTC-4, Russell Keith-Magee wrote:

Hi Alvaro,

Firstly - congratulations - sounds like an interesting package. However, you may find that django-users is a better forum to announce something like that. Django-developers is the forum for discussing the development of Django itself, and while most of the people on this list also use django, django-users is a bigger audience.

Thank you--that's a good suggestion. I'll post an announcement there after ironing out any remaining major issues.

 

Secondly - the "would I use this" question... maybe. Faster templates are obviously a good thing; I'd be interested to see exactly how much faster (and how much faster compared to a "known fast" alternative template language like Jinja2). I'm especially interested to see how it's handled custom template tags, because that has always been the sticking point for faster *native* Django templates.

Makes sense. I have a simple benchmark script (synth/tests/timings.py) that roughly suggests ~27x faster on Ubuntu, ~20x faster on OS X and ~11x faster on Windows, given a somewhat esoteric set of machines, templates and data. (The details are at the end of this message, and all the standard benchmark caveats apply.) If there's an official benchmark suite I can run it as well and post the timings. At the moment, overall, the difference seems to range from 10x to 30x. (Python2 vs. Python3 doesn't appear to affect the outcome much, nor does 32-bit vs 64-bit.)

Anyway, with the bulk of the functionality completed, I can now focus on performance, so I expect those figures to go up, as there's a lot of low-hanging fruit.

Separately, I've been studying the differences between Django's template language and Jinja2 and I'm considering implementing the latter natively as well, especially because: there'd be a lot of shared code; the Django integration glue is already there; and it seems to allow even more opportunities for speed gains. Any thoughts?

 

Lastly - you've implemented a binding layer to a new template language. Is there anything in Django's core that you would like to see modified that would make your life easier? Using a custom template language is an area where many people claim it's impossible to do (despite evidence to the contrary), so we'd like to make it as easy as possible to do. Any suggestions/requests?

Thanks for the offer! Currently, the best thing I can think of is to actually try it on real projects and report significant issues and differences in output.

One relatively small improvement I just thought of is to allow template loaders to parse top-level templates from a file path rather than accepting the source as a string. (Perhaps this is possible already?)

Though by far, the trickiest parts of the binding layer are the various gymnastics necessary to make arbitrary custom tags work (custom filters being much simpler.) I have some ideas about how that could be improved which I'll elaborate in a follow-up post. 

Thanks again!

Alvaro

Sample Timings:

 * Ubuntu 14.04.1 SMP / Linux 3.13.0-32-generic / 6GB RAM:

~/code/synth$ python tests/timings.py 

Loaded synth; version: 1.0.3

=== Prepping ===

Directories: ('tests/templates/django/blog', 'tests/templates/django', '.')

Data: {'bar': 2, 'qux': 3, 'posts': [{'url': 'http://example.org/posts/1', 'category': 'movies', 'title': 'Rocky'}, {'url': 'http://example.org/posts/2', 'category': 'shows', 'title': 'Blackjack'}], 'user': 'Dolph Lundgren', 'subtitle': 'A blog about movies and TV shows', 'foo': 1, 'categories': [{'url': 'http://example.org/categories/movies', 'title': 'Movies'}, {'url': 'http://example.org/categories/shows', 'title': 'Shows'}]}

Files: ['base.html', 'home.html', 'index.html', 'A.tpl', 'B.tpl', 'base.tpl', 'C.tpl', 'D.tpl', 'derived.tpl', 'empty.tpl', 'layout.html', 'messages.html', 'variables.tpl', 'X.tpl']

Debug: False

=== Starting ===

Iterations: 1000

Django: 0.384s; Synth: 0.018s; +/-: 21.6x; File: 'base.html'

Django: 1.301s; Synth: 0.077s; +/-: 16.8x; File: 'home.html'

Django: 1.042s; Synth: 0.070s; +/-: 14.8x; File: 'index.html'

Django: 0.276s; Synth: 0.012s; +/-: 23.8x; File: 'A.tpl'

Django: 0.555s; Synth: 0.016s; +/-: 34.9x; File: 'B.tpl'

Django: 0.326s; Synth: 0.014s; +/-: 22.9x; File: 'base.tpl'

Django: 0.835s; Synth: 0.022s; +/-: 37.4x; File: 'C.tpl'

Django: 1.081s; Synth: 0.027s; +/-: 40.3x; File: 'D.tpl'

Django: 0.694s; Synth: 0.023s; +/-: 29.6x; File: 'derived.tpl'

Django: 0.229s; Synth: 0.008s; +/-: 29.8x; File: 'empty.tpl'

Django: 1.338s; Synth: 0.055s; +/-: 24.2x; File: 'layout.html'

Django: 0.573s; Synth: 0.021s; +/-: 27.8x; File: 'messages.html'

Django: 0.413s; Synth: 0.017s; +/-: 24.1x; File: 'variables.tpl'

Django: 0.549s; Synth: 0.016s; +/-: 34.8x; File: 'X.tpl'

Average: 27.4x

=== Finished ===

 * Mac OS X 10.9.4 Mavericks / Intel Core i7 2.7GHz / 16GB RAM:

/Projects/code/synth$ python3 tests/timings.py

Loaded synth; version: 1.0.3

=== Prepping ===

Directories: ('tests/templates/django/blog', 'tests/templates/django', '.')

Data: {'categories': [{'title': 'Movies', 'url': 'http://example.org/categories/movies'}, {'title': 'Shows', 'url': 'http://example.org/categories/shows'}], 'posts': [{'title': 'Rocky', 'url': 'http://example.org/posts/1', 'category': 'movies'}, {'title': 'Blackjack', 'url': 'http://example.org/posts/2', 'category': 'shows'}], 'user': 'Dolph Lundgren', 'qux': 3, 'subtitle': 'A blog about movies and TV shows', 'foo': 1, 'bar': 2}

Files: ['base.html', 'home.html', 'index.html', 'A.tpl', 'B.tpl', 'base.tpl', 'C.tpl', 'D.tpl', 'derived.tpl', 'empty.tpl', 'layout.html', 'messages.html', 'variables.tpl', 'X.tpl']

Debug: False

=== Starting ===

Iterations: 1000

Django: 0.352s; Synth: 0.030s; +/-: 11.9x; File: 'base.html'

Django: 1.733s; Synth: 0.128s; +/-: 13.5x; File: 'home.html'

Django: 1.337s; Synth: 0.117s; +/-: 11.4x; File: 'index.html'

Django: 0.284s; Synth: 0.018s; +/-: 16.0x; File: 'A.tpl'

Django: 0.683s; Synth: 0.027s; +/-: 25.7x; File: 'B.tpl'

Django: 0.353s; Synth: 0.022s; +/-: 15.8x; File: 'base.tpl'

Django: 1.066s; Synth: 0.034s; +/-: 31.3x; File: 'C.tpl'

Django: 1.451s; Synth: 0.041s; +/-: 35.2x; File: 'D.tpl'

Django: 0.852s; Synth: 0.036s; +/-: 23.9x; File: 'derived.tpl'

Django: 0.239s; Synth: 0.016s; +/-: 14.9x; File: 'empty.tpl'

Django: 2.014s; Synth: 0.084s; +/-: 24.0x; File: 'layout.html'

Django: 0.750s; Synth: 0.047s; +/-: 15.9x; File: 'messages.html'

Django: 0.417s; Synth: 0.028s; +/-: 15.0x; File: 'variables.tpl'

Django: 0.622s; Synth: 0.027s; +/-: 23.0x; File: 'X.tpl'

Average: 19.8x

=== Finished ===

 * Windows 8.1 Pro / Intel Core i7-4790 3.60Ghz / 16GB RAM:

c:\code\synth>python.exe tests\timings.py

Loaded synth; version: 1.0.3

=== Prepping ===

Directories: ('tests/templates/django/blog', 'tests/templates/django', '.')

Data: {'user': 'Dolph Lundgren', 'subtitle': 'A blog about movies and TV shows', 'foo': 1, 'posts': [{'url': 'http://example.org/posts/1', 'category': 'movies', 'title': 'Rocky'}, {'url': 'http://exam

ple.org/posts/2', 'category': 'shows', 'title': 'Blackjack'}], 'categories': [{'url': 'http://example.org/categories/movies', 'title': 'Movies'}, {'url': 'http://example.org/categories/shows', 'title'

: 'Shows'}], 'qux': 3, 'bar': 2}

Files: ['base.html', 'home.html', 'index.html', 'A.tpl', 'B.tpl', 'base.tpl', 'C.tpl', 'D.tpl', 'derived.tpl', 'empty.tpl', 'layout.html', 'messages.html', 'variables.tpl', 'X.tpl']

Debug: False

=== Starting ===

Iterations: 1000

Django: 0.339s; Synth: 0.049s; +/-: 7.0x; File: 'base.html'

Django: 1.438s; Synth: 0.152s; +/-: 9.4x; File: 'home.html'

Django: 1.089s; Synth: 0.126s; +/-: 8.6x; File: 'index.html'

Django: 0.274s; Synth: 0.035s; +/-: 7.8x; File: 'A.tpl'

Django: 0.680s; Synth: 0.056s; +/-: 12.1x; File: 'B.tpl'

Django: 0.326s; Synth: 0.038s; +/-: 8.7x; File: 'base.tpl'

Django: 1.049s; Synth: 0.077s; +/-: 13.6x; File: 'C.tpl'

Django: 1.431s; Synth: 0.096s; +/-: 14.9x; File: 'D.tpl'

Django: 0.797s; Synth: 0.065s; +/-: 12.2x; File: 'derived.tpl'

Django: 0.220s; Synth: 0.030s; +/-: 7.2x; File: 'empty.tpl'

Django: 1.566s; Synth: 0.092s; +/-: 17.1x; File: 'layout.html'

Django: 0.632s; Synth: 0.052s; +/-: 12.1x; File: 'messages.html'

Django: 0.391s; Synth: 0.042s; +/-: 9.3x; File: 'variables.tpl'

Django: 0.619s; Synth: 0.055s; +/-: 11.2x; File: 'X.tpl'

Average: 10.8x

=== Finished ===

[Russell Keith-Magee]

On Wed, Aug 20, 2014 at 3:00 AM, Alvaro J. Genial <gen...@alva.ro> wrote:

On Wednesday, August 6, 2014 7:51:09 PM UTC-4, Russell Keith-Magee wrote:

Hi Alvaro,

Firstly - congratulations - sounds like an interesting package. However, you may find that django-users is a better forum to announce something like that. Django-developers is the forum for discussing the development of Django itself, and while most of the people on this list also use django, django-users is a bigger audience.

Thank you--that's a good suggestion. I'll post an announcement there after ironing out any remaining major issues.

 

Secondly - the "would I use this" question... maybe. Faster templates are obviously a good thing; I'd be interested to see exactly how much faster (and how much faster compared to a "known fast" alternative template language like Jinja2). I'm especially interested to see how it's handled custom template tags, because that has always been the sticking point for faster *native* Django templates.

Makes sense. I have a simple benchmark script (synth/tests/timings.py) that roughly suggests ~27x faster on Ubuntu, ~20x faster on OS X and ~11x faster on Windows, given a somewhat esoteric set of machines, templates and data. (The details are at the end of this message, and all the standard benchmark caveats apply.) If there's an official benchmark suite I can run it as well and post the timings. At the moment, overall, the difference seems to range from 10x to 30x. (Python2 vs. Python3 doesn't appear to affect the outcome much, nor does 32-bit vs 64-bit.)

Is that comparison against Django templates on CPython? What about comparisons with PyPy? Or with the equivalent templates in Jinja2? I'd also be interested to see your results against the speed.python.org Django benchmark (which is mostly template driven)

 

Anyway, with the bulk of the functionality completed, I can now focus on performance, so I expect those figures to go up, as there's a lot of low-hanging fruit.

Separately, I've been studying the differences between Django's template language and Jinja2 and I'm considering implementing the latter natively as well, especially because: there'd be a lot of shared code; the Django integration glue is already there; and it seems to allow even more opportunities for speed gains. Any thoughts?

If you've got the time and inclination to implement it, then I'm sure nobody would object - especially if you get an improvement in performance :-) 

 

Lastly - you've implemented a binding layer to a new template language. Is there anything in Django's core that you would like to see modified that would make your life easier? Using a custom template language is an area where many people claim it's impossible to do (despite evidence to the contrary), so we'd like to make it as easy as possible to do. Any suggestions/requests?

Thanks for the offer! Currently, the best thing I can think of is to actually try it on real projects and report significant issues and differences in output.

One relatively small improvement I just thought of is to allow template loaders to parse top-level templates from a file path rather than accepting the source as a string. (Perhaps this is possible already?)

It's not possible in the loaders that Django ships, but you can write your own loader (and it's a pretty simple piece of code, too); see django.templates.loaders.filesystem for the current filesystem loader. The only caveat - and I'm not sure how you plan to reconcile this - is that templates are "named" in code - so, you say "render(request, "my template.html", mycontext)". That means that the public API to template is string based; no scope where the end user is passing around file pointers. 

Yours,

Russ Magee %-)

[Shai Berger]

Hi,

As this is the 4th example I see of a templating engine that is inspired by
the Django Template Language, I started a small collection of them on the
wiki[1]. I usually work with other parts of Django, but thought people who are
more into the front-end might find this interesting.

HTH,
Shai.

[1] https://code.djangoproject.com/wiki/TemplateEnginesInspiredByDjango

[Michael Buckley]

I have seen benchmarks that show Django can process its templates faster if you use django.template.loaders.cached.Loader.  Does django_synth work with that, what effect does that have on the benchmarks?

[Aymeric Augustin]

On 31 août 2014, at 21:36, Michael Buckley <test...@gmail.com> wrote:

> I have seen benchmarks that show Django can process its templates faster if you use django.template.loaders.cached.Loader. Does django_synth work with that, what effect does that have on the benchmarks?

If you look at the first email in this thread, you’ll see that django_synth provides its own template loader.

In fact, three expensive things happen when you use a Django template:

1 - Finding it on the disk. Depending on your project structure, this may require many I/O calls.
2 - Parsing and “compiling” it. This allocates lots of small objects, which can be expensive.
3 - Evaluating it. This makes many function calls, which is expensive too.

Django’s cached loader optimizes 1 and 2 through a simple in-memory cached. All it has to do is remember what templates are loaded and keep track of the tree resulting from the compilation step.

Since this strategy exists, 3 is the only thing that matters for real-world performance, where templates are loaded once and rendered many times.

Considering the approach taken by django-synth, it also takes care of 2 and 1, but that’s an implementation detail.

--
Aymeric.

[Michael Buckley]

On Sunday, August 31, 2014 10:43:48 PM UTC+2, Aymeric Augustin wrote:

If you look at the first email in this thread, you’ll see that django_synth provides its own template loader.

Sure it does, but does that mean I can do this?

TEMPLATE_LOADERS = (
    ('django.template.loaders.cached.Loader', (
        'django_synth.loaders.FilesystemLoader',
        'django_synth.loaders.AppDirectoriesLoader',
    )),
)

Or is that not needed?  I am guessing from your answer that it is not.

Also is the 10 - 30x faster, done against the speed of the django.template.loaders.cached.Loader or the default  filesystem.Loader and app_directories.Loader?  It makes a large differences.

[Vladimir U.]

Alvaro J. Genial

It would be great if someone with C++ backgrounds could participate in the development of fast Json Encoderr/Decoder - https://github.com/esnme/ultrajson (They license similar to Apache and it seems they moved away from development of the project)

Given the fact that javascript client frameworks are becoming more and more functional in nowadays, and also given the fact that all the work of rendering lays on the browser(that with new versions are getting faster), it seems to me, this is a very good direction to be able to deliver json to them as fast as it could be.

Sorry for offtopic and thanks for your efforts!