Closure Templates Status Update

[Talin]

I wanted to let folks know that Closure Templates is under heavy development within Google right now, with a lots of new stuff coming down the pipe. As you probably know, our main focus with Closure Templates is performance and security, and we're constantly working to improve the code in those areas.

We're not planning on making any changes to the templating language itself, but there's a lot of exciting stuff happening under the hood:

• A completely rewritten (and much faster) engine for server-side rendering in Java.
• Improved error reporting, including source snippets. No longer does the compiler simply quit after the first error.
  
• A backend for generating Python code.
• Architectural changes that allow partially-rendered templates on the server to be sent to the browser earlier, while the rest of the template is still being rendered. This includes lazy evaluation of sub-templates so that content is rendered in page order instead of call-tree order.

The other thing I'd like to mention is that we haven't done a very good job of evangelizing Closure Templates to the rest of the world. I'd be interested if anyone has any ideas or suggestions for letting people know about the work we're doing.

--

-- Talin

[Rob Figueiredo]

Hi Talin!

Hearing about the roadmap is really helpful, thank you!  I'm very glad to hear about the focus (improving performance, security, portability instead of new language changes).

I have a couple questions
- is the final state for escaping going to be strict by default and all variables / params having kinds?
- what is the end goal for the dichotomy between SoyData and SoyValue? Presently we're using the last release (april 2014) and the api seems to be In transition and hard to use. For example there is no built-in SoyValue map type that is mutable
- any plans for a go backend?
- how do you achieve sending page content before rendering has completed without showing a garbled page in the event of an execution exception? (I always write responses to a temporary buffer first for this reason)

On the topic of evangelization, here is my take on why closure templates has not done as well as it merits:
- not integrated into any popular web framework. Most developers use frameworks and most frameworks are very particular about which template language they use.
- in terms of mind share, java is not exactly at the front for web development, and this is mostly targeted at the java users
- previously, it was a source drop with no information about the roadmap or community efforts (looks like that's changed!)

It's great to see more activity on the open source project. One thing that would be helpful is having official releases, including notes on changes and a set of the documentation for that version in particular. Right now I have to select a revision at random, host my own javadoc, gloss over any language differences between that and the last release, etc. Means I basically can't upgrade from last year's release. An example of doing it right is play framework, where you can select the version you're using and see the docs as of that release (javadoc, framework details, etc).

I think closure templates are technically superior to everything else, so that's a good place to start :).

Thanks again,
Rob

n Tue, Mar 31, 2015, 13:52 'Talin' via Closure Templates Discuss <closure-temp...@googlegroups.com> wrote:

I wanted to let folks know that Closure Templates is under heavy development within Google right now, with a lots of new stuff coming down the pipe. As you probably know, our main focus with Closure Templates is performance and security, and we're constantly working to improve the code in those areas.

We're not planning on making any changes to the templating language itself, but there's a lot of exciting stuff happening under the hood:

A completely rewritten (and much faster) engine for server-side rendering in Java.Improved error reporting, including source snippets. No longer does the compiler simply quit after the first error.
A backend for generating Python code.Architectural changes that allow partially-rendered templates on the server to be sent to the browser earlier, while the rest of the template is still being rendered. This includes lazy evaluation of sub-templates so that content is rendered in page order instead of call-tree order.

The other thing I'd like to mention is that we haven't done a very good job of evangelizing Closure Templates to the rest of the world. I'd be interested if anyone has any ideas or suggestions for letting people know about the work we're doing.

--

-- Talin

--

---
You received this message because you are subscribed to the Google Groups "Closure Templates Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to closure-templates-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Talin]

I don't have all the answers - perhaps one of the other folks will chime in. I will tell you a few things.

First, one of the things we're currently working on is revamping the documentation, and the plan is to host all of the reference documentation on GitHub. This has turned out to be a fairly difficult task - for one thing, the docs are all written in hand-coded HTML, which is something we'd like to change - but it is being worked on.

Strict autoescaping is the current default, and we're working on fixing holes in the current type system. One area that needs improvement is callable functions from within expressions, which are currently untyped.

I remember hearing about a Go implementation of Soy, but I believe it's written entirely in Go and thus is essentially a fork. Given the size of the project, a Go backend isn't something we'd take on outselves unless one of our internal customers asked for it, which seems unlikely.

I agree with you about Soy map types. Internally we have a type provider class that allows Soy to talk directly to protocol buffers without the need to convert data to the various SoyValue types, and we've been using that more and more lately. There's no reason we couldn't open-source the type provider, except that (a) we didn't want to create a library dependency from Soy to libproto, and (b) it assumes that on the browser you are using JSPB as your protocol buffer implementation, which most people don't use. Anyway, we didn't think that external users would be too interested in this. But in general, the direction I'd like to see is for Soy to be able to use reflection to access object fields directly instead of having to convert all of the data to a form that Soy understands.

--

-- Talin

[Luke Sandberg]

RE: SoyData v.s. SoyValue.  This is something i would like to clean up (it is a constant confusion) and it has been on our list of things to fix for a while.  This appears to be the result of an old migration that was never finished before the previous maintainer left, unfortunately none of the current maintainers have a good understanding of exactly what the goals were :(  As part of my work on the new backend I will definitely be re-evaluating the entire set of runtime data structures that are used by soy, these included.  So there i no current plan other than i expect i will learn enough about these things to fix this eventually.

However, as a user i would recommend just using SoyValueHelper to convert objects before calling templates whenever possible and avoiding using any of these types (if you can).  e.g. just use a HashMap or an ImmutableMap instead any of the SoyMap types.

Re: how do you achieve sending page content before rendering has completed without showing a garbled page in the event of an execution exception?

This is a difficult issue so we mostly kick it back to our users.  If you don't want garbled content you either need to give us futures that won't fail, or need to render into a buffer.  Some of our more sophisticated applications have frameworks that wrap soy rendering and handle this.  Or sometimes, depending on the request type we just serve garbled content.

Re: not integrated into any popular web framework.

yeah i agree, unfortunately the major web platforms that use soy at google are all proprietary (though they do have nice easy integrations with soy!), so this is definitely something we should improve on, though the current development team isn't in a great position to provide such integrations (i don't even know what web platforms are popular! play? tomcat? netty? do these words even make sense?).  I think this would be a great place for external contributions.

[Luke Sandberg]

actually, It just ocurred to me, talin@ has a document about technical debt in soy and things we have been meaning to fix.  talin, what do you think of just converting that to markdown and adding it to the soy project? or maybe we should just convert everything to github issues

[Talin]

Actually, about half of the things in that document have already been fixed or are being worked on :) We should probably update it to reflect our current understanding before we turn it loose on an unsuspecting world :)

[Paul Johnston]

A couple of other things to consider:

* There are few examples of how to use closure templates.  If there were more concrete examples showing use of the features, particularly in the client or hybrid use cases, that would be very helpful. Providing a background of how these use cases are applied for google products would be a good marketing angle.

* Most people (including myself!) have only a basic understanding of security as it applies to templating and XSS attacks in general, and rely on the tools to hopefully provide some insulation for this.  A blog post on XSS and templating security, and how closure templates evolved to circumvent these would be great.

* If closure templates has better security than alternatives, perhaps a direct comparison with popular templating engines would be interesting.

* Unclear integration with popular build tools.  A summary document of how to use closure templates in rails, node.js, etc would be good.

* This is getting a little petty, but like most programmers, I experience emotional responses to the naming of things. I was put off by the severe namespace pollution of Soy everywhere in the framework, and Soy is up there for my least favorite named project, although I like the name 'closure templates'.  I even almost performed the experiment of replacing every instance of 'Soy' in the codebase with the empty string.  My prediction is that it would compile just fine, and be clearer.  I'd keep the filename extension *.soy and ditch the rest.  I don't expect this to actually occur.  Sorry for the rant. 

* I do agree closure templates is technically better than the alternatives, and should be used more.

Paul

[Rob Figueiredo]

Aw, I like the same soy :). Much less cumbersome to say and type than "closure templates, not to be confused with clojure, actual closures, or any of the companion parts of the suite"

+1 on the utility of tutorials / examples for how to get started with the popular frameworks. Here are the main ones that I'm under the impression are popular:
Js: node.js, express
Java: play, dropwizard
Python: django

If you published benchmarks, you would crush the alternatives. In particular, for play 1.x, the built-in groovy templates are incredibly slow and I found it took 300ms(!!!!) To render a reasonably complicated page. Soy was 40x faster (no exaggeration). Oh, and groovy templates just do basic escaping (like soy v1)

Also, "data driven" templates have mind share, and soy is basically that, since it forces you to create view models rather than writing code in the template -- in my experience, this leads to wayyyy better code.

--

[Talin]

I don't recall whether I mentioned this, but there was a period in Google's history where it was popular to name large Java-based technologies after various kinds of beans - Fava, Pinto, Soy, etc. I guess someone was riffing on Java Beans and things kind of got out of hand.

--

-- Talin

[Nick Santos]

Excited about this! My wish list would be:
- a pluggable codegen backend (in particular for Go; Rob's fork has
been a big help here but there are some subtle differences caused by
implementation skew)
- better error reporting (i'm sad every time i see a java stack trace
in my compilation error report)
- compile-time and run-time checks that $ij was passed in properly
- a way to use compile-time constants in expressions

i'm also sad how little the templating community seems to care about
security and escaping approaches. It would be cool to see some blog
posts that made the case for why this is an important factor in
choosing a template engine. in particular, I'd love to read more of
the technical details on why contextual autoescaping was deemed a
failure, and explains strict autoescaping more fully.

(as an aside, "strict" autoescaping is a terrible name for it)

[Nick Santos]

Also, for PR purposes, one thing I've found helps a lot is just to
have a page that says, "here are good ways to use this in environments
"X, Y, and Z."

For example, I think there is a gulp plugin
(https://www.npmjs.com/package/gulp-soynode) and some grunt plugins,
but I have no idea if they're any good?

[Luke Sandberg]

 better error reporting (i'm sad every time i see a java stack trace
in my compilation error report)

I agree!  There has been a lot of work on this recently, at HEAD most parse errors will now be reported in the style of traditional compiler errors and you will get more than one at a time!  It will be a while before this is propagated through all the compiler passes (end of q2?), but it will happen!

[Luke Sandberg]

FYI: the design doc for the new server backend that talin mentioned above has just been released:

https://github.com/google/closure-templates/tree/master/java/src/com/google/template/soy/jbcsrc

[Rob Figueiredo]

Very cool, thanks for sharing. 

I think that the bytecode generation should provide a nice speedup.  I'm curious about the value of eliminating all blocking though -- I recall seeing research (including from Google) indicating that current-day threading is really fast and the complication necessary to avoid blocking is not worth while (that's essentially what the threading system does anyway).  As reference, here's a paper written (admittedly a long time ago) by Rob von Behren, who IIRC wrote Google's rpc3 Java library

http://www.cs.berkeley.edu/~brewer/papers/threads-hotos-2003.pdf

It's probably too much work to be worth it, but I would be curious how the blocking bytecode solution compares against the non-blocking (maybe a toy example).  If it performs similarly, maybe it's not worth the extra work to go full non-blocking. In that case, you could actually go the route of Java source generation instead of direct-to-bytecode.

[Luke Sandberg]

(unscientific claim, it seems like the events vs. threads debate is on a 5 year cycle where the collective consensus flips back and forth :) )

A lot of the motivation was actually about the _increasing_ cost of threads.  In 64bit java the default stack size (at least on the servers i run) is 1MB, and 4MB is not unheard of,, so to handle thousands of connections to slow clients (phones) you would need thousands of threads (and thus many GBs of RAM).  Most of the servers i run would rather spend that RAM on caches and other data structures.   So the real issue about blocking/non-blocking is actually about utilization, not performance.   I think it is quite likely that the blocking solution would perform slightly better (since you don't have to execute the save/restore logic, you just switch to a new thread stack).  However, the new compiler should be significantly faster than the current interpreter (i do plan on releasing benchmarks eventually), so this extra cost is believed to be minuscule compared to the overall performance gains.

The primary customers that i am trying to support with these new features are large servers (as in 1000s of instances across many data centers), and these servers are already often RAM constrained, so being able to run with fewer threads (and faster) should be an all around win.

I realize that these utilization concerns aren't necessarily relevant or important to many of the opensource community, but the overall performance gains from compilation should be :)

[Hochhaus, Andy]

Talin,

Sorry to dig up an old thread...

On Thu, Apr 2, 2015 at 10:24 AM, 'Talin' via Closure Templates Discuss <closure-temp...@googlegroups.com> wrote:

I agree with you about Soy map types. Internally we have a type provider class that allows Soy to talk directly to protocol buffers without the need to convert data to the various SoyValue types, and we've been using that more and more lately. There's no reason we couldn't open-source the type provider, except that (a) we didn't want to create a library dependency from Soy to libproto, and (b) it assumes that on the browser you are using JSPB as your protocol buffer implementation, which most people don't use. Anyway, we didn't think that external users would be too interested in this.

When you mention JSPB are you speaking of the goog.proto2 implementation in closure-library? If so, at least I'm interested (as an external user). Any chance of having this open sourced? :)

-Andy

[Rob Figueiredo]

Hi all,

Just curious if anyone knows what's changed / completed / improved since the April 2014 release. I'm generally curious how progress is going on the jbsrc & incremental dom backends, and if there's any other changes of note. I'm considering upgrading my company to latest of all closure tools revisions (mostly to get latest closure compiler)...

Thanks,

Rob

[Luke Sandberg]

The jbcsrc backend is considered mature and is in use by several large services at google.  The incremental dom backend is unfortunately not really ready yet and the main developer on the project left the team and has yet to be replaced, so the project is in stasis.  The other major thing that happened is that we released our support for protocol buffers, so you can now pass protos to your templates as parameters. 

[Mike Samuel]

On Thu, Sep 8, 2016 at 10:49 AM, Luke Sandberg <lukeis...@gmail.com> wrote:
> The jbcsrc backend is considered mature and is in use by several large
> services at google. The incremental dom backend is unfortunately not really
> ready yet and the main developer on the project left the team and has yet to
> be replaced, so the project is in stasis. The other major thing that
> happened is that we released our support for protocol buffers, so you can
> now pass protos to your templates as parameters.

If you want an example of how to provision the jbcsrc backend, see
https://github.com/mikesamuel/closure-maven-plugin/blob/master/module/src/main/java/com/google/closure/module/ClosureModule.java
which also wires up protobuf support.

That's part of a side project of mine, closure-maven-plugin, which
ties Soy + JSCompiler + CSSCompiler + protoc together on a maven
stack.

https://github.com/mikesamuel/closure-maven-plugin/blob/5a57dd003e548615e3604742e20ba5b152748139/plugin/src/it/demo/src/main/java/com/example/demo/Demo.java
shows the same template being used from both the jbcsrc & jssrc Soy
backends.

<shameless-plug>
I'll be talking about that demo app in the context of secure code
patterns at JavaOne.
https://oracle.rainfocus.com/scripts/catalog/oow16.jsp?event=javaone&search=samuel&search.event=javaone

[Hochhaus, Andy]

Thanks for the extra context Mike and Luke!

Any chance that the official soy go implementation could be open sourced? :)

I know about Rob's implementation but it has a few limitations:

• Not as widely used/tested as the Google propretary version
• No contextual autoescaping
  
• Missing newer features like @param and protobuf support

On Thu, Sep 8, 2016 at 8:55 AM, Mike Samuel <mikes...@gmail.com> wrote:

On Thu, Sep 8, 2016 at 10:49 AM, Luke Sandberg <lukeis...@gmail.com> wrote:
> The jbcsrc backend is considered mature and is in use by several large
> services at google.  The incremental dom backend is unfortunately not really
> ready yet and the main developer on the project left the team and has yet to
> be replaced, so the project is in stasis.  The other major thing that
> happened is that we released our support for protocol buffers, so you can
> now pass protos to your templates as parameters.

If you want an example of how to provision the jbcsrc backend, see
https://github.com/mikesamuel/closure-maven-plugin/blob/master/module/src/main/java/com/google/closure/module/ClosureModule.java
which also wires up protobuf support.

That's part of a side project of mine, closure-maven-plugin, which
ties Soy + JSCompiler + CSSCompiler + protoc together on a maven
stack.

https://github.com/mikesamuel/closure-maven-plugin/blob/5a57dd003e548615e3604742e20ba5b152748139/plugin/src/it/demo/src/main/java/com/example/demo/Demo.java
shows the same template being used from both the jbcsrc & jssrc Soy
backends.

Very helpful examples.

 

<shameless-plug>
I'll be talking about that demo app in the context of secure code
patterns at JavaOne.
https://oracle.rainfocus.com/scripts/catalog/oow16.jsp?event=javaone&search=samuel&search.event=javaone

Excellent. I'll check it out.

-Andy

[Mike Samuel]

On Thu, Sep 8, 2016 at 12:24 PM, Hochhaus, Andy <ahoc...@samegoal.com> wrote:
> Thanks for the extra context Mike and Luke!
>
> Any chance that the official soy go implementation could be open sourced? :)
>
> I know about Rob's implementation but it has a few limitations:
>
> Not as widely used/tested as the Google propretary version
> No contextual autoescaping
> Missing newer features like @param and protobuf support

Not Soy relevant, but I wrote https://golang.org/pkg/html/template/
for Go. It has support for contextual autoescaping and I believe it
has protobuf support via text/template but I'm not too sure about
that. It doesn't have some of the strict mode bells and whistles that
the Soy autoescaper has.

[Hochhaus, Andy]

Thanks Mike.

I love html/template and use it for cases where all of my templating is done on the backend in go.

However, sharing the same template files between the frontend and backend allows for whole new designs (eg: render the initial page from a template on the backend for quick page loads, update [a portion] of the page based on user interaction on the front end). Sharing templates makes this much easier.

For rich web apps I see a lot of value to having first class go soy support (as distinct from html/template).

-Andy

[Mike Samuel]

On Thu, Sep 8, 2016 at 2:00 PM, Hochhaus, Andy <ahoc...@samegoal.com> wrote:
> However, sharing the same template files between the frontend and backend
> allows for whole new designs (eg: render the initial page from a template on
> the backend for quick page loads, update [a portion] of the page based on
> user interaction on the front end). Sharing templates makes this much
> easier.

+1.

https://github.com/mikesamuel/closure-maven-plugin/blob/e53bd43f0b6d6c718e889bdc90b4fe0b95ee288d/plugin/src/it/demo/src/main/js/com/example/demo/wall-item-fixed.js#L88-L103

[Hochhaus, Andy]

On Thu, Sep 8, 2016 at 11:44 AM, Mike Samuel <mikes...@gmail.com> wrote:
> On Thu, Sep 8, 2016 at 2:00 PM, Hochhaus, Andy <ahoc...@samegoal.com> wrote:
> > However, sharing the same template files between the frontend and backend
> > allows for whole new designs (eg: render the initial page from a template on
> > the backend for quick page loads, update [a portion] of the page based on
> > user interaction on the front end). Sharing templates makes this much
> > easier.
>
> +1.

Awesome. :) Your demo code is an excellent example of this in practice.

Any thoughts around an open source release of go soy?

-Andy

[Mike Samuel]

[off list]

> Any thoughts around an open source release of go soy?

I'm unaware of a Soy Go backend within Google, but it's a large
organization and I'm pretty blinkered.
Luke is much better positioned to field this question.