Cross-compilation with Temple (and announcing Skim -- client side Slim with embedded CoffeeScript)

[John Firebaugh]

Hi all,

My desire for using Slim as a client-side templating language led me
to explore the feasibility of using Temple in a cross-compilation
scenario. My goal was implement a Slim-alike language (called Skim)
that embeds CoffeeScript rather than Ruby and compiles to a JavaScript
template (JST) function. My strategy for getting there was to reuse as
much of Slim's existing Temple-based pipeline as possible,
substituting a CoffeeScript generator at the end and CoffeeScript-
specific filters where necessary. I found this quite feasible, and I
was able to get Skim passing most of Slim's test suite (adapted where
necessary to CoffeeScript) in a couple days. With a few tweaks, I
think the Ruby-specific parts of Temple (and Slim) could be further
abstracted, and it could become a framework for implementing templates
that compile to not just to Ruby but to any language that has a Temple
"backend".

https://github.com/jfirebaugh/skim

Here are some more details. Slim makes use of the following Temple
filters that assume that the target language is Ruby:

* `DynamicInliner` assumes interpolation syntax, line continuations
via backslash, and implicit string literal concatenation.
* `HTML::Pretty` uses bits of Ruby code.
* `AttributeMerger` and `AttributeRemover` use `[:if, "!
#{tmp}.empty?", sexp]`.
* `Escapable` uses a Ruby-specific `:escape_code` option by default,
and assumes that it can both be embedded in the output *and* evaled as
Ruby code. In my case, I wanted to use
`Temple::Utils.escape_html[_safe]` Ruby-side and `@escape` (i.e.
`this.escape`) in CoffeeScript.

I worked around these issues by providing CoffeeScript-specific
implementations of the essential filters (`AttributeMerger`,
`AttributeRemover`, and `Escapable`) and by avoiding the use of
`DynamicInliner` and `HTML::Pretty` (as a consequence, Skim doesn't
support pretty printing and is presumably slightly less efficient than
it could be).

Since CoffeeScript is whitespace sensitive, I also had to add one new
core abstraction: `[:indent, sexp]`, and use it appropriately in
`ControlFlow` and `Skim::Compiler`, so that the CoffeeScript
`Generator` can emit correctly-indented CoffeeScript.

Fortunately, I found that most of Slim's pipeline could be used as-is,
because it is mostly language agnostic, and where it isn't, it uses
syntax that is mostly common to Ruby and CoffeeScript (e.g. string
interpolation). I had to override only a couple `Slim::Compiler`
methods that emit several Ruby-specific fragments. One of them happens
also to be valid CoffeeScript, but the one dealing with dynamic array-
valued HTML attributes will need to be ported. I punted on it for the
time being. I also have yet to implement sections (logicless mode),
though I don't think it will be difficult.

I've kept the CoffeeScript portions of the implementation separated
from the rest of Skim, with the thought that they could later be
extracted to a temple-coffee gem for reuse.

Take a look, let me know if you think cross-compilation is an
interesting direction for Temple.

Cheers,
John

[Magnus Holm]

On Mon, Jan 30, 2012 at 02:17, John Firebaugh <john.fi...@gmail.com> wrote:
> Hi all,
>
> My desire for using Slim as a client-side templating language led me
> to explore the feasibility of using Temple in a cross-compilation
> scenario. My goal was implement a Slim-alike language (called Skim)
> that embeds CoffeeScript rather than Ruby and compiles to a JavaScript
> template (JST) function. My strategy for getting there was to reuse as
> much of Slim's existing Temple-based pipeline as possible,
> substituting a CoffeeScript generator at the end and CoffeeScript-
> specific filters where necessary. I found this quite feasible, and I
> was able to get Skim passing most of Slim's test suite (adapted where
> necessary to CoffeeScript) in a couple days. With a few tweaks, I
> think the Ruby-specific parts of Temple (and Slim) could be further
> abstracted, and it could become a framework for implementing templates
> that compile to not just to Ruby but to any language that has a Temple
> "backend".

The reason I haven't made Temple more language-agnostic is that it's
much more useful to have the template engine implemented in the same
language as the runtime language; less dependencies, more dynamic
possibilities (see Handlebars-JIT and Heffigy) and it makes everything
easier.

If we want to go this way, it means we'll need to more clearly specify
which compilers are language-agnostic, and split more compilers into
two parts (general part, code generation part).

Another possibility is to say that everything we have today (below the
parser) is considered Ruby-stuff. So we need to write a completely new
HTML::Pretty::CoffeeScript (or whatever). These can share code from
the default HTML::Pretty (either by inheritance or some other way).

> https://github.com/jfirebaugh/skim
>
> Here are some more details. Slim makes use of the following Temple
> filters that assume that the target language is Ruby:
>
> * `DynamicInliner` assumes interpolation syntax, line continuations
> via backslash, and implicit string literal concatenation.

Language-specific optimization; doesn't make sense to port to other
languages (other languages might have other useful optimization
filters).

> * `HTML::Pretty` uses bits of Ruby code.
> * `AttributeMerger` and `AttributeRemover` use `[:if, "!
> #{tmp}.empty?", sexp]`.

These would need to split up…

> * `Escapable` uses a Ruby-specific `:escape_code` option by default,
> and assumes that it can both be embedded in the output *and* evaled as
> Ruby code. In my case, I wanted to use
> `Temple::Utils.escape_html[_safe]` Ruby-side and `@escape` (i.e.
> `this.escape`) in CoffeeScript.

This could be better separated into a method/proc (for static stuff)
and :escape_code (for code generation).

> I worked around these issues by providing CoffeeScript-specific
> implementations of the essential filters (`AttributeMerger`,
> `AttributeRemover`, and `Escapable`) and by avoiding the use of
> `DynamicInliner` and `HTML::Pretty` (as a consequence, Skim doesn't
> support pretty printing and is presumably slightly less efficient than
> it could be).
>
> Since CoffeeScript is whitespace sensitive, I also had to add one new
> core abstraction: `[:indent, sexp]`, and use it appropriately in
> `ControlFlow` and `Skim::Compiler`, so that the CoffeeScript
> `Generator` can emit correctly-indented CoffeeScript.

Interesting. I've always heard that generation whitespace sensitive
code can be tricky, but I assume it's much easier when you already
everything as a tree-structure.

> Fortunately, I found that most of Slim's pipeline could be used as-is,
> because it is mostly language agnostic, and where it isn't, it uses
> syntax that is mostly common to Ruby and CoffeeScript (e.g. string
> interpolation). I had to override only a couple `Slim::Compiler`
> methods that emit several Ruby-specific fragments. One of them happens
> also to be valid CoffeeScript, but the one dealing with dynamic array-
> valued HTML attributes will need to be ported. I punted on it for the
> time being. I also have yet to implement sections (logicless mode),
> though I don't think it will be difficult.
>
> I've kept the CoffeeScript portions of the implementation separated
> from the rest of Skim, with the thought that they could later be
> extracted to a temple-coffee gem for reuse.

Yes, if we go for cross-compilation I'd prefer to make the temple gem
just contain language-agnostic and Ruby-specific code.

> Take a look, let me know if you think cross-compilation is an
> interesting direction for Temple.

I'm not sure. It's pretty amazing seeing (pretty much) the same
compilers generate code for both CoffeeScript and Ruby, but wow long
will it take before someone ports Skim to JavaScript? This isn't
actually a million-lines-codebase so porting it to another language
(while basing it on Temple ideas) wouldn't take so long. And for how
many languages is it actually useful to be able to cross-compile? Is
it worth the hassle?

Bernard Lambeau has also been working on cross-compilation for WLang2,
so we should probably figure what he needs too:
https://github.com/blambeau/wlang/tree/wlang2

> Cheers,
> John

Thanks for a *very* cool project, John :-)

[John Firebaugh]

On Wed, Feb 1, 2012 at 1:38 AM, Magnus Holm <jud...@gmail.com> wrote:

The reason I haven't made Temple more language-agnostic is that it's

much more useful to have the template engine implemented in the same
language as the runtime language; less dependencies, more dynamic
possibilities (see Handlebars-JIT and Heffigy) and it makes everything
easier.

For a Ruby webapp developers specifically -- the target audience for Temple-based languages -- I would actually argue the opposite. It's less useful and makes everything harder. Look at a template engine like Haml that started out as a Ruby library. It gets ported to JavaScript to run client-side (haml-js), but it turns out we'd rather serve the client something precompiled, so instead we use ExecJS to get back to compiling them from Ruby, either server-side or as part of release deployment. We've added a major dependency to our deployment or runtime -- a JS runtime (ask a Heroku engineer how that went down). And assuming that we also want to use Haml as a server-side language, we now have to track two separate (and inevitably subtly different) implementations.

I'd rather share a parser and filter pipeline, as Slim and Skim do, and stick to Ruby on the server.

 

> * `AttributeMerger` and `AttributeRemover` use `[:if, "!
> #{tmp}.empty?", sexp]`.

These would need to split up…

I think it's worth looking at this one carefully even if Temple remains Ruby-specific, because I think there's an abstraction there (shared with Slim's `[:slim, :attr]` logic) trying to get out.

 

Interesting. I've always heard that generation whitespace sensitive
code can be tricky, but I assume it's much easier when you already
everything as a tree-structure.

Yeah, it was easier than I thought it would be too. Yay for ASTs!

 

Yes, if we go for cross-compilation I'd prefer to make the temple gem
just contain language-agnostic and Ruby-specific code.

Agreed.

 

> Take a look, let me know if you think cross-compilation is an
> interesting direction for Temple.

I'm not sure. It's pretty amazing seeing (pretty much) the same
compilers generate code for both CoffeeScript and Ruby, but wow long
will it take before someone ports Skim to JavaScript? This isn't
actually a million-lines-codebase so porting it to another language
(while basing it on Temple ideas) wouldn't take so long. And for how
many languages is it actually useful to be able to cross-compile? Is
it worth the hassle?

It's a valid question. I think if we had more Temple-based languages that people wanted to use client-side, the balance would tip towards cross-compilation. But it may not be worth too much effort at this point.

BTW, do you know of any Temple equivalent's for JS? I wouldn't want to do a port without one. :)

John