using clojure for (java) code generation; advice sought

677 views
Skip to first unread message

B Smith-Mannschott

unread,
Mar 28, 2011, 4:36:09 PM3/28/11
to clo...@googlegroups.com
Hi all!

I'm using Clojure to generate java source (enums, specifically) and am
looking for ideas.

Currently my generator consists of a collection of function which call
each other tree-recursion-like.

Input:
Each function takes a map of input. Much of this is constant for any
one run of the generator. Only a few of the leaf functions don't
follow this convention and instead ask for exactly what they need. I
hit on this solution as a way to avoid a lot of global state while at
the same time not causing rippling changes in parameter lists every
time some leaf procedure needs some extra bit of information.

Output:
Instead of printing directly to *out* or returning a String, I have
all the functions return a sequence of values. The result of a run of
the code generator is then initially a tree of
things-that-can-be-made-into-strings, which I flatten and concatenate
at the very end. Strings concatenate as themselves. Symbols and
keywords as their (name). Class objects print as their fully qualified
class name. BigDecimals are formatted as appropriate constructor
calls. By not printing eagerly, I can revise the result of called
procdures, e.g. by using interpose to insert commas between a sequence
of fragments produce by another function or for comprehension.

A small example:

(defn getter
[method field type]
["
public " (unbox type) " " (name method) "() {
return " (name field) ";
}\n"])

Another:

(defn constructor
"Defines the enum's constructor. The constructor uses the arguments
it is given to initialize all fields declared by (fields). Each (member)
is a call to this constructor."
[{:keys [class-name fields] :as cfg}]
[" " class-name "(" (formal-params cfg) ") {\n"
(statement-list
(for [f fields]
[" this." f " = " f ]))
" }\n"])

Where:

(defn statement-list [list]
(concat (interpose ";\n" list) [";\n"]))

The templating aspect of all this is solved with constructs Clojure
already gives me (vectors and plain old clojure expressions), which is
kind of nice. It's also reasonably friendly for exploratory testing
and interactive development. Still, I'm not really happy with it.

This works, OK, but it feels pretty hokey. Indentation is totally
ad-hoc. I mean, it works, but I can't help but think that there must
be a better way.

I've had a few false starts trying to improve on this idea and am
looking for a new approach.

Currently I think I'd like a way to write/construct a *simple*
parse-tree for a java subset as Clojure data along with some way to
serialize this as syntactically correct Java. That sounds like a lot
of work, maybe there's an easier way that I'm overlooking?

// Ben

Saul Hazledine

unread,
Mar 29, 2011, 4:06:48 AM3/29/11
to Clojure
Hello Ben,

On Mar 28, 10:36 pm, B Smith-Mannschott <bsmith.o...@gmail.com> wrote:
> Hi all!
>
> I'm using Clojure to generate java source (enums, specifically) and am
> looking for ideas.
>

I've done this quite a lot in the past using Python to generate C++.
After trying different methods, I used a templating system (Cheetah)
so that the C++ was in separate files with minimal markup (ie it still
looked like C++).

> A small example:
>
> (defn getter
>   [method field type]
>   ["
>     public " (unbox type) " " (name method) "() {
>         return " (name field) ";
>     }\n"])
>


> The templating aspect of all this is solved with constructs Clojure
> already gives me (vectors and plain old clojure expressions), which is
> kind of nice. It's also reasonably friendly for exploratory testing
> and interactive development. Still, I'm not really happy with it.
>

This is very much a personal opinion but I'd disagree with you there.
My guess is that you've put some work into the templating and that you
are ignoring that it might be holding you back (I make this sort of
mistake all of the time which is why I think I recognise it). A famous
film director once said (I can't remember who) something like "you
know you've matured as a film director when you can cut out your
favourite scene because the cut improves the film as a whole".

I'd suggest using a separate text templating system such as:
clojure.contrib.strint [1]
Velocity [2]
StringTemplate [3]

That way you write the Java as Java but replace occasional things with
markup e.g $varname$ and then pump data from Clojure into the
template.

Incidently, I've also used a similar technique to mark up SVG to
generate multiple images that are then built into movies - it works
like a charm considering how low tech it is.


Saul

[1] http://clojure.github.com/clojure-contrib/strint-api.html
[2] http://www.javaworld.com/javaworld/jw-12-2001/jw-1228-velocity.html
[3] http://weblogs.java.net/blog/aberrant/archive/2010/05/25/using-stringtemplate-part-1-introduction-stringtemplate

B Smith-Mannschott

unread,
Mar 29, 2011, 9:13:23 AM3/29/11
to clo...@googlegroups.com

I can see the advantages of going this route for simple templates that
contain no optional elements and no repetition, but that's not going
to get me far. I still don't understand how these kinds of issues:

(defn constructor


[{:keys [class-name fields] :as cfg}]
[" " class-name "(" (formal-params cfg) ") {\n"
(statement-list
(for [f fields]
[" this." f " = " f ]))
" }\n"])

... are handled in the usual templating solutions without needing a
turing complete templating language. But, at that point, I'm not sure
what I've gained, as I already have a perfectly servicable programming
language in Clojure. How does one really separate "content" from
"presentation" in such a case?

// Ben

> Incidently, I've also used a similar technique to mark up SVG to
> generate multiple images that are then built into movies - it works
> like a charm considering how low tech it is.
>
>
> Saul
>
> [1] http://clojure.github.com/clojure-contrib/strint-api.html
> [2] http://www.javaworld.com/javaworld/jw-12-2001/jw-1228-velocity.html
> [3] http://weblogs.java.net/blog/aberrant/archive/2010/05/25/using-stringtemplate-part-1-introduction-stringtemplate
>

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

Saul Hazledine

unread,
Mar 29, 2011, 10:04:19 AM3/29/11
to Clojure
On Mar 29, 3:13 pm, B Smith-Mannschott <bsmith.o...@gmail.com> wrote:
> I can see the advantages of going this route for simple templates that
> contain no optional elements and no repetition, but that's not going
> to get me far. I still don't understand how these kinds of issues:
>
> (defn constructor
>  [{:keys [class-name fields] :as cfg}]
>  ["    " class-name "(" (formal-params cfg) ") {\n"
>   (statement-list
>    (for [f fields]
>      ["        this." f " = " f ]))
>   "    }\n"])
>
> ... are handled in the usual templating solutions without needing a
> turing complete templating language. But, at that point, I'm not sure
> what I've gained, as I already have a perfectly servicable programming
> language in Clojure.  How does one really separate "content" from
> "presentation" in such a case?

Unlike a lot of people, I personally I don't see the problem as
avoiding expressiveness in the templates.

I see the problem as one of context switching between generating code
and the template. The Hiccup library does this nicely by allowing
datastructures to be the template and clojure s-expressions to be the
generating code. The cost of this is that formatting is lost -
however, this is actually a feature since the minimized output is read
by a browser rather than a human being.

There are two additional problems when generating a algol-like
programming language from within Clojure. The language itself is hard
to express within clojure and the formatting is usually important.
Because of this I would recommend a templating system that has been
specifically written to preserve formatting. If you need loops then
StringTemplate supports them.

I haven't done this within Clojure but below is small example of a C++
Cheetah template:

https://gist.github.com/892415

Although it looks a bit of a mess the general structure of the C++ is
clear and the templating code is not too intrusive.

Saul

Jonathan Smith

unread,
Mar 29, 2011, 10:30:42 AM3/29/11
to Clojure
You might want to read through the source of scriptjure:
https://github.com/arohner/scriptjure

For one way to do this sort of thing. Pretty much, you would make a
basic recursive descent parser that operates on a tree of clojure
primitives. You could then wrap this in a macro, and go from there.

Note that there is a lot more similarity between clojure and
javascript than there is between clojure and java (oddly enough).

I also wonder if there is a way to do what you are trying to do
without emitting text.
(Possibly by extending the java.lang.Enum class? No idea...)

Meikel Brandmeyer

unread,
Mar 29, 2011, 10:56:01 AM3/29/11
to Clojure
Hi,

On 29 Mrz., 15:13, B Smith-Mannschott <bsmith.o...@gmail.com> wrote:

> How does one really separate "content" from
> "presentation" in such a case?

Maybe not feasible, but a thought, that just came to my mind: do it
like enlive. Parse the templates into an abstract tree and then use
selectors to modify the tree. Do the indentation for readability when
printing. Of course you cannot select on tags, but one can find a some
way to do it. Eg. (snippet "class X { void some-method()
{ template; } }" [methods] [:method] (clone-for [[args body] methods]
[:fn-arg-list] (content (interpose \, args)) [:fn-body] (substitute
body))).

Just a weird idea.

Sincerely
Meikel

Saul Hazledine

unread,
Mar 29, 2011, 12:44:48 PM3/29/11
to Clojure
I had a think about using Clojure rather than go to a separate
template system. Here's a horrible hack that uses eval to support
string templates:

(ns clj-template.core
(:require [clojure.contrib.string :as string]))

(defn remove-templating [s]
(string/replace-re #"#" "\"" s))

(defn build-code [s]
(str "(str \"" (remove-templating s) "\")"))

(defmacro foreach [loop-stuff & body]
`(apply str
(for ~loop-stuff
(str ~@body))))

(defn eval-template [s]
(-> (build-code s)
read-string
eval))

In the templates anything surrounded by #'s (hashes) is a clojure form
that should be a compatible argument to str.

On Mar 29, 3:13 pm, B Smith-Mannschott <bsmith.o...@gmail.com> wrote:
>
> (defn constructor
>  [{:keys [class-name fields] :as cfg}]
>  ["    " class-name "(" (formal-params cfg) ") {\n"
>   (statement-list
>    (for [f fields]
>      ["        this." f " = " f ]))
>   "    }\n"])
>

Then becomes:

(eval-string "
public Example(#(format-params cfg)#)
{
#(foreach [f fields]#
this.#f# = #f#;
#)#
}")

The general idea might be applicable to what you want.

Saul

B Smith-Mannschott

unread,
Mar 29, 2011, 4:43:24 PM3/29/11
to clo...@googlegroups.com
On Tue, Mar 29, 2011 at 16:04, Saul Hazledine <sha...@gmail.com> wrote:
> On Mar 29, 3:13 pm, B Smith-Mannschott <bsmith.o...@gmail.com> wrote:
...

> I haven't done this within Clojure but below is small example of a C++
> Cheetah template:
>
> https://gist.github.com/892415
>
> Although it looks a bit of a mess the general structure of the C++ is
> clear and the templating code is not too intrusive.

Ah, thanks for example. One question though:

enum Var_name { #for $var in $variables.keys() # $var , #end for # VAR_END };

This seems almost like cheating because having VAR_END means we don't
have to deal with every item, but the last is followed by a comma
problem. Does velocity provide a way to address this easily?

// Ben

B Smith-Mannschott

unread,
Mar 29, 2011, 4:43:34 PM3/29/11
to clo...@googlegroups.com

Thanks Meikel,

I'm currently looking at enlive to see if I can understand how it
works and perhaps adapt its approach to my needs. At first blush, it
looks technically interesting. :)

// ben

B Smith-Mannschott

unread,
Mar 29, 2011, 4:43:28 PM3/29/11
to clo...@googlegroups.com

Horrible hack, maybe, but it got me thinking. What you seem to be
doing is moving between "code" and "literal" mode by quoting with #.
This is a little like traditional quasi-quote...

And that got me thinking about Scribble [1] again. In this context I
think of scribble as being sort of an inverse of normal scheme syntax.
In the end, the scribble reader produces the same kind of data
structures as the normal scheme reader, but the emphasis is moved from
code to textual content. Source is content by default, but you can
escape into logic.

That's what your little hack made me think of. The text embedding in
the string literal is content by default, but # # allows one to escape
into "code mode".

Perhaps a Scribble's approach could be useful for a templating system?
I'm not sure. I haven't really grokked it yet.

[1] http://download.plt-scheme.org/doc/html/scribble/index.html

// Ben

> The general idea might be applicable to what you want.
>
> Saul
>

B Smith-Mannschott

unread,
Mar 29, 2011, 4:43:43 PM3/29/11
to clo...@googlegroups.com
On Tue, Mar 29, 2011 at 16:30, Jonathan Smith
<jonathan...@gmail.com> wrote:
> You might want to read through the source of scriptjure:
> https://github.com/arohner/scriptjure
>
> For one way to do this sort of thing. Pretty much, you would make a
> basic recursive descent parser that operates on a tree of clojure
> primitives. You could then wrap this in a macro, and go from there.
>
> Note that there is a lot more similarity between clojure and
> javascript than there is between clojure and java (oddly enough).

I've looked a little at scriptjure. I could see writing a tree walker
for some subset of pseudo-clojure that spits out java code, though it
sounds like a lot of work. I think I'll save that up in case I don't
find another satisfactory approach.

> I also wonder if there is a way to do what you are trying to do
> without emitting text.
> (Possibly by extending the java.lang.Enum class? No idea...)

I thought of that, but didn't pursue the idea because it's useful to
my colleagues to have access to readable source code of the enum
classes. This is particularly relevant because most of these enums
carry not just their identity (i.e. their name()), but also additional
metadata immutable data accessible through final fields on the enum
instances of any given type.

It's useful to have source to look at to see what those values are,
particularly during the refactoring when we're replacing our previous
approach: value objects named uniquely by a pair of strings (the first
being analogous to a class), containing additional data, and loaded
from a database on startup. This turns out to be a unit-testing
nightmare. Unit testing was not considered in the original design of
the system. Replacing these objects with enums will help here.

Also, I was reluctant to go the route of using Clojure to try to roll
my own Enums because I didn't want to inadvertently pull in runtime
dependencies on Clojure. Also, I'd want to imitate what the Java
compiler does so my Enums work just like the "real thing". A potential
source of error.

// Ben

Ken Wesson

unread,
Mar 29, 2011, 7:56:59 PM3/29/11
to clo...@googlegroups.com

TeX and LaTeX work that way, too. And in a certain sense so does
syntax quote in Clojure itself, where the macro expansion source is
the "content" and unquoted expressions can fill bits in by "escaping
into logic".

Stuart Sierra

unread,
Mar 30, 2011, 9:00:24 AM3/30/11
to clo...@googlegroups.com
Take a look at http://www.stringtemplate.org/ for a Turing-complete, purely-functional, Java-based template language designed for code generation.

-Stuart Sierra
clojure.com

Eli Barzilay

unread,
Mar 30, 2011, 11:00:18 PM3/30/11
to Clojure
On Tue, Mar 29, 2011 at 4:43 PM, B Smith-Mannschott wrote:
> Horrible hack, maybe, but it got me thinking. What you seem to be
> doing is moving between "code" and "literal" mode by quoting with #.
> This is a little like traditional quasi-quote...
>
> And that got me thinking about Scribble [1] again.  In this
> context I think of scribble as being sort of an inverse of normal
> scheme syntax. In the end, the scribble reader produces the same
> kind of data structures as the normal scheme reader, but the
> emphasis is moved from code to textual content. Source is content
> by default, but you can escape into logic.

This is kind of a good summary of the reader's intended use, beyond
our documentation system: it's fitting for any kind of code that deals
with a lot of textual content -- and having a kind of cheap DSLs for
interacting with other tools by generating source is exactly one of
the things that you can do with it. But getting that from the
scribble documentation will be a little hard -- I wrote a paper on
just the implementation and use cases of the reader that will be much
more fitting in this context, with some examples of such uses:
http://barzilay.org/misc/scribble-reader.pdf


On Mar 29, 7:56 pm, Ken Wesson <kwess...@gmail.com> wrote:
> TeX and LaTeX work that way, too. And in a certain sense so does
> syntax quote in Clojure itself, where the macro expansion source is
> the "content" and unquoted expressions can fill bits in by "escaping
> into logic".

Yes for TeX and LaTeX, but that's obfuscated by the programming model
that makes it necessary (since macros "call" other macros by producing
text with those macros). But IIUC, Clojure uses a traditional
lisp-like quote mechanism, and Scribble is different from that in an
important way. The problem with quasiquotes is that you need to be
aware of the context where code is written -- you need to explicitly
unquote code that is intended to run, and that's hard and error prone.
(My favorite example is an old version of the PLT Scheme pages that
had lots of "<quote>nbsp</quote>"s because 'nbsp was used in a context
that was already quoted.) In contrast, the Scribble reader's approach
is that all scribble forms are, in a sense, "unquoted". This wouldn't
be as convenient for the kind of lispish code generation that you do
with quasiquotes -- roughly speaking, you'd need to quote each head
form that is intended to produce code -- but it's extremely convenient
for dealing with text. (This is actually based on a way of reflecting
logical terms in a theorem prover I worked on...)

In any case, that reader paper describes all the decisions that lead
to this design, and how it works in providing a better facility than
here-strings and string-interpolation, with lots of examples. (And in
case it wasn't clear -- I strongly believe that this kind of syntax is
useful in any language, but much more in lispish languages since they
already enjoy the huge benefits of a uniform representation of
sexprs.)

Jules

unread,
Mar 31, 2011, 8:43:59 AM3/31/11
to clo...@googlegroups.com
I haven't had time to digest this whole thread and this solution is a bit off the wall... but do you really have a requirement to generate Java source ? Would Java bytecode not suffice ?

If you were generating classes/interfaces you could ignore all the issues with Java syntax and just write a clojure macro to generate a definterface/type, then compile that.

If you really wanted source code, you could then run a decompiler on it - I've done this before on a project, that used byte code emission, so that we could demonstrate 100% code coverage :-) - I'm not sure if Clojure generates bytecode that cannot be decompiled into Java - but for interfaces and enums it should not be an issue.

I'm not aware of Clojure support for generating enums, but I suspect that it would be considerably less work and more fun to extend Clojure's byte code emmission code to handle this. Enums have the added benefit that they would be really simple to emit and decompile reliably.

Hope this is not too off the wall !

Jules



B Smith-Mannschott

unread,
Apr 1, 2011, 4:18:33 PM4/1/11
to clo...@googlegroups.com
On Wed, Mar 30, 2011 at 15:00, Stuart Sierra
<the.stua...@gmail.com> wrote:
> Take a look at http://www.stringtemplate.org/ for a Turing-complete,
> purely-functional, Java-based template language designed for code
> generation.

Thanks! This is actually the second time I've run across
stringtemplate, only now i'm taking the time to really dig into it.
I'm making progress and it looks like the right solution for my
immediate needs. I particularly like the effort it makes to enforce a
clean separation of template from business logic.

Since I'm a total beginner with stringtemplate, I'm trying to suss out
where to draw the line between putting complexity into the templates
and/or keeping it in clojure. (i.e. should I have a big template group
with one or two entry points where the templates call eachother), or
should I do composition from the outside via Clojure. I'll start with
the latter, but perhaps I'll move more into the templates as I gain
experience.

// Ben

> -Stuart Sierra
> clojure.com

B Smith-Mannschott

unread,
Apr 1, 2011, 4:23:25 PM4/1/11
to clo...@googlegroups.com
On Thu, Mar 31, 2011 at 05:00, Eli Barzilay <e...@barzilay.org> wrote:
> On Tue, Mar 29, 2011 at 4:43 PM, B Smith-Mannschott wrote:
>> Horrible hack, maybe, but it got me thinking. What you seem to be
>> doing is moving between "code" and "literal" mode by quoting with #.
>> This is a little like traditional quasi-quote...
>>
>> And that got me thinking about Scribble [1] again.  In this
>> context I think of scribble as being sort of an inverse of normal
>> scheme syntax.  In the end, the scribble reader produces the same
>> kind of data structures as the normal scheme reader, but the
>> emphasis is moved from code to textual content. Source is content
>> by default, but you can escape into logic.
>
> This is kind of a good summary of the reader's intended use, beyond
> our documentation system: it's fitting for any kind of code that deals
> with a lot of textual content -- and having a kind of cheap DSLs for
> interacting with other tools by generating source is exactly one of
> the things that you can do with it.  But getting that from the
> scribble documentation will be a little hard -- I wrote a paper on
> just the implementation and use cases of the reader that will be much
> more fitting in this context, with some examples of such uses:
>  http://barzilay.org/misc/scribble-reader.pdf

Thanks very much for the link. I'm enjoying the paper. When I've
addressed my immediate needs (with stringtemplate), I hope to return
to it again in more depth. I have this notion, that an alternate
reader for Clojure in the style of Scribble combined with core
concepts of stringtemplate could make a convenient and yet lispy
templating system for producing non-lisp code and other textual
output.

// Ben

Eli Barzilay

unread,
Apr 1, 2011, 5:23:45 PM4/1/11
to Clojure
On Apr 1, 4:23 pm, B Smith-Mannschott <bsmith.o...@gmail.com> wrote:
> Thanks very much for the link. I'm enjoying the paper. When I've
> addressed my immediate needs (with stringtemplate), I hope to return
> to it again in more depth. I have this notion, that an alternate
> reader for Clojure in the style of Scribble combined with core
> concepts of stringtemplate could make a convenient and yet lispy
> templating system for producing non-lisp code and other textual
> output.

At the time I worked on this, I've looked at a number of systems, and
none were close to being complete in what they were doing. IIRC, I
got the general impression that they were all dealing with HTML
output, and things started to go downhill when people tried to
generalize that. I definitely remember Cheetah being one of these,
where there were some really odd corners around indentation -- an
issue that came up when people tried to use it as a code generator.

OTOH, I made the scribble thing flexible enough that I could even
tackle the odd rules of indenting CPP #-lines. You can see an example
of that in our ffi code:

http://git.racket-lang.org/plt/blob/HEAD:/src/foreign/foreign.c

and the source file that generates that:

http://git.racket-lang.org/plt/blob/HEAD:/src/foreign/foreign.rktc

Look for "(ffi-lib filename no-error?)" in both files, and see how the
output gets indented. See also some of that documented in the text
languague description at:

http://docs.racket-lang.org/scribble/preprocessor.html

(but in retrospect I dislike the overloading of lists as markers for
an indentation group, and will change that in the very near future to
`block'.)

I think that I didn't look at StringTemplate at the time -- maybe it
wasn't too known then. In any case, looking at some examples (eg,
http://www.antlr.org/wiki/display/ST/Five+minute+Introduction), I
don't see anything radically different from the other tools in the
same category... (The main difference is a little too much
buzzword-itis...) So it looks like it's very similar to do these
things with plain scribble. For example, if I translate the
"homepage.st" example from that page to Racket, I'd do this (the first
line specifies using the @-reader, and the `require' line gives me the
`output' function which, roughly speaking, displays its argument(s)):

#lang at-exp racket
(require scribble/text)
(define (hello-world title name friends)
@list{
<html>
<head>
<title>@|title|</title>
</head>
<body>
<p>Hello again, @name !</p>
<p>Greetings to your friends @(add-between friends ", ")</
p>
</body>
</html>
})
(output (hello-world "Welcome To NotStringTemplate"
"World"
'("Ter" "Kunle" "Micheal" "Marq")))

Note that the @|...| thing is needed when the racket expression
(`title' in the above) needs to be separated from the text around it.
The thing that makes this attractive to me is that there is no need to
talk about "attributes", "properties", etc -- it's *just* code. If
you have some objects then things work fine too, of course, but you
can mix things in way you want. For example, I can easily change the
above with dropping the `title' argument and instead define a single
global value for it -- so the choice of using an argument or a fixed
global is something that is up to the code.

But there is an important point that they're talking about there --
the model-view separation -- which (I think) is coming from the
HTML-generating heritage of these systems, where you want some people
to do the HTML design, and other people to do the hacking. (Seems
that "logic" is the canonical term for "programming" in these
contexts...) Racket has an `include' thing that can basically read
some source code file into any context, and the `scribble/text' module
uses that and provides an `include/text' which is the same thing, only
using the @-reader to read the file. Think about `include' as a kind
of a macro that `read's some random file to produce its output, and
`include/text' does the same with the extended readtable. So to get
this separation, take the textual contents of the `hello-world'
function and dump it in a "homepage.st" file (unindented, of course),
then change the above code to be:

#lang at-exp racket
(require scribble/text)
(define (hello-world title name friends)
(include/text "homepage.st"))
(output ...same as above...)

You can see a number of similar examples in the templates chapter of
our web-server manual, which uses the same thing.

B Smith-Mannschott

unread,
Apr 3, 2011, 3:45:00 PM4/3/11
to clo...@googlegroups.com
On Fri, Apr 1, 2011 at 22:18, B Smith-Mannschott <bsmit...@gmail.com> wrote:
> On Wed, Mar 30, 2011 at 15:00, Stuart Sierra
> <the.stua...@gmail.com> wrote:
>> Take a look at http://www.stringtemplate.org/ for a Turing-complete,
>> purely-functional, Java-based template language designed for code
>> generation.
>

Well, I've spent the weekend wrangling with StringTemplate (ST) and
though I'd share my experiences so far. I'm sorry; this is a bit
rambling.

I chose to jump straight to ST 4, which came out at the end of March,
figuring there was no point in learning 3.x since 4.0 seems to be
where active development will go in the future.

This was a mixed blessing because the practical examples available on
the net and the majority of the documentation is actually still for
3.x, which is not completely compatible with 4.0.

Also, I found ST's behavior with respect to syntax errors in templates
to be less than helpful. Basically it'll throw a NullPointerException,
or maybe an IllegalArgumentException or just puke some stuff on the
console (generally hidden behind my emacs window). This is rough for a
learner.

Also, I'm generating java source code which includes uses of generics.
This interacts hideously with ST's default of using < and > to delimit
a template expression and << and >> to delimit the beginning and end
of a template definition. This can be customized, but the
customization is not all I hoped for.

At first I tried to customize to « and » but this breaks the ability
to include comments in the string template group file. (Normally
comments are <! ... !>, presumably they would be «! ... !» when one
has customized the delimiter characters like I had. But, the fact is
neither syntax seemed to work. This cost me *hours* because while the
console printed out a huge number of error messages, they were always
the same error messages printed in such a way that the console never
seemed to scroll, so I thought I was looking at old output and had
fixed the problem (by changing <! to «!). The parse errors caused the
non-comment comment caused all kinds of strange behavior in later
templates, such as crashes. I found, for example, that:

someTemplate(a,b) ::= « ... »
«list1,list2:someTemplate()»

which is roughly analogous to:

(defn someTemplate [a b] ...)
(map someTemplate list1 list2)

Would crash complaining about not having definitions for i0 and i
(these variables are something ST defines implicitly for iteration. I
don't know the details.)

I found I could fix that by changing the template definition to
include i0 and i arguments, though I never used them in the template.
This just seemed demented. It was at that point that I went to bed.

This morning, I finally understood the comment problem and found that
if I changed the delimiters to $ and $ and the comments to $! and !$
all was once again well with the world. It's annoying though that
opening $ and closing $ are the same character, which doesn't aid
readability.

So, I've made some progress in my understanding of ST. Now that I sit
down and try to re-implement my existing code emission using ST, I
find a design question rearing its head. I'm not sure how much
complexity to push into the templates and how much to leave in
Clojure.

I could have a large number of very simple templates and use Clojure
to drive and assemble them. Alternatively, I could have a single
"entry point" to a large set of templates which call each other
internally. That means using Clojure to construct a data model (a map
of lists of maps of ...) suitable for consumption by ST.

First I through I'd take the first alternative, but then I decided to
try the second one first, though now it's looking like I will go back
to doing more in clojure and less in ST. I already have a perfectly
serviceable model but it seems increasingly like I'd have to tear it
apart and build it up completely differently to get it into a form
that ST can consume in a reasonable fashion.

Here's a simplified fragment of my model:

{ :class-name "SomeEnum"
:id-field :id
:records [{:id "A", :payload 3},
{:id "B", :payload 4}] }
produces:

enum SomeEnum {
A(3),
B(4);
int payload;
SomeEnum(int payload) {
this.payload = payload;
}
}

- I'm using keywords, but StringTemplate can't tolerate those as keys
- ST's variable's can't contain -, so I can't just convert the keyword keys
to strings, I have to rename them too.
- the type of :payload is not stored explicitly anywhere, but it's a real
java Integer in the model, so I just use (type) to figure out what it is.
- I have an function (unbox) which gives me the unboxed equivalent
where appropriate,
otherwise identity. (ST provides Maps, as part of its template
language which could
almost do this for me, but not quite.)
- I have logic to generate "java literal" versions of various values:
i.e. 1.0M -> BigDecimal.ONE; 11.0M -> new BigDecimal("11.0")
(ST supports ModelRenderers for this, I think).

It's looking more and more like using ST is going to involve as much
Clojure code as not using ST -- and that's not counting the additional
template code. This doesn't feel like a clear win.

I'm still very much an ST beginner, but I feel that I struggling
against an impedance mismatch of a sort. Maybe it's because ST is
designed to feed off an OO model, and I've got one that's functional
(all immutable data really).

Indeed, I ran across this just this evening:

http://steve.vinoski.net/pdf/IC-Clojure_Templating_Libraries_Fleet_and_Enlive.pdf

The author seems to have found the same mismatch between Clojure and
ST that I have, though he does a better job of explaining it. One of
the two alternatives he recommends https://github.com/Flamefork/fleet
seems like it could fit to what I'm trying to accomplish. Still, I'd
like to first finish my attempt with ST before I decide wether to
stick with it or not.

// ben

B Smith-Mannschott

unread,
Apr 4, 2011, 1:10:59 PM4/4/11
to clo...@googlegroups.com
The issue of Multi-threading has cropped up WRT templating. It's not
clear from the StringTemplate documentation what is and what isn't
thread safe. From the look of the API, though I wouldn't be on
anything.

Fleet, mentioned in the previous message seems to have a real problem
here too since fleet templates /have no parameters/ it seems they
communicate with the model by calling functions defined in some
namespace.

My code generator is multi-threaded, i.e. creates a bunch of thunks
for the classes to be generated and then runs these with pmap. This
works great when you whole "templating system", such as it is, is
build on Clojure's persistent data structures and pure functions. This
becomes a problem when trying to offload the templating work onto
StringTemplate, a Java library that's not obviously given much though
to the issue. Fleet could be made to work with creative use of
(binding [...]), but I hesitate to go there because an earlier version
of my code generator did this and I was happy when I'd removed that
wart and replaced it with explicit arguments.

// Ben

Ilia Ablamonov

unread,
Apr 17, 2011, 3:21:11 AM4/17/11
to clo...@googlegroups.com
Hello!

Fleet have parameters for sure :)

Creating template from string:
(fleet [post] "<p><(post :body)></p>")
returns function of single argument post.
(fleet [post, comments] "<p><(post :body)> <(comments :count)></p>")
returns function of two arguments post, comments.

Creating multiple templates from dir:
(fleet-ns 'tpl "some/template/dir")
having that some/template/dir contain post.html.fleet
will produce 3 similar functions tpl/post, tpl/post-html, tpl/post-html-fleet
that has 2 arguments: post, data where data arg by default = post arg and both are by default = nil

Latter case is very flexible and convenient: you can treat template post as:
- function of one argument post - for usage like rails partials
- function of one argument data - for general usage
- function of two arguments: main arg post and 'additional' argument data - to combine both

Producing 3 functions from one file allow you to have/use tpl/post-html and tpl/post-json templates
and use simple tpl/post if only one really present. tpl/post-html-fleet there is for additional disambiguation.
Reply all
Reply to author
Forward
0 new messages