I'm using Clojure to generate java source (enums, specifically) and am
looking for ideas.
Currently my generator consists of a collection of function which call
each other tree-recursion-like.
Input:
Each function takes a map of input. Much of this is constant for any
one run of the generator. Only a few of the leaf functions don't
follow this convention and instead ask for exactly what they need. I
hit on this solution as a way to avoid a lot of global state while at
the same time not causing rippling changes in parameter lists every
time some leaf procedure needs some extra bit of information.
Output:
Instead of printing directly to *out* or returning a String, I have
all the functions return a sequence of values. The result of a run of
the code generator is then initially a tree of
things-that-can-be-made-into-strings, which I flatten and concatenate
at the very end. Strings concatenate as themselves. Symbols and
keywords as their (name). Class objects print as their fully qualified
class name. BigDecimals are formatted as appropriate constructor
calls. By not printing eagerly, I can revise the result of called
procdures, e.g. by using interpose to insert commas between a sequence
of fragments produce by another function or for comprehension.
A small example:
(defn getter
[method field type]
["
public " (unbox type) " " (name method) "() {
return " (name field) ";
}\n"])
Another:
(defn constructor
"Defines the enum's constructor. The constructor uses the arguments
it is given to initialize all fields declared by (fields). Each (member)
is a call to this constructor."
[{:keys [class-name fields] :as cfg}]
[" " class-name "(" (formal-params cfg) ") {\n"
(statement-list
(for [f fields]
[" this." f " = " f ]))
" }\n"])
Where:
(defn statement-list [list]
(concat (interpose ";\n" list) [";\n"]))
The templating aspect of all this is solved with constructs Clojure
already gives me (vectors and plain old clojure expressions), which is
kind of nice. It's also reasonably friendly for exploratory testing
and interactive development. Still, I'm not really happy with it.
This works, OK, but it feels pretty hokey. Indentation is totally
ad-hoc. I mean, it works, but I can't help but think that there must
be a better way.
I've had a few false starts trying to improve on this idea and am
looking for a new approach.
Currently I think I'd like a way to write/construct a *simple*
parse-tree for a java subset as Clojure data along with some way to
serialize this as syntactically correct Java. That sounds like a lot
of work, maybe there's an easier way that I'm overlooking?
// Ben
I can see the advantages of going this route for simple templates that
contain no optional elements and no repetition, but that's not going
to get me far. I still don't understand how these kinds of issues:
(defn constructor
[{:keys [class-name fields] :as cfg}]
[" " class-name "(" (formal-params cfg) ") {\n"
(statement-list
(for [f fields]
[" this." f " = " f ]))
" }\n"])
... are handled in the usual templating solutions without needing a
turing complete templating language. But, at that point, I'm not sure
what I've gained, as I already have a perfectly servicable programming
language in Clojure. How does one really separate "content" from
"presentation" in such a case?
// Ben
> Incidently, I've also used a similar technique to mark up SVG to
> generate multiple images that are then built into movies - it works
> like a charm considering how low tech it is.
>
>
> Saul
>
> [1] http://clojure.github.com/clojure-contrib/strint-api.html
> [2] http://www.javaworld.com/javaworld/jw-12-2001/jw-1228-velocity.html
> [3] http://weblogs.java.net/blog/aberrant/archive/2010/05/25/using-stringtemplate-part-1-introduction-stringtemplate
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
Ah, thanks for example. One question though:
enum Var_name { #for $var in $variables.keys() # $var , #end for # VAR_END };
This seems almost like cheating because having VAR_END means we don't
have to deal with every item, but the last is followed by a comma
problem. Does velocity provide a way to address this easily?
// Ben
Thanks Meikel,
I'm currently looking at enlive to see if I can understand how it
works and perhaps adapt its approach to my needs. At first blush, it
looks technically interesting. :)
// ben
Horrible hack, maybe, but it got me thinking. What you seem to be
doing is moving between "code" and "literal" mode by quoting with #.
This is a little like traditional quasi-quote...
And that got me thinking about Scribble [1] again. In this context I
think of scribble as being sort of an inverse of normal scheme syntax.
In the end, the scribble reader produces the same kind of data
structures as the normal scheme reader, but the emphasis is moved from
code to textual content. Source is content by default, but you can
escape into logic.
That's what your little hack made me think of. The text embedding in
the string literal is content by default, but # # allows one to escape
into "code mode".
Perhaps a Scribble's approach could be useful for a templating system?
I'm not sure. I haven't really grokked it yet.
[1] http://download.plt-scheme.org/doc/html/scribble/index.html
// Ben
> The general idea might be applicable to what you want.
>
> Saul
>
I've looked a little at scriptjure. I could see writing a tree walker
for some subset of pseudo-clojure that spits out java code, though it
sounds like a lot of work. I think I'll save that up in case I don't
find another satisfactory approach.
> I also wonder if there is a way to do what you are trying to do
> without emitting text.
> (Possibly by extending the java.lang.Enum class? No idea...)
I thought of that, but didn't pursue the idea because it's useful to
my colleagues to have access to readable source code of the enum
classes. This is particularly relevant because most of these enums
carry not just their identity (i.e. their name()), but also additional
metadata immutable data accessible through final fields on the enum
instances of any given type.
It's useful to have source to look at to see what those values are,
particularly during the refactoring when we're replacing our previous
approach: value objects named uniquely by a pair of strings (the first
being analogous to a class), containing additional data, and loaded
from a database on startup. This turns out to be a unit-testing
nightmare. Unit testing was not considered in the original design of
the system. Replacing these objects with enums will help here.
Also, I was reluctant to go the route of using Clojure to try to roll
my own Enums because I didn't want to inadvertently pull in runtime
dependencies on Clojure. Also, I'd want to imitate what the Java
compiler does so my Enums work just like the "real thing". A potential
source of error.
// Ben
TeX and LaTeX work that way, too. And in a certain sense so does
syntax quote in Clojure itself, where the macro expansion source is
the "content" and unquoted expressions can fill bits in by "escaping
into logic".
Thanks! This is actually the second time I've run across
stringtemplate, only now i'm taking the time to really dig into it.
I'm making progress and it looks like the right solution for my
immediate needs. I particularly like the effort it makes to enforce a
clean separation of template from business logic.
Since I'm a total beginner with stringtemplate, I'm trying to suss out
where to draw the line between putting complexity into the templates
and/or keeping it in clojure. (i.e. should I have a big template group
with one or two entry points where the templates call eachother), or
should I do composition from the outside via Clojure. I'll start with
the latter, but perhaps I'll move more into the templates as I gain
experience.
// Ben
> -Stuart Sierra
> clojure.com
Thanks very much for the link. I'm enjoying the paper. When I've
addressed my immediate needs (with stringtemplate), I hope to return
to it again in more depth. I have this notion, that an alternate
reader for Clojure in the style of Scribble combined with core
concepts of stringtemplate could make a convenient and yet lispy
templating system for producing non-lisp code and other textual
output.
// Ben
Well, I've spent the weekend wrangling with StringTemplate (ST) and
though I'd share my experiences so far. I'm sorry; this is a bit
rambling.
I chose to jump straight to ST 4, which came out at the end of March,
figuring there was no point in learning 3.x since 4.0 seems to be
where active development will go in the future.
This was a mixed blessing because the practical examples available on
the net and the majority of the documentation is actually still for
3.x, which is not completely compatible with 4.0.
Also, I found ST's behavior with respect to syntax errors in templates
to be less than helpful. Basically it'll throw a NullPointerException,
or maybe an IllegalArgumentException or just puke some stuff on the
console (generally hidden behind my emacs window). This is rough for a
learner.
Also, I'm generating java source code which includes uses of generics.
This interacts hideously with ST's default of using < and > to delimit
a template expression and << and >> to delimit the beginning and end
of a template definition. This can be customized, but the
customization is not all I hoped for.
At first I tried to customize to « and » but this breaks the ability
to include comments in the string template group file. (Normally
comments are <! ... !>, presumably they would be «! ... !» when one
has customized the delimiter characters like I had. But, the fact is
neither syntax seemed to work. This cost me *hours* because while the
console printed out a huge number of error messages, they were always
the same error messages printed in such a way that the console never
seemed to scroll, so I thought I was looking at old output and had
fixed the problem (by changing <! to «!). The parse errors caused the
non-comment comment caused all kinds of strange behavior in later
templates, such as crashes. I found, for example, that:
someTemplate(a,b) ::= « ... »
«list1,list2:someTemplate()»
which is roughly analogous to:
(defn someTemplate [a b] ...)
(map someTemplate list1 list2)
Would crash complaining about not having definitions for i0 and i
(these variables are something ST defines implicitly for iteration. I
don't know the details.)
I found I could fix that by changing the template definition to
include i0 and i arguments, though I never used them in the template.
This just seemed demented. It was at that point that I went to bed.
This morning, I finally understood the comment problem and found that
if I changed the delimiters to $ and $ and the comments to $! and !$
all was once again well with the world. It's annoying though that
opening $ and closing $ are the same character, which doesn't aid
readability.
So, I've made some progress in my understanding of ST. Now that I sit
down and try to re-implement my existing code emission using ST, I
find a design question rearing its head. I'm not sure how much
complexity to push into the templates and how much to leave in
Clojure.
I could have a large number of very simple templates and use Clojure
to drive and assemble them. Alternatively, I could have a single
"entry point" to a large set of templates which call each other
internally. That means using Clojure to construct a data model (a map
of lists of maps of ...) suitable for consumption by ST.
First I through I'd take the first alternative, but then I decided to
try the second one first, though now it's looking like I will go back
to doing more in clojure and less in ST. I already have a perfectly
serviceable model but it seems increasingly like I'd have to tear it
apart and build it up completely differently to get it into a form
that ST can consume in a reasonable fashion.
Here's a simplified fragment of my model:
{ :class-name "SomeEnum"
:id-field :id
:records [{:id "A", :payload 3},
{:id "B", :payload 4}] }
produces:
enum SomeEnum {
A(3),
B(4);
int payload;
SomeEnum(int payload) {
this.payload = payload;
}
}
- I'm using keywords, but StringTemplate can't tolerate those as keys
- ST's variable's can't contain -, so I can't just convert the keyword keys
to strings, I have to rename them too.
- the type of :payload is not stored explicitly anywhere, but it's a real
java Integer in the model, so I just use (type) to figure out what it is.
- I have an function (unbox) which gives me the unboxed equivalent
where appropriate,
otherwise identity. (ST provides Maps, as part of its template
language which could
almost do this for me, but not quite.)
- I have logic to generate "java literal" versions of various values:
i.e. 1.0M -> BigDecimal.ONE; 11.0M -> new BigDecimal("11.0")
(ST supports ModelRenderers for this, I think).
It's looking more and more like using ST is going to involve as much
Clojure code as not using ST -- and that's not counting the additional
template code. This doesn't feel like a clear win.
I'm still very much an ST beginner, but I feel that I struggling
against an impedance mismatch of a sort. Maybe it's because ST is
designed to feed off an OO model, and I've got one that's functional
(all immutable data really).
Indeed, I ran across this just this evening:
http://steve.vinoski.net/pdf/IC-Clojure_Templating_Libraries_Fleet_and_Enlive.pdf
The author seems to have found the same mismatch between Clojure and
ST that I have, though he does a better job of explaining it. One of
the two alternatives he recommends https://github.com/Flamefork/fleet
seems like it could fit to what I'm trying to accomplish. Still, I'd
like to first finish my attempt with ST before I decide wether to
stick with it or not.
// ben
Fleet, mentioned in the previous message seems to have a real problem
here too since fleet templates /have no parameters/ it seems they
communicate with the model by calling functions defined in some
namespace.
My code generator is multi-threaded, i.e. creates a bunch of thunks
for the classes to be generated and then runs these with pmap. This
works great when you whole "templating system", such as it is, is
build on Clojure's persistent data structures and pure functions. This
becomes a problem when trying to offload the templating work onto
StringTemplate, a Java library that's not obviously given much though
to the issue. Fleet could be made to work with creative use of
(binding [...]), but I hesitate to go there because an earlier version
of my code generator did this and I was happy when I'd removed that
wart and replaced it with explicit arguments.
// Ben