Jiffle language

[Michael Bedward]

Hello all,

I've started work on re-writing the Jiffle compiler to use ANTLR
version 4 (http://www.antlr.org/). The main aims are:

- Reduce the current ANTLR grammars to a single, simpler, language-neutral file.

- Move all compilation steps that are presently coded as embedded
actions in ANTLR v3 grammars into Java classes.

- Take advantage of the visitor / listener design approach of ANTLR v4
to make the compiler more modular, and therefore easier to maintain
and extend.

- Make the Jiffle dependencies cleaner.

The plan is that all existing Jiffle scripts should work with the new
compiler, so I'll be pestering Moovida to help with testing at some
stage.

Michael

[Simone Giannecchini]

Ciao Michael,
good timing, we have project where we need to build a prototype of a
Process that does (relatively limited) raster algebra.
I was thinking about relying on Jiffle, so my question is what is the
timing of this development? Can we help somehow?

Regards,
Simone Giannecchini
==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.
==

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

> --
> You received this message because you are subscribed to the Google Groups "jai-tools" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jai-tools+...@googlegroups.com.
> To post to this group, send email to jai-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/jai-tools.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Michael Bedward]

Hi Simone,

Great - it's always nice to hear about another use for Jiffle !

Depending on your needs, you might be fine with the existing version
which has been well tested and used for some time as the raster
calculator for JGrassTools. Have a look at the user guide
(jaitools.org) and the demo programs in the distribution to see what
it can do currently.

I've been wanting to move Jiffle over to ANTLR v4 for some time
because it will make the code for the compiler much cleaner and easier
to extend. I wasn't planning any changes to Jiffle's syntax in this
round - but if you had requirements or ideas for new language features
this would be a good time to discuss them. I've got some ideas for
down the track (e.g. specified iteration order).

There is also the Jiffle image op. It's just a skeleton at the moment
and has never been used for anything, but if your work would require
using Jiffle in a JAI chain then you might like to take a look at it.
However, for a Process I'd imagine you would work directly with the
Jiffle compiler (e.g. JiffleBuilder class).

Regarding timing - hard to say... It depends on how fast I can get my
head around the finer points of ANTLR v4 (and un-learn ANTLR v3). I'm
aiming for late next month at the moment. Regarding help... once the
back-end is moved over to ANTLR v4 the code will be much easier to
work with and I'd be more than happy to share the hacking with you.
Alternatively, if you use the current version but need changes to the
API (e.g. other entry points) then pull requests would be the way to
go.

Michael


On 19 July 2013 05:02, Simone Giannecchini

[andrea antonello]

Hi Michael,
great to hear you will do the port!

Our community uses the mapcalculator quite often, therefore I can sure
help with the testing.

Thanks,
Andrea



On Fri, Jul 19, 2013 at 3:16 AM, Michael Bedward

[Michael Bedward]

On 19 July 2013 15:17, andrea antonello <andrea.a...@gmail.com> wrote:
> Our community uses the mapcalculator quite often, therefore I can sure
> help with the testing.
>

Have no fear - I'll be onto you to try and break it (again)

Michael

[Michael Bedward]

Here is an issue and a new feature branch for the port to ANTLR v4...

https://github.com/mbedward/jiffle/issues/29
https://github.com/mbedward/jiffle/tree/antlr4

For starters the branch just has all of the Java sources from the
ANTLR v3 version on master, but the old grammar files have been
replaced by a new, single grammar...

https://github.com/mbedward/jiffle/blob/antlr4/language/src/main/antlr4/org/jaitools/jiffle/parser/Jiffle.g4

This is already quite nice because we have a single, readable grammar
describing the language in a bit over 300 lines as opposed to the
previous multiple files with mixed grammar and Java snippets that
totalled over 1800 lines.

Next I'll start progressively replacing the support classes that
worked with the ANTLR 3 chain with new classes based on ANTLR 4's
visitor / listener scheme. It will be a while it's possible to do a
build, although it should be possible to test the validity of the
grammar against existing Jiffle scripts pretty soon.

Michael

[Michael Bedward]

Making progress on the port. Current work is going into the antlr4
branch. It doesn't build yet but feel free to look at the code if
you're interested.

At the moment I'm working on the translation from Jiffle to Java
source. I'll then work backwards from this and look at the validation
and optimizing bits.

Previously, Jiffle used the stringtemplate library to emit the Java
sources. Unfortunately, the latest version of stringtemplate depends
on yet another ANTLR version (3.3) which would put us back into the
confusing state of depending on multiple ANTKRs. So instead of
stringtemplate, I'm writing the source emitting logic into Jiffle
itself. That seems to be working ok.

Michael

[Jody Garnett]

What kind of stuff does String Template do? I always found message format and friends to be fairly useful on their own?

-- 

Jody Garnett

[Michael Bedward]

On 25 July 2013 16:02, Jody Garnett <jody.g...@gmail.com> wrote:
> What kind of stuff does String Template do? I always found message format
> and friends to be fairly useful on their own?
>

It's a small but very useful text templating engine...
http://www.stringtemplate.org/

With ANTLR v3 there was the option of having a template file linked to
a grammar. This was convenient for Jiffle when generating Java sources
from the final AST. However, that style of working has disappeared in
v4. It would still be handy but it's not a big deal to work without
it.

I got some advice from Ter (Mr Antlr) today about a general approach
for the new Jiffle, working off a parse tree for validation / checking
and then labelling the nodes with objects that serve the same role as
nodes in the intermediate ASTs did previously. These objects will be
instances of classes which know how to render themselves as Java
snippets. A final parse tree walker will put the snippets together
into the source for a class which is then fed to Janino.

Despite it being a bit of a learn-as-you-go exercise for me, I'm
pleased with the progress so far and hope to have a snapshot for
Moovida to torture test next week.

Michael

[Michael Bedward]

A brief update...

I got sidetracked by other things this week but still made a bit of
progress with Jiffle. Most of the script syntax and type checking code
has been moved over to ANTLR v4. The next step is to generate Java
sources from a script. With this, I'm going to try to use as much of
the existing plumbing as possible (ie. the stuff that does not
directly depend on ANTLR v3 and StringTemplate) with the aim of having
a working snapshot available soon-ish.

Michael

[Michael Bedward]

Hello all,

Still beavering away on Jiffle / ANTLR4. It's turned into a much
bigger job than I originally anticipated (as always happens) but is
still going very well. More new code has been committed to the antlr4
branch this week.

I spent some time trying to convert the existing ANTLR3 /
StringTemplate based system that emits Java sources into pure Java
string formatting, but it was too tedious to bear :) So I've decided
instead to use FreeMarker which is relatively lightweight and does not
bring in any further dependencies. I hope this will not cause any
problems for users (Jody: will FreeMarker pass your eclipse hygiene
filter ?)

As I work though the port I'm compiling a wish-list of tweaks and
additions to the Jiffle language itself. If anyone has their own ideas
for such changes let me know because now would be an ideal time to
consider them.

Michael

[Jody Garnett]

Lets check: http://freemarker.org

FreeMarker is Free, with BSD-style license. See the license here...

That sounds good, no idea about their contribution policies. Still let me check to see if it has ever been reviewed before.

- Freemarker 2.3.13 (excluding some nested jar files) - odd it lists it as Apache Software License 1.2 + CDDL

- Freemarker 2.3.16 (excludes some nested jars) mix of BSD / Apache 1.1 / CDDL

- Freemarker 2.3.18 listed as a "works with", but correctly says BSD / Apache 1.1 / Apache 2 / CDDL 

(The CDDL is for some sun XSD files)

So yeah it looks like it can go, what version of Freemarker is the latest?

-- 

Jody Garnett

[Michael Bedward]

Hi Jody,

> So yeah it looks like it can go, what version of Freemarker is the latest?
>

Version 2.3.20 is the one that I'm playing with at the moment - it
seems to be the most recent one at maven central.

Michael

[Jody Garnett]

udpate - I have found that stringtemplate 3.2.1 (ie the exact version you are using) was reviewed and approved for use in another project.

It looks like it is *just* the antlr 2.7.x series that is messed up - hopefully the above is good news and makes things easier for you?

Michael

[Michael Bedward]

On 13 August 2013 08:29, Jody Garnett <jody.g...@gmail.com> wrote:
> udpate - I have found that stringtemplate 3.2.1 (ie the exact version you
> are using) was reviewed and approved for use in another project.
> It looks like it is *just* the antlr 2.7.x series that is messed up -
> hopefully the above is good news and makes things easier for you?
>

Ah grasshopper - all reality is circle which we are destined to go
round and round and round...

Here is my understanding...

1) If we have Jiffle use ST 3.2.1 we end up with ANTLR 2.7.7 as a
transitive dependency and there is no way to avoid it.

2) If we use ST 4.0.x we get ANTLR 3.5 as a transitive dependency.

3) If we use FreeMarker you obviously ending up with two ANTLR
versions plus it has no other dependencies.

Our anchor in all this is that I'm definitely going to use ANTLR 4 -
it makes the Jiffle grammars and compiler code so much cleaner. That,
in turn, means that (1) is obsolete so we can forget about it. Out of
(2) and (3) I think my preference is (2). The reason for that is that
ST seems a bit easier than FreeMarker for turning the intermediate
representation of a Jiffle program (annotated parse tree) into the
final Java sources.

Michael

[Michael Bedward]

> 3) If we use FreeMarker you obviously ending up with two ANTLR
> versions plus it has no other dependencies.

Sorry - that was meant to read "you obviously avoid ending up..."