PEG: a relatively new word in parsing world

[Phil Pirozhkov]

When it comes to any web framework, it comes to templates.
You know many, i am sure. But how do they work?
Does anyone analyze them each time a new page request occurs?
No. They are parsed once and this parsed structure can be used to
create an HTML page (sometimes other formats, too).

So how does this look like? What's the source, what's the product?
Imagine we just want to display a user name on the screen:

{if user_logged?} {user_name} {else} Not logged in {end}

What kind of structure is easy to work with to build a page with
given variable bindings?
Let's assume that some kind of a tree structure will be ok, so
that we can have nested expressions (if's inside other if's etc).

Let's begin with the most simple:
{user_name}
This tells us we should take some variable caller 'user' from
bindings,
get its value and write the result to the page we are rendering.
Reia tuple like (:id, :user_name) will fit okay for that.

To come quickly to the main part, i will write down the parsed
structure
for the whole if immediately:
[ (:if, [((:id, :user_logged), (:id, :user_name))], (:text, "Not
logged in")) ]

Looks fairly simple, isn't it? And it can be used to render a page
using
this file http://github.com/pirj/ryan/blob/ea7101af7436f403708b34fe24dfc2e63ff83373/src/retem/retem.re

This all look very fancy, but how do we get that machine-readable
tree-structure out of the human readable and writable template?
How do we describe how our template language should look like?

After looking into the files below, please don't run away! there's a
cure few lines further.
We have a lexical analyzer written like this:
http://github.com/pirj/ryan/blob/ea7101af7436f403708b34fe24dfc2e63ff83373/src/retem/retem_scan.xrl
And a parser described like that:
http://github.com/pirj/ryan/blob/ea7101af7436f403708b34fe24dfc2e63ff83373/src/retem/retem_parse.yrl

What a mess! Yes, that was me mostly, i'm sure one can describe it
better, but not twice better.

What does it do? Does it describe all the complex features that are in
specification already?
http://github.com/pirj/ryan/tree/ea7101af7436f403708b34fe24dfc2e63ff83373/src/retem
No.
It only can take a value out of the Map, resolve If statement without
Else, and understand
a For loop.

To make all other features work, one will need a year and a year after
that to relax.

Recently Sean Cribbs came up with an implementation of a different
language descriptor,
suggested by Bryan Ford few years ago (the lexer and parser above are
almost 50y.o.),
Neotoma. It's good, because
- the syntax (Parsing Expression Grammars / PEG) is shorter and more
readable
- it works faster
- it disallows ambiguities
It can consume more memory than ordinary lexer/parser pair. But
immediately after parsing,
the memory is free'd out, isn't it?

Please take a look at http://github.com/pirj/ryan/blob/5a7ddfab71a3b32d54abebd8cb0ad6d12f002bed/src/retem/retem.peg
It describes the same as the .xrl and .yrl files, but with no erlang
skills required.

Neotoma can make this syntax even simplier, but Sean have a different
sight on that, and
I'm currently working on my own branch to make web templating engine
development
as easy as possible.

Cheers, Phil

[Tony]

Neotoma looks awesome.  I haven't had the opportunity to play around with it much, but I've run into problems implementing Reia which a PEG would solve quite gracefully, specifically interpolated strings.  Robert Virding told me that Joe Armstrong did a proof-of-concept PEG for Erlang which I was really hoping he would release, but that never happened and I'm quite glad Neotoma materialized.

I would like to rewrite the Reia grammar with Neotoma but I've been pretty strapped for time for the past month or so.  Glad to see you're investigating it for Ryan.

--
Tony Arcieri
medioh.com