[erlang-questions] [ANN] Neotoma 1.5

[Sean Cribbs]

It's been a year coming, so I'm proud to announce a new version of
Neotoma, the PEG/Packrat toolkit and parser-generator. The primary
feature of version 1.5 is that it uses significantly less memory and
also supports UTF8 (via binaries), thanks to some awesome work by
Graeme Defty, who has also converted Reia to use a PEG.

You can get the latest source on Github: https://github.com/seancribbs/neotoma

Neotoma is rebar-friendly as well, so adding it to a rebar project is
a simple deps declaration like so:

{neotoma, "1.5", {git, "git://github.com/seancribbs/neotoma", {tag, "1.5"}}}

Hope to see some of you at Erlang Factory SF next week.

Cheers,

Sean Cribbs

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questio...@erlang.org

[Tony Arcieri]

Any plans for further performance improvements in the roadmap? :)

On Sun, Mar 13, 2011 at 4:25 PM, Sean Cribbs <seanc...@gmail.com> wrote:

> It's been a year coming, so I'm proud to announce a new version of
> Neotoma, the PEG/Packrat toolkit and parser-generator. The primary
> feature of version 1.5 is that it uses significantly less memory and
> also supports UTF8 (via binaries), thanks to some awesome work by
> Graeme Defty, who has also converted Reia to use a PEG.
>
> You can get the latest source on Github:
> https://github.com/seancribbs/neotoma
>
> Neotoma is rebar-friendly as well, so adding it to a rebar project is
> a simple deps declaration like so:
>
> {neotoma, "1.5", {git, "git://github.com/seancribbs/neotoma", {tag,
> "1.5"}}}
>
> Hope to see some of you at Erlang Factory SF next week.
>
> Cheers,
>
> Sean Cribbs
>

--
Tony Arcieri
Medioh! Kudelski

[Dmitrii Dimandt]

Awesome!

Is reia grammar available? Could make a great place to start learning from

[Sean Cribbs]

One thing I neglected to emphasize in the announcement was that where
Neotoma's parsing functions would return lists or even single
characters before, they now return binaries. Some grammars will
require drastic changes to accommodate this difference. My apologies
to those who tried the new version and had difficulties (e.g. Cliff).

Regarding Tony's question about future performance improvements, I
think the next one is obvious: Although switching to binaries helped
reduce copying, it can be reduced further by refusing to memoize the
input in its various states of consumption, but simply to scan and
split the single input binary when fetching a memoization (h/t Ben
Black). This will require adding another counter to the "Index" tuple
that keeps the raw byte-offset. I also want to investigate using the
process dictionary a bit more instead of ETS, or at least get a
comparison metric of the two approaches.

I've also been thinking about what should happen for Neotoma 2.0,
although as you can see by the release schedule over the last year, it
has not been high enough on my priorities. Nevertheless, here's my
wishlist:

1) Generate an AST from the metagrammar/grammar parser instead of raw
code, decoupling parsing from code generation.
2) Create some EQC generators that can use the AST of your grammar to
provide automated parser tests.
3) Allow code generation into multiple languages from the AST. One
could generate Erlang, LFE, Reia, Efene, or any language for that
matter. I'm thinking this is what I REALLY wanted to do when I
experimented with parse_transform early on.
4) Let the grammar designer select whether to use binaries or lists as
output of parsing functions.
5) Create analysis tools for the grammar AST that can discover
problems and suggest optimizations.
6) Create a "regular expression" terminal in the metagrammar that can
more succinctly capture strings which currently require combinations
of string literals, character classes and repetition operators.

Thanks for using Neotoma, and as always I'll gladly accept patches to
implement any of the above.

Cheers,

Sean

[Torben Hoffmann]

Hi Sean,

Perhaps 1.5 should have been 2.0?

See http://semver.org/ where rule 9 states that:

Major version X (X.y.z | X > 0) MUST be incremented if any backwards
> incompatible changes are introduced to the public API. It MAY include minor
> and patch level changes.
>

Just a thought.

Cheers,
Torben

--
http://www.linkedin.com/in/torbenhoffmann

[Tony Arcieri]

The Reia grammar is available here:

https://github.com/tarcieri/reia/blob/peg/src/compiler/reia_parse.peg

[Dmitrii Dimandt]

Thank you!

[Sean Cribbs]

For those who aren't in like with the new binaries-by-default feature,
I've branched off 1.4 and tagged 1.4.1 on github. The tag is after
the rebar-izing of Neotoma, so it should still work with as a dep with
your rebar-based projects.

Cheers,

Sean