[erlang-questions] [ANN] Neotoma 1.5

20 views
Skip to first unread message

Sean Cribbs

unread,
Mar 13, 2011, 6:25:14 PM3/13/11
to Erlang Questions, neoto...@googlegroups.com
It's been a year coming, so I'm proud to announce a new version of
Neotoma, the PEG/Packrat toolkit and parser-generator. The primary
feature of version 1.5 is that it uses significantly less memory and
also supports UTF8 (via binaries), thanks to some awesome work by
Graeme Defty, who has also converted Reia to use a PEG.

You can get the latest source on Github: https://github.com/seancribbs/neotoma

Neotoma is rebar-friendly as well, so adding it to a rebar project is
a simple deps declaration like so:

{neotoma, "1.5", {git, "git://github.com/seancribbs/neotoma", {tag, "1.5"}}}

Hope to see some of you at Erlang Factory SF next week.

Cheers,

Sean Cribbs

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questio...@erlang.org

Tony Arcieri

unread,
Mar 14, 2011, 4:56:41 PM3/14/11
to neoto...@googlegroups.com, Sean Cribbs, Erlang Questions
Any plans for further performance improvements in the roadmap? :)

On Sun, Mar 13, 2011 at 4:25 PM, Sean Cribbs <seanc...@gmail.com> wrote:

> It's been a year coming, so I'm proud to announce a new version of
> Neotoma, the PEG/Packrat toolkit and parser-generator. The primary
> feature of version 1.5 is that it uses significantly less memory and
> also supports UTF8 (via binaries), thanks to some awesome work by
> Graeme Defty, who has also converted Reia to use a PEG.
>
> You can get the latest source on Github:
> https://github.com/seancribbs/neotoma
>
> Neotoma is rebar-friendly as well, so adding it to a rebar project is
> a simple deps declaration like so:
>
> {neotoma, "1.5", {git, "git://github.com/seancribbs/neotoma", {tag,
> "1.5"}}}
>
> Hope to see some of you at Erlang Factory SF next week.
>
> Cheers,
>
> Sean Cribbs
>

--
Tony Arcieri
Medioh! Kudelski

Dmitrii Dimandt

unread,
Mar 15, 2011, 5:33:42 AM3/15/11
to neoto...@googlegroups.com, Erlang Questions

Awesome!

Is reia grammar available? Could make a great place to start learning from

Sean Cribbs

unread,
Mar 15, 2011, 8:58:14 AM3/15/11
to neoto...@googlegroups.com, Erlang Questions
One thing I neglected to emphasize in the announcement was that where
Neotoma's parsing functions would return lists or even single
characters before, they now return binaries. Some grammars will
require drastic changes to accommodate this difference. My apologies
to those who tried the new version and had difficulties (e.g. Cliff).

Regarding Tony's question about future performance improvements, I
think the next one is obvious: Although switching to binaries helped
reduce copying, it can be reduced further by refusing to memoize the
input in its various states of consumption, but simply to scan and
split the single input binary when fetching a memoization (h/t Ben
Black). This will require adding another counter to the "Index" tuple
that keeps the raw byte-offset. I also want to investigate using the
process dictionary a bit more instead of ETS, or at least get a
comparison metric of the two approaches.

I've also been thinking about what should happen for Neotoma 2.0,
although as you can see by the release schedule over the last year, it
has not been high enough on my priorities. Nevertheless, here's my
wishlist:

1) Generate an AST from the metagrammar/grammar parser instead of raw
code, decoupling parsing from code generation.
2) Create some EQC generators that can use the AST of your grammar to
provide automated parser tests.
3) Allow code generation into multiple languages from the AST. One
could generate Erlang, LFE, Reia, Efene, or any language for that
matter. I'm thinking this is what I REALLY wanted to do when I
experimented with parse_transform early on.
4) Let the grammar designer select whether to use binaries or lists as
output of parsing functions.
5) Create analysis tools for the grammar AST that can discover
problems and suggest optimizations.
6) Create a "regular expression" terminal in the metagrammar that can
more succinctly capture strings which currently require combinations
of string literals, character classes and repetition operators.

Thanks for using Neotoma, and as always I'll gladly accept patches to
implement any of the above.

Cheers,

Sean

Torben Hoffmann

unread,
Mar 15, 2011, 10:36:31 AM3/15/11
to neoto...@googlegroups.com, Sean Cribbs, Erlang Questions
Hi Sean,

Perhaps 1.5 should have been 2.0?

See http://semver.org/ where rule 9 states that:

Major version X (X.y.z | X > 0) MUST be incremented if any backwards
> incompatible changes are introduced to the public API. It MAY include minor
> and patch level changes.
>

Just a thought.

Cheers,
Torben

--
http://www.linkedin.com/in/torbenhoffmann

Tony Arcieri

unread,
Mar 15, 2011, 12:50:31 PM3/15/11
to Dmitrii Dimandt, neoto...@googlegroups.com, Erlang Questions

Dmitrii Dimandt

unread,
Mar 16, 2011, 6:50:42 AM3/16/11
to neoto...@googlegroups.com, Erlang Questions

Thank you!

Sean Cribbs

unread,
Mar 16, 2011, 4:09:53 PM3/16/11
to neoto...@googlegroups.com, Erlang Questions
For those who aren't in like with the new binaries-by-default feature,
I've branched off 1.4 and tagged 1.4.1 on github. The tag is after
the rebar-izing of Neotoma, so it should still work with as a dep with
your rebar-based projects.

Cheers,

Sean

Reply all
Reply to author
Forward
0 new messages