Beancount grammar for tree-sitter

[Daniele Nicolodi]

Hello,

tree-sitter is getting quite some attention as a parsing library
suitable for being used for syntactic support in editors. So much that
even Emacs will include support for it in the next major release. I was
curious to try it out and I recently had a couple of very long train
journeys...

I wrote a Beancount grammar for tree-sitter. It can be found here

https://github.com/dnicolodi/tree-sitter-beancount

It went through only limited testing and I never used tree-sitter before
and I haven't used this grammar for anything real just yet. Therefore,
if you decide to build something on top of it, expect some changes in
the emitted syntax tree. In particular, I think I need to add names to
some of the emitted nodes to make it really usable.

I would be happy to move this project to the beancount Github
organization. Martin, can you arrange for a project to be created and to
give me owner rights for it? Thank you!

Cheers,
Dan

[Martin Blais]

You got it 

https://github.com/beancount/tree-sitter-beancount 

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/3701d857-c91f-0313-e79f-90fc055a4174%40grinta.net.

[Daniele Nicolodi]

On 24/01/2023 03:17, Martin Blais wrote:
> I would be happy to move this project to the beancount Github
> organization. Martin, can you arrange for a project to be created
> and to give me owner rights for it? Thank you!
>
>
> You got it
> https://github.com/beancount/tree-sitter-beancount

Thank you Martin. However, reading GitHub documentation it makes more
sense to transfer my repository to the beancount organization rather
than creating a new repository.

If you delete beancount/tree-sitter-beancount I'll setup the transfer.

Thank you.

Cheers,
Dan

[polarmutex]

i also wrote a tree-sitter grammar a while ago and have it officially submitted to nvim-treesitter , nix package manager, and use it in my beancount lsp. I am happy to work to gather to merge in changes but think we should work together. I would rather move my repo to the group.

[polarmutex]

forgot link to repo: https://github.com/polarmutex/tree-sitter-beancount

[Daniele Nicolodi]

Hello,

thank you for letting me know of this concurrent effort. I had a quick
look at your implementation and I noticed a few important things:

1. as far as I can tell, it does not handle indentation correctly, it
just ignores it,

2. it does not correctly work with account names in the whole Unicode
range, it mimics the hack implemented for

3. it mimics too closely the Yacc grammar, resulting in a parse tree
structure that seems hard to work with both because of the node nesting
and to what nodes are exposed,

4. less important: I don't like how you choose to handle org-mode like
sectioning having Beancount directive nested into section nodes. I don't
think is compatible with how Emacs handles sectioning.

On the other hand, it adds field names to some node, something that my
grammar does not yet to.

If you want to collaborate on this, I'm more than happy to add any
structure to the parse nodes emitted by my version of the grammar to
make it suitable to be used in your use cases. But I think it is better
to start with a correct and clean parser.

Cheers,
Dan

> --
> You received this message because you are subscribed to the Google
> Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to beancount+...@googlegroups.com

> <mailto:beancount+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/beancount/89848b9c-0d5b-4142-a2f6-5d9acec7fbaan%40googlegroups.com <https://groups.google.com/d/msgid/beancount/89848b9c-0d5b-4142-a2f6-5d9acec7fbaan%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Daniele Nicolodi]

(I realized I left a sentence incomplete)

> 2. it does not correctly work with account names in the whole Unicode
> range, it mimics the hack implemented for

Flex, which has only primitive regular expressions. tree-sitter supports
regular expressions matching unicode character classes, thus a much
better job can be made.

Cheers,
Dan

[polarmutex]

 I am happy to work together and make changes to clean it up(no one has submitted issues about it). I just feel slighted that I did not get a chance to be merged into the group when I have been working it for a while. Should I merge my lsp in to claim ownership? I have been making changes to the grammar based upon issues and trying to be a nice player in the OSS community. I never said my grammar was perfect but I have posted about it on this group before and have been working it for a while. 

[Daniele Nicolodi]

On 26/01/2023 16:31, polarmutex wrote:
>
>  I am happy to work together and make changes to clean it up(no one has
> submitted issues about it). I just feel slighted that I did not get a
> chance to be merged into the group when I have been working it for a
> while.

I forgot about your messages to the mailing list about your efforts to
implement the tree-sitter grammar for Beancount. I didn't see any
request to bring it into the beacount organization on Github. If that
would have happened, I would have had a look at it and I would have
noticed the shortcomings before and we could have worked to resolve them.

It's not ideal that there are two projects with the same scope. I'm
happy to collaborate on this. However, I think that the changes required
to fix your parser are very close to a rewrite: the scanner needs to be
extended to handle indentation and the grammar nodes extensively reworked.

I would greatly appreciate if you could look at my grammar and determine
whether there are shortcomings that make it unusable for your use cases.
I know that node field names are missing. However, these can be added
where needed, and I believe this is much less invasive than fixing your
grammar according to my vision.

> Should I merge my lsp in to claim ownership?

Ownership of what? Isn't it your project already?

Cheers,
Dan

> I have been making
> changes to the grammar based upon issues and trying to be a nice player
> in the OSS community. I never said my grammar was perfect but I have
> posted about it on this group before and have been working it for a while. >
> On Thursday, January 26, 2023 at 10:20:05 AM UTC-5 dan...@grinta.net wrote:
>
> (I realized I left a sentence incomplete)
>
> > 2. it does not correctly work with account names in the whole
> Unicode
> > range, it mimics the hack implemented for
>
> Flex, which has only primitive regular expressions. tree-sitter
> supports
> regular expressions matching unicode character classes, thus a much
> better job can be made.
>
> Cheers,
> Dan
>

> --
> You received this message because you are subscribed to the Google
> Groups "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to beancount+...@googlegroups.com
> <mailto:beancount+...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/beancount/4e7193e6-c860-4c83-8014-357295b3e117n%40googlegroups.com <https://groups.google.com/d/msgid/beancount/4e7193e6-c860-4c83-8014-357295b3e117n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[polarmutex]

ownership as in the official repo. being in the official group means something. I feel like you are just dismissing my efforts. 

[Martin Blais]

Just catching up.

I have no opinion on this, happy to move the polarmutex repo into the beancount org as well if you want, and it's true you've been at this for a long while and very active.

(Can you choose unique names? Looks like you both choose the very same repo name.)

FWIW, "official" isn't all that meaningful other than it makes it easier for people to find related repos to Beancount, especially things like syntax parsers.

When I search for "beancount" in github I find 590 repos (I'm amazed and feel ashamed I'm not able to spend much time on it anymore).

Let me know what you want to do,

To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/3e2fcf8d-1915-465a-911f-f25edd8c05ebn%40googlegroups.com.

[Stefano Zacchiroli]

On Fri, Jan 27, 2023 at 10:19:20PM -0500, Martin Blais wrote:
> I have no opinion on this, happy to move the polarmutex repo into the
> beancount org as well if you want, and it's true you've been at this for
> a long while and very active.
> (Can you choose unique names? Looks like you both choose the very same repo
> name.)
> FWIW, "official" isn't all that meaningful other than it makes it easier
> for people to find related repos to Beancount, especially things like
> syntax parsers.

If I may, as a mere user, having a repo located under
github.com/beancount/ is quite meaningful to me, precisely because it's
more prominent than other repos out there (both on github and
elsewhere). It comes with an expectation that the repo implements the
"recommended" Beancount way of doing something for which alternative
implementations exist. I understand you Martin doesn't mean it that way,
but it *will* be interpreted that way by users, especially those who are
not following closely this list.

Re tree-sitter (which is something I might be interested in using myself
in the future), I'd be better served by having either 0 or exactly 1
tree sitter grammar implementation available under beancount/ on github.
The ideal way forward here would be for Daniele and polarmutex to work
together into a single project and having that one located at
https://github.com/beancount/tree-sitter-something.

Just my 0.02 EUR,
Cheers
--
Stefano Zacchiroli . za...@upsilon.cc . https://upsilon.cc/zack _. ^ ._
Full professor of Computer Science o o o \/|V|\/
Télécom Paris, Polytechnic Institute of Paris o o o </> <\>
Co-founder & CTO Software Heritage o o o o /\|^|/\
https://twitter.com/zacchiro . https://mastodon.xyz/@zacchiro '" V "'

[Martin Blais]

On Sat, Jan 28, 2023 at 3:01 AM Stefano Zacchiroli <za...@upsilon.cc> wrote:

On Fri, Jan 27, 2023 at 10:19:20PM -0500, Martin Blais wrote:
> I have no opinion on this, happy to move the polarmutex repo into the
> beancount org as well if you want, and it's true you've been at this for
> a long while and very active.
> (Can you choose unique names? Looks like you both choose the very same repo
> name.)
> FWIW, "official" isn't all that meaningful other than it makes it easier
> for people to find related repos to Beancount, especially things like
> syntax parsers.

If I may, as a mere user, having a repo located under
github.com/beancount/ is quite meaningful to me, precisely because it's
more prominent than other repos out there (both on github and
elsewhere). It comes with an expectation that the repo implements the
"recommended" Beancount way of doing something for which alternative
implementations exist. I understand you Martin doesn't mean it that way,
but it *will* be interpreted that way by users, especially those who are
not following closely this list.

Re tree-sitter (which is something I might be interested in using myself
in the future), I'd be better served by having either 0 or exactly 1
tree sitter grammar implementation available under beancount/ on github.
The ideal way forward here would be for Daniele and polarmutex to work
together into a single project and having that one located at
https://github.com/beancount/tree-sitter-something.

Alright, fair enough.

Daniele has pretty strong views about how he likes to do things (and that's okay), so asking these two to collaborate is not something I want to get involved in.

And it's true polarmutex has been at this for a VLT (I followed the changes for a while) so doesn't deserve to get eclipsed.

I also see no less than FOUR "tree-sitter-beancount" repos on github (once again, amazed, I supposed I'm going to have to get on with the times at some point and check it out too, but it seems the last few years there's bursts of excitement over new parser generators).

I think we should

- remove the newly created repo to avoid confusion

- update the dedicated section for alternative parser in the contribs doc (done)  - https://docs.google.com/document/d/1Z37bQ45wDtjTPaMQ_x-f33p1trH9fNosEAUgbQXwp30/edit#heading=h.vdgjwzkgrfcy

If you have yet another alternative parser with <insert-fancy-new-parser-generator-here> let me know I'll add to the doc.

(I managed to do a little tiny bit of work over advancing the v3 state of the world over the holiday down under, I'll try to send an update soon.)

 

Just my 0.02 EUR,
Cheers
--
Stefano Zacchiroli . za...@upsilon.cc . https://upsilon.cc/zack  _. ^ ._
Full professor of Computer Science              o     o   o     \/|V|\/
Télécom Paris, Polytechnic Institute of Paris     o     o o    </>   <\>
Co-founder & CTO Software Heritage            o o o     o       /\|^|/\
https://twitter.com/zacchiro . https://mastodon.xyz/@zacchiro   '" V "'

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20230128080135.sldrzkksgrhomzhu%40upsilon.cc.

[fin]

Martin Blais wrote:
...

> (I managed to do a little tiny bit of work over advancing the v3 state of
> the world over the holiday down under, I'll try to send an update soon.)

...

glad to hear any news at all. :) thanks so much for all you've
done so far.


fin