PEG and class_methods

[Graeme Defty]

Tony,

The only outstanding issue at the moment is in the parse trees returned for the source in test/core/class_test.re.

This module defines a class method as follows:

  def self.foo
    :class_method
  end

The leex parser returns this as

   {class_method,9,{self,9},[],{var,1,'_'},foo}

which appears to me to be incorrect.

The PEG parser gives

   {class_method,9,foo,[],{var,1,'_'},[{atom,10,class_method}]}

which (against all expectations) appears to me to be right.

Thoughts? (Or more likely, what have I missed?)

g

[Graeme Defty]

On the off-chance that I was right about the cause of this mis-match, I have pushed a change to the peg branch which fixes the yecc parser. (I will undo it if you disagree with my diagnosis.)

With that in place, the peg branch now builds correctly including
- the unit tests for peg,
- the tests for Reia itself, and
- my A/B tests comparing all reia source in the source tree under the two parsers to ensure that the parse trees are identical.

I also pushed my temporary patch for Neotoma to speed the parser build.

See what you think, but I believe we may have a useable peg parser!  (Thanks for holding off changes while I got to this point.)

Two outstanding areas are:
- a permanent improvement for Neotoma caching, and
- the peg parser unit tests are a bit 'thin' in a couple of places. I would like to see them rounded out a bit.

I will work on these in this order, and track/mirror any changes you make in the yecc parser.

g

PS
I have also added a page to the wiki to describe the rules for newline handling, though I think there are probably still a few changes to come in this area.

____________________________________________

[Tony Arcieri]

Awesome, great work!

You're correct that class methods weren't getting parsed correctly. I never managed to get them completely wired up. The next step is rather involved :/

The changes you made to Neotoma definitely do speed things up. Rather than spending several minutes sucking up memory and swapping, the parser builds almost instantly. The resulting parser though seems to be quite slow compared to the yecc version:

yecc: Finished in 0.331979 seconds

neotoma: Finished in 20.085591 seconds

So my question is, is this speed difference due to the modifications you made to improve the speed of generating the parser itself?

I'm thinking the thing to do, before merging this into master (and I'd really like to) is try switching to Sean's "memory" branch of Neotoma, and once that's done, checking the resulting parser into git so it only needs to be rebuilt if modifications are made to the grammar.

If we do that, it might be possible to stop shipping a vendored neotoma and switch to the version that's in agner too:

https://github.com/agner/neotoma.agner

One thing I did run into... it looks like you're checking the grammar of all .re files in the reia directory. I have this checked out:

https://github.com/tarcieri/lolcat

When "lolcat.re" gets checked, I see this error:

Verifying parse for ./lolcat/lolcat.re

=====> Unknown Match!!!!

{'receive',34,

    [{clause,35,

         [{tuple,35,[{atom,35,tcp},{var,35,'_'},{var,35,line}]}],

         [{local_call,36,process_line,

              [{remote_call,36,

                   {remote_call,36,{var,36,line},to_s,[],{nil,1}},

                   chop,[],

                   {nil,1}}],

              {nil,1}}]},

     {clause,37,

         [{tuple,37,[{atom,37,line},{var,37,line}]}],

         [{local_call,38,send_message,

              [{dstring,38,

                   [{string,38,"PRIVMSG "},

                    {ivar,38,channel},

                    {string,38," :"},

                    {var,38,line}]}],

              {nil,1}}]},

     {clause,39,[{var,39,msg}],[{nil,1}]}],

    void}

{'receive',34,

    [{clause,35,

         [{tuple,35,[{atom,35,tcp},{var,35,'_'},{var,35,line}]}],

         [{local_call,36,process_line,

              [{remote_call,36,

                   {remote_call,36,{var,36,line},to_s,[],{nil,1}},

                   chop,[],

                   {nil,1}}],

              {nil,1}}]},

     {clause,37,

         [{tuple,37,[{atom,37,line},{var,37,line}]}],

         [{local_call,38,send_message,

              [{dstring,38,

                   [{string,38,"PRIVMSG "},

                    {ivar,38,channel},

                    {string,38," :"},

                    {var,38,line}]}],

              {nil,1}}]},

     {clause,39,[{var,39,msg}],[{nil,39}]}],

    void}

--
You received this message because you are subscribed to the Google Groups "Reia" group.
To post to this group, send email to re...@googlegroups.com.
To unsubscribe from this group, send email to reia+uns...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/reia?hl=en.

--
Tony Arcieri
Medioh! Kudelski

[Graeme Defty]

Tony,

I assume that the next step for class methods is on the code generation side?

Yes, my version of Neotoma is much better at building the parser (though not instant - you must have a nice machine :-D ), but the generated parser is still not really useable.

However, last night I looked at Sean's memory branch and could not believe how quick it was. Apparently instantaneous! I thought it had done nothing, but when I looked, the generated file looked fine. I have not timed it, but it looks to be in the same sub-second ballpark as leex/yecc, so I agree that is definitely the way to go.

I will ensure that it is building cleanly and that all tests pass, then wire it into the peg branch - hopefully by end of today.

>  ... it looks like you're checking the grammar of all .re files in the reia directory ...
Yes indeed. A temporary step to give us a bit of confidence during the transition to the peg parser (for as long as we can be bothered maintaining the leex/yecc version) - I thought it would be handy while you acquaint yourself with the peg version.  The error is puzzling since the two parse trees look the same to me - I will look at the error to see what's going on and get back to you. The B tester was a quick hack so there could be a whoopsie there.

g
_______________________________________

[Graeme Defty]

Tony,

ok - I have had Neotoma/memory working and it FLIES. The parser now builds quicker than it compiles! I assume that reia compilations will be quick too.

Now the bad news . . . before pushing the changes, I took a look at your lolcat problem. Actually, the only issue here is that the peg parser normally puts real line numbers into the {nil,...} items, where leex/yecc often puts '1'. Not really a problem at all, but In trying to remedy this things went awry and then I lost the ability to build a working version (and the build was working perfectly - including my peg tests, reia unit tests, leex/yecc compirisons - the lot! To say I am disappointed would be an understatement.)

Anyway, I have pushed all the source as it stands right now. I am fairly sure that it is the source that works, and what I have broken is actually the build system (never really did understand how the neotoma build worked).

Anyway, I found and fixed a problem in neotoma/memory and have checked in the changes (see my version of neotoma) and copied this into reia/peg, which also contains all my changes so far (AFAIK).

It may be that I have missed something, but it is past 8.00pm, I am in Manila and I need a beer. I will do no more tonight.

Feels a bit like snatching bitter disappointment from the jaws of glorious triumph :-(   sigh!

   (but hey that parser FLIES ! ! !  :-D )

We are close.

g
_____________________________________________

On 20 February 2011 06:33, Tony Arcieri <tony.a...@medioh.com> wroteok:

[Tony Arcieri]

Great news, both regarding the generation speed and parsing performance. Sorry to hear you broke something in the build before committing. I  might poke at what you committed tonight, and I look forward to checking it out when it's working.

Maybe I can look through the commit and spot something obvious.

[Graeme Defty]

Yeah - sorry to have pushed a broken version, but I wanted to get a stake in the ground before I made it any worse :-(

I have a feeling it is in the build mechanism itself (rakes and makes) but I am not sure. The way Neotoma is integrated is a little fragile right now.

Anyway, I will look again tonight and try to retrace my steps.

Definitely worth the effort, though!

g
_________________________________

[Tony Arcieri]

I took a look... not really sure what's going on. Neotoma seems to work. Calling reia:parse/1 directly from Erlang seems to work and produces the expected result. But trying to load any nontrivial file seems to crash the Reia compiler.

I noticed dir.re compiled to this:

[{attribute,1,module,'/Users/tony/src/reia/lib/dir'},

 {attribute,1,file,{"/Users/tony/src/reia/lib/dir.re",1}},

 {attribute,1,code,[]},

 {attribute,1,parent,'/Users/tony/src/reia/lib/dir'},

 {attribute,1,submodules,[]},

 {function,1,toplevel,0,[{clause,1,[],[],[]}]}]

*All* of the code is missing for some reason. This same problem seems to be happening on the parser tests.

Not sure why as the parser appears to be working?

[Graeme Defty]

Yes, peculiar in the extreme. The biuld process overall seems not to be repeatable (and that may be me, because I haven't really followed it through properly.) Whatever, I will take a look tonight and get it sorted out.

Damned annoying though, since I have seen it working flawlessly ("honest, guv'nor")

g
___________________________________

[Tony Arcieri]

Any luck? Are you actually having problems compiling, or are you running into the same problems as me? I get "Internal Erlang compiler error" on all of the parser tests.