Fixes pushed - Neotoma/memory version working (allegedly)

[Graeme Defty]

Tony,

I have pushed a new version which seems to fix the problems I put up before.

I went back to basics and re-constructed the Neotoma build, as it seemed to be overwriting my changes in ways I did not understand.

Unfortunately I had to go from Make to Rake in order to get the flexibility I needed. (Not that Rake is a bad thing. I just do not like it when I am forced to load another language processor just to build a project which is not even in that language, so I hate to impose the same thing on others. For Reia it is not a problem as are are using Rake anyway, but I may be in Sean's bad books over it.)

Anyway, all seems to be working.
- The build of the parser is durned near instantaneous.
- All parser unit tests pass.
- All Reia unit tests pass.
- All A/B ests pass (including lolcat, which I added to the examples dir) and
- the speed increment in those AB tests seems to indicate that the reia parser is as fast as the parser build.

Happy days!

Take a look.

g

[Tony Arcieri]

Awesome work! Everything appears to be working for me as well.

Definitely a massive improvement on the performance. Building the parser goes very quickly (and doesn't grind my machine to a halt). Rather than 20 seconds, the tests run in 6.6 seconds.

That's still pretty far short of the yecc parser, though. That runs the tests for me in 0.37 seconds.

Sean, any advice as to what the next steps are for improving the performance?

--
Tony Arcieri
Medioh! Kudelski

[Tony Arcieri]

I made a minimal case of parsing all of the core Reia files available and ran it through fprof for generating profile data. My results are available here:

https://gist.github.com/9ab47f7322e07772653

I'm really not sure how to interpret them though. This is my first time using fprof and I need to learn more about its analysis functions.

[Graeme Defty]

Tony,

I get 404 from this page :-(

Is the link right?

g
_________________________________

[Graeme Defty]

Sean,

One of the things that was getting me all twisted in the build was the fact that I had Neotoma installed and I did not realise that I was pulling parts of the install into my development version.

To fix this I made a change to pull the includes from the priv directory relative to the .beam file, rather than relative to the app (which seems always to get the installed version). The other alternative would always be to replace the installed version, but that seemed a little too intrusive.

The other thing I did was to rename the peg_includes.erl to peg_includes.hrl, which seemed more in keeping with its role. As a knock-on, this led to a few other changes, of course. I hope I have not done violence to any of your more treasured concepts.

One other slight change I made was to improve the encapsulation of the memo mechanism by collecting all detail of the internals into the access functions get_memo() and memoize(). The intent was to make it easier to experiment with different memoizing mechanisms.  I have already made use of this to test out a theory that ets may be costing us heavily, and so I replaced it with a proplist (so that made a proplist keyed by Index with each entry being a proplist keyed by Name). I was curious as to how much copying ets was doing, and thought that this may perform better. It didn't - lol.   I will not bother to post that branch unless anyone is really interested to see what I did. (May be that I made some error - I confess to some surprise that it was about twice a slow!)

Anyway, pull request on its way.

g
_____________________________________________________

On 1 March 2011 08:12, Sean Cribbs <seanc...@gmail.com> wrote:

Graeme,

This is exciting news! Send me a pull-request for your changes to
neotoma and I'll merge them in.

Sean