PEG Progress : Everything compiles - tests fail

[Graeme Defty]

Well, I got to the point where all the bultins, library modules etc. are compiling and the unit tests are being executed.

The latest stopping point is the following:

<HUGE SNIP>

E.................
=============================Parsing==========================
#
# MessagingTest: Tests for Reia's asynchronous messaging
# Copyright (C)2010 Tony Arcieri
#
# Redistribution is permitted under the MIT license.  See LICENSE for details.
#

module MessagingTest
  def run
    [messaging_test()]
  end

  def messaging_test
    # FIXME: this test could use some work
    TestHelper.expect("Messaging", "works") do
      # Send ourselves a message
      Process.pid() ! (:theanswer, 42)
     
      receive
      when (:theanswer, answer)
        (42, answer)
      end
    end
  end
end
--------------------------------------------------------------
{{line,8},{column,1}} : got token 'module'
{{line,8},{column,8}} : got token '"Mod:MessagingTest"'
{{line,9},{column,3}} : got token 'def'
{{line,9},{column,7}} : got token '"ID:run"'
{{line,10},{column,6}} : got token '"ID:messaging_test"'
{{line,10},{column,6}} : got token '"ID:messaging_test"'
{{line,10},{column,23}} : got token 'eol'
{{line,11},{column,3}} : got token ''end''
{{line,13},{column,3}} : got token 'def'
{{line,13},{column,7}} : got token '"ID:messaging_test"'
{{line,15},{column,5}} : got token '"Mod:TestHelper"'
{{line,15},{column,5}} : got token '"Mod:TestHelper"'
{{line,15},{column,16}} : got token '"ID:expect"'
{{line,15},{column,45}} : got token 'do'
{{line,17},{column,7}} : got token '"Mod:Process"'
{{line,17},{column,7}} : got token '"Mod:Process"'
{{line,17},{column,15}} : got token '"ID:pid"'
{{line,17},{column,7}} : got token '"Mod:Process"'
{{line,17},{column,7}} : got token '"Mod:Process"'
{{line,17},{column,24}} : got token '":theanswer"'
{{line,17},{column,24}} : got token '":theanswer"'
{{line,17},{column,36}} : got token '"int:42"'
{{line,17},{column,36}} : got token '"int:42"'
{{line,17},{column,39}} : got token 'eol'
{{line,19},{column,7}} : got token ''receive''
{{line,20},{column,7}} : got token ''when''
{{line,20},{column,13}} : got token '":theanswer"'
{{line,20},{column,13}} : got token '":theanswer"'
{{line,20},{column,25}} : got token '"ID:answer"'
{{line,20},{column,25}} : got token '"ID:answer"'
{{line,20},{column,25}} : got token '"ID:answer"'
{{line,20},{column,25}} : got token '"ID:answer"'
{{line,20},{column,32}} : got token 'eol'
{{line,21},{column,10}} : got token '"int:42"'
{{line,21},{column,10}} : got token '"int:42"'
{{line,21},{column,14}} : got token '"ID:answer"'
{{line,21},{column,14}} : got token '"ID:answer"'
{{line,21},{column,14}} : got token '"ID:answer"'
{{line,21},{column,14}} : got token '"ID:answer"'
{{line,21},{column,21}} : got token 'eol'
{{line,22},{column,7}} : got token ''end''
{{line,19},{column,7}} : got token ''receive''
{{line,22},{column,10}} : got token 'eol'
{{line,23},{column,5}} : got token ''end''
{{line,15},{column,5}} : got token '"Mod:TestHelper"'
{{line,15},{column,5}} : got token '"Mod:TestHelper"'
{{line,23},{column,8}} : got token 'eol'
{{line,24},{column,3}} : got token ''end''
{{line,25},{column,1}} : got token ''end''
{{line,8},{column,1}} : got token 'module'
----------Parse finished : Result--------

[{module,8,'MessagingTest',
     [{function,9,run,[],
          {var,1,'_'},
          [{cons,10,{local_call,10,messaging_test,[],{nil,1}},{empty,10}}]},
      {function,13,messaging_test,[],
          {var,1,'_'},
          [{remote_call,15,
               {module_name,15,'TestHelper'},
               expect,
               [{string,15,"Messaging"},{string,15,"works"}],
               {lambda,15,[],
                   [{send,17,
                        {remote_call,17,
                            {module_name,17,'Process'},
                            pid,[],
                            {nil,1}},
                        {tuple,17,[{atom,17,theanswer},{integer,17,42}]}},
                    {'receive',19,
                        [{clause,20,
                             [{tuple,20,
                                  [{atom,20,theanswer},{var,20,answer}]}],
                             [{tuple,21,[{integer,21,42},{var,21,answer}]}]}],
                        []}]}}]}]}]

----------- End of Parse Result ---------

EE
\n'List retrieves the nth element correctly' FAILED\nexpected true, actual false\n\n'List replaces elements in pattern matching expressions ' FAILED\nexpected [1,42,69], actual 69\n\n'List replaces elements in complex pattern matching expressions ' FAILED\nexpected [8,9,10,4], actual 10\n\n'Tuple retrieves the nth element correctly' FAILED\nexpected true, actual false\n\n'Tuple replaces elements in pattern matching expressions ' FAILED\nexpected (1,42,69), actual 69\n\n'Tuple replaces elements in complex pattern matching expressions ' FAILED\nexpected (8,9,10,4), actual 10\n\n'String inspects properly' FAILED\nexpected \""foobar"\", actual \"\"foobar\"\"\n\n'Dict gets values by key' FAILED\nexpected \"bar\", actual {:foo=>\"bar\"}\n\n'Dict casts to a string' FAILED\nexpected \"{:foo=>"bar"}\", actual \"{:foo=>\"bar\"}\"\n\n'Dict inspects properly' FAILED\nexpected \"{:foo=>"bar"}\", actual \"{:foo=>\"bar\"}\"\n\nbuiltins/atom.re ERROR: SyntaxError: /home/graeme/projects/reia/test/builtins/atom_test.re:8: Parse error near "module AtomTest" (Column: 1)\n\tfrom Object: in 'call'
\tfrom JsonParser: in 'transform'
\tfrom JsonParser: in '-transform/2-lc$^0/1-0-'
\tfrom JsonTest: in '-list_test/2-fun-0-'
\tfrom TestHelper: in 'expect'
\tfrom JsonTest: in 'run'
\tfrom /home/graeme/projects/reia/test/runner: in '-toplevel/0-fun-0-'
\tfrom lists: in 'map'\n\ncore/object.re ERROR: SyntaxError: /home/graeme/projects/reia/test/core/object_test.re:30: Parse error near "module ObjectTest" (Column: 1)\n\tfrom Object: in 'call'
\tfrom JsonParser: in 'transform'
\tfrom JsonParser: in '-transform/2-lc$^0/1-0-'
\tfrom JsonTest: in '-list_test/2-fun-0-'
\tfrom TestHelper: in 'expect'
\tfrom JsonTest: in 'run'
\tfrom /home/graeme/projects/reia/test/runner: in '-toplevel/0-fun-0-'
\tfrom lists: in 'map'\n\ncore/messaging.re ERROR: (:case_clause,[])\n\nlib/json.re ERROR: badarg\n\nexception error: bad argument in an arithmetic expression
  in function  'TestHelper':duration/2
  in call from '/home/graeme/projects/reia/test/runner':toplevel/0
  in call from reia_bytecode:load/2
  in call from reia_internal:execute_file/1
  in call from init:start_it/1
  in call from init:start_em/1


I spot a fondness for the third element of everything in the FAILED messages there. Could we be generating bad executable code?

I post this here purely as an interesting exercise for the reader - just in case anything obvious leaps out.

I have pretty much convinced myself that I need to hack up a way to send each of the *.re modules to the current parser and the PEG and compare the outputs. It is probably quicker in the long run than investigating individual problems, and it will give a much higher level of confidence in the end.

Unless you think otherwise?

g

[Tony Arcieri]

I'm visiting Switzerland now for business, but when I get back home to the US I can take a look and compare the output of the two parsers.

--
You received this message because you are subscribed to the Google Groups "Reia" group.
To post to this group, send email to re...@googlegroups.com.
To unsubscribe from this group, send email to reia+uns...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/reia?hl=en.

--
Tony Arcieri
Medioh! Kudelski

[Graeme Defty]

Sorted! I have built a small test rig that trawls through the directory structure looking for '.re' files, passes them to both parsers and compares the output.

The comparison was non-trivial because the old parser puts incorrect line numbers in some places, and so the comparison had to walk the parse tree in a (semi-) intelligent manner. Neverthess, I am glad I have this, because it has highlighted a couple of oddities (with more to come no doubt) and at the end of this exercise, I will be very confident that we are generating the right tree. Not sure yet if I have come to the file that generates the error below.

I will push it as part of the project once it has found all the errors it can.

At some point we will not be able to compare with the old parser, but I am also adding unit tests to the parser tester to cover the odd cases that I uncover (and that I failed to consider initially)

As a side note - one bizarre aspect of all this is that I have yet to write a single line of Reia code myself!

_________________________________________________

[Tony Arcieri]

Great news! I was actually just poking around at it myself and made this:

https://gist.github.com/809362

That's the output from your parser and subsequently the Reia compiler.

I committed a change which disables the caching of compiled Reia bytecode, since the first thing I encountered when I tried to run the tests was all of them passing as the only file it compiled was test/runner.re (which seemed to work?)

I was just about to commit some debugging that would print out the output of the entire Reia compiler which it then passes to the Erlang compiler.

[Phil Pirozhkov]

That's great success already! Congratulations on that.

Phil


Thu, 3 Feb 2011 18:13:41 +0700 О©╫О©╫О©╫О©╫О©╫О©╫ О©╫О©╫ Graeme Defty <graeme...@gmail.com>:

Sorted! I have built a small test rig that trawls through the directory structure looking for '.re' files, passes them to both parsers and compares the output.

The comparison was non-trivial because the old parser puts incorrect line numbers in some places, and so the comparison had to walk the parse tree in a (semi-) intelligent manner. Neverthess, I am glad I have this, because it has highlighted a couple of oddities (with more to come no doubt) and at the end of this exercise, I will be very confident that we are generating the right tree. Not sure yet if I have come to the file that generates the error below.

I will push it as part of the project once it has found all the errors it can.

At some point we will not be able to compare with the old parser, but I am also adding unit tests to the parser tester to cover the odd cases that I uncover (and that I failed to consider initially)

As a side note - one bizarre aspect of all this is that I have yet to write a single line of Reia code myself!

_________________________________________________

On 2 February 2011 15:45, Tony Arcieri <tony.a...@medioh.com> wrote:

I'm visiting Switzerland now for business, but when I get back home to the US I can take a look and compare the output of the two parsers.

On Tue, Feb 1, 2011 at 5:01 PM, Graeme Defty <graeme...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Reia" group.
To post to this group, send email to re...@googlegroups.com.
To unsubscribe from this group, send email to reia+uns...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/reia?hl=en.

--
Tony Arcieri
Medioh! Kudelski

--
You received this message because you are subscribed to the Google Groups "Reia" group.
To post to this group, send email to re...@googlegroups.com.
To unsubscribe from this group, send email to reia+uns...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/reia?hl=en.

[Graeme Defty]

SUCCESS!

Well, . . . kinda.

I have got through to the end of all the files, but i had a couple of stumbles that I had to work around:

1) src/builtins/exceptions.re
    has a spelling error on line 43 (Excpetions)

2) test/core/operator_test.re
    on line 80 tests for equality of (42, ~-43). For the -43 the leex/yecc parser constructs a unary minus operator with operand {integer,x,43}. In the PEG grammar I have made a slight simplification by making the integer's value signed (i.e. constructing {integer,x,-43)) whenever the sign is adjacent to the digits. In other words, a leading sign on a float or integer is considered to be part of the number. (I do not expect that this will break any existing code.) To avoid the discrepancy in the parse trees I changed the test to compare   (-44, ~43).

3) test/builtins/atom_test.re
    seemed not to parse properly under either parser. Had to comment out lines 14-25

4) class/test/core/class_test.re
    also did not parse properly. Commented out lines 9-18.

These last two are a little puzzling so I would be interested to know if they are valid code.

Anyway, too late for further thoughts on this. I will tidy things up a bit and push all this tomorrow.

g
____________________________________________

[Graeme Defty]

Tony.

I lied a bit - I have pushed the code already.

Not exactly a thing of beauty, but we are getting there.

Next steps as I see it:
1) resolve those two odd points
2) update it to take account of the changes you have made in the interim
3) relax the syntax of regex's (see - I haven't forgotten)
4) Add in all the 'FIXME' unit tests
5) look at Neotoma for performance and error reporting

Happy to hear other views on priorities.

g
_____________________________

[Tony Arcieri]

Great news! I'll be sure to check it out when I get home.

[Tony Arcieri]

I might be mistaken, but it doesn't look like you're handling string escapes either? (e.g. \n)

Great work though.

Regarding neotoma performance, have you checked out Sean's "memory" branch?

[Graeme Defty]

Tony,

There was one point last night where I noticed that escapes were being handled for plain strings but i had omitted to allow them in interpolated ones (don't ask!)

That should be fixed now, so let me know if you still think it is not working properly.

I did take a look at Neotma "memory" a while back but did not really understand Neotoma well enough to know what I was looking at. I will check it out again shortly.

g
______________________________

[Graeme Defty]

Did I forget to mention . . . all the reia unit tests pass too - apart from atom_test.re not parsing.

g

PS
You may find things build slowly. I have a slightly 'doctored' version of Neotoma with no caching which builds the parser quickly, but builds it as a standard (i.e. caching) Neotoma parser so that the reia coded parses quickly. I do not want to push the changes (mostly because I have forgotten how they work ;-) )

When we have improvements to Neotoma I will replace the whole thing in the source tree.

_______________________________________

[Graeme Defty]

Well, I am struggling a bit here.

The problem in class_test.re is that the code looks as follows:

class IHasSingleton
  def self.example1
  end
 
  class self
    def example2
    end
   
    def example3
    end
  end
end

now the yecc source for function IDs looks as follows:

%% Allowable prefixes for defs
def_prefix -> def function_identifier : ?identifier_name('$2').
def_prefix -> def '[' ']' : '[]'.
def_prefix -> def '[' ']' '=' : '[]='.

%% Function identifiers
function_identifier -> identifier : '$1'.
function_identifier -> punctuated_identifier : '$1'.
function_identifier -> class : {identifier, ?line('$1'), class}.
function_identifier -> self  : {identifier, ?line('$1'), self}.

As far as I can make out, this allows for funtions called 'class' and 'self' but not names of the form 'self.identifier'.

Am I missing something?

g
______________________________________

[Tony Arcieri]

On Sun, Feb 13, 2011 at 8:36 AM, Graeme Defty <graeme...@gmail.com> wrote:

now the yecc source for function IDs looks as follows:

%% Allowable prefixes for defs
def_prefix -> def function_identifier : ?identifier_name('$2').
def_prefix -> def '[' ']' : '[]'.
def_prefix -> def '[' ']' '=' : '[]='.

%% Function identifiers
function_identifier -> identifier : '$1'.
function_identifier -> punctuated_identifier : '$1'.
function_identifier -> class : {identifier, ?line('$1'), class}.
function_identifier -> self  : {identifier, ?line('$1'), self}.

As far as I can make out, this allows for funtions called 'class' and 'self' but not names of the form 'self.identifier'.

Am I missing something?

This was added very recently so it might've slipped under your radar:

%% Class methods

class_method -> def self '.' function_name eol body 'end' : 

  #class_method{

    line = ?line('$1'), 

    name = '$2', 

    body = '$4'

  }.

class_method -> def self '.' function_name args eol body 'end' :

  #class_method{

    line  = ?line('$1'), 

    name  = '$2', 

    args  = '$3'#args.args,

    block = '$3'#args.block,

    body  = '$5'

  }.

Note the eventual goal is to allow arbitrary expressions in class/module bodies (for metaprogramming).

And just a heads up, I might try getting rid of "hashrockets", changing the grammar for dicts from {foo => bar} to {foo: bar}

This is the direction 1.9 is going in and also what JavaScript and Python do... I think visually it's more pleasing.

[Graeme Defty]

Aaargh! I have deliberately not taken any upgrades since I started so I would not have goalposts galloping all over the field. I knew that I would have a catch-up step at some point. I just did not realise I was there already.

One puzzling thing is that the reia code I have and the yecc parser seem out of sync, although I pulled them both at the same time AFIK (which is what threw me a bit). Does that make sense based on the way you did/pushed the updates?

Anyway, onward and upward...

g
______________________________________

[Tony Arcieri]

On Sun, Feb 13, 2011 at 6:42 PM, Graeme Defty <graeme...@gmail.com> wrote:

Aaargh! I have deliberately not taken any upgrades since I started so I would not have goalposts galloping all over the field. I knew that I would have a catch-up step at some point. I just did not realise I was there already.

This is pretty much the only change to the grammar I've made recently.

 

One puzzling thing is that the reia code I have and the yecc parser seem out of sync, although I pulled them both at the same time AFIK (which is what threw me a bit). Does that make sense based on the way you did/pushed the updates?

You're working on a different branch. I can try to update it to the latest code (using a "rebase" in git terminology) when I get home tonight.

 

[Tony Arcieri]

So I tried to do a rebase and it wasn't entirely successful... I ended up needing to merge some changes in addition to the rebase. Not entirely sure why, and the resulting network graph appears to be a bit funky, but everything should now be up-to-date between the two branches and I hope I didn't break anything!

[Graeme Defty]

Thanks Tony.

I think that maybe I need to re-base too. I don't seem to be able to do anything due to non-fast-forward issues :-(

(Did I mention that I hate git?)

I have tried lots of ways around over the last day or two and nothing seems to work. Maybe it is time for the 'brute force and ignorance' approach.

Before I do, I will try a new clone (again) incase my last did not pick up your re-base. I will let you know how it goes.

g
_______________________________

--

[Tony Arcieri]

There is a rebase plugin for Mercurial:

http://mercurial.selenic.com/wiki/RebaseProject

I honestly don't know how my typical git workflow (everyone working on master and rebasing before they push their changes) is supposed to work with Mercurial.

Can you try to rebase?