binary-based parsing

[Graeme Defty]

Tony,

Well, the next step was not the trivial exercise I expected.

          "Ah, but I was so much older then, I’m younger than that now"

I have spent today with the 'memory' branch of Neotoma. One of the more whimsical aspects of it is that since the source file is retained in binary form, it now returns all text elements found as binaries, rather than lists.

Initially, I started to change Neotoma/memory so that it behaved as before, i.e. returning lists, to make it a plug-in replacement for Neotoma/master.

Then I realised that apart from deviating from Sean's version/concept, it also involved a lot of translation in Neotoma for no particular benefit.

So, I started to change my peg code to accept binaries (e.g. for method names, class names, identifiers, operators etc).

Then I realised that I had just moved the problem back a level, since I still had to convert into atoms etc to fill the parse tree.

It would appear that if we continue along this line of reasoning, then I should put binaries into the parse tree too, but then your subsequent code-generation stages would need to be changed to expect binaries in lots of places too.

Either way, there is a bit of work to be done.

I am inclined to think that

1) the least effort (though this is by no means certain) is probably to do it in Neotoma, as I originally began to do, whereas

2) the most efficient (and the most 'natural' approach) is to carry binaries right through the compiler.

Thoughts?

g

[Tony Arcieri]

At one point or other the method/class/module names need to get changed to atoms, as that's what the Erlang compiler itself is expecting. Reia compiles to the Erlang abstract format, so the Reia parse tree tries to match that format as closely as possible.

Why can't you just do binary_to_atom/2 where you were formerly doing list_to_atom/1?

> binary_to_atom(<<"foobar">>, utf8).

foobar

--
Tony Arcieri
Medioh! Kudelski

[Graeme Defty]

You know I really shouldn't work late at night. I can't hack it any more :-(

For some unknown reason, I had myself convinced there *was* no binary_to_atom.

Sigh.

Oh well. On the bright side - it's a nice easy answer :-)     (well . . . easy-ish)

I will tackle it forthwith (i.e. tomorrow)

Thanks for injecting some sanity into my deranged musings.

g
_______________________________________________________________