Strings and things

[candlerb]

I've not looked at Reia for a while, but I like the new website, and
I've just built the latest git master. A bunch of tests fail, but it
runs. It's looking very impressive.

I notice from the website:

# Strings (unlike Erlang, Reia has a real String type!)
string = "I'm a string! Woohoo I'm a string! #{'And I interpolate
too!'}"

I think it ought to mention <["binaries"]> too, as I had to go into
the test code to find the syntax.

Also, what's a .reb file, and is it possible to convert it to .beam?
Does it contain an erlang abstract parse tree? I'd be interested in
seeing what reia creates in terms of erlang, but if I rename .reb
to .beam it's not readable.

2> beam_lib:chunks("x", [abstract_code]).
{error,beam_lib,{not_a_beam_file,'x.beam'}}

Finally, how do you call an erlang bif from reia?

>> x = "foo".to_list()
=> [102,111,111]
>> list_to_existing_atom(x)
reia_eval#1290_602175_189618:1: function list_to_existing_atom/2
undefined
exception throw: {error,"internal Erlang compiler error"}
in function reia_bytecode:compile/3
in call from reia_eval:exprs/2
in call from ire:eval/2
in call from ire:loop/1
>> erlang:list_to_existing_atom(x)
Error: Line 1: syntax error before: list_to_existing_atom
>> Erlang.list_to_existing_atom(x)
exception error: undefined function 'Erlang':list_to_existing_atom/2
in function 'reia_eval#1290_602303_39603':toplevel/2
in call from reia_bytecode:load/2
in call from reia_eval:exprs/2
in call from ire:eval/2
in call from ire:loop/1

Thanks,

Brian.

[candlerb]

> Finally, how do you call an erlang bif from reia?

Don't worry about that one, I got it:

>> erl.list_to_existing_atom(x)
=> :foo

This gives a useful way to look at the internal representation of
reia's data structures:

>> erl.io.write(123)
123=> :ok
>> erl.io.write("foo")
{reia_string,[<<102,111,111>>]}=> :ok
>> erl.io.write({1=>2})
{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
[]},{{[],[],[],[],[],[],[],[],[],[],[],[[1|2]],[],[],[],[]}}}=> :ok

I see that reia strings are basically iolists. I guess that's for fast
concatenation, except I can't see how to concatenate in reia (both '+'
and '<<' are treated as arithmetic operators). ISTR that erlang R12B+
implements efficient concatenation of binaries anyway.

I notice that converting a binary to a string doesn't wrap it in a
list, is this intentional?

>> erl.io.write("foo")
{reia_string,[<<102,111,111>>]}=> :ok
>> erl.io.write(<["foo"]>.to_s())
{reia_string,<<102,111,111>>}=> :ok

Also, is there any run-time introspection of available methods on an
object?

>> <["xxx"]>.to_s()
=> "xxx"
>> <["xxx"]>.methods()
NoMethodError: undefined method 'methods' for Binary

Regards,

Brian.

[Tony Arcieri]

On Wed, Nov 24, 2010 at 5:53 AM, candlerb <b.ca...@pobox.com> wrote:

I've not looked at Reia for a while, but I like the new website, and
I've just built the latest git master. A bunch of tests fail, but it
runs. It's looking very impressive.

Awesome, great work. Is it available on github?

 

I notice from the website:

   # Strings (unlike Erlang, Reia has a real String type!)
   string = "I'm a string! Woohoo I'm a string! #{'And I interpolate
too!'}"

I think it ought to mention <["binaries"]> too, as I had to go into
the test code to find the syntax.

Yeah, this has come up a few times in recent history now. I'm going to add an additional section that goes over all the builtin types and their syntax.

 

Also, what's a .reb file, and is it possible to convert it to .beam?

It's a collection of all of the classes and modules contained within a particular file (including the toplevel function). Every class and module compiles to its own .beam file, and the entire collection is stored in a .reb file.

 

Does it contain an erlang abstract parse tree? I'd be interested in
seeing what reia creates in terms of erlang, but if I rename .reb
to .beam it's not readable.

What I think would be great for that is an option like Mirah. With mirahc you can do mirahc --java and have it output Java source code. I've been meaning to add an option to reiac ala reiac --erl that would output pretty printed Erlang source code.

I see that reia strings are basically iolists. I guess that's for fast
concatenation, except I can't see how to concatenate in reia (both '+'
and '<<' are treated as arithmetic operators).

Yeah that needs to get added as a method to the String class (perhaps String#concat)

 

ISTR that erlang R12B+ implements efficient concatenation of binaries anyway.

That likewise needs to be added as a method to Binary (Binary#concat I guess). Going down the road of having Strings be binaries internally is another option. iolists seemed like the POLS for Erlang, though. Many functions that work on strings spit out lists rather than binaries, and using iolists lets you concatenate them with binaries and not have to flatten everything out until absolutely necessary. It does, however, preclude pattern matching on strings, so maybe it'd be worth it to eagerly flatten everything out to binaries and leverage the pattern matching available on binaries.

 

I notice that converting a binary to a string doesn't wrap it in a
list, is this intentional?

>> erl.io.write("foo")
{reia_string,[<<102,111,111>>]}=> :ok
>> erl.io.write(<["foo"]>.to_s())
{reia_string,<<102,111,111>>}=> :ok

Yes, my understanding is that a naked binary is still a valid iolist.

 

Also, is there any run-time introspection of available methods on an
object?

>> <["xxx"]>.to_s()
=> "xxx"
>> <["xxx"]>.methods()
NoMethodError: undefined method 'methods' for Binary

Not yet, unfortunately, but I'd definitely like to add them. For now your best bet is probably to look at the source code under src/builtins. I'd like to figure out how to run something like RDoc or Docco on it and at least spit out some searchable HTML documentation.

 

--
Tony Arcieri
Medioh! A Kudelski Brand

[candlerb]

On Nov 24, 5:08 pm, Tony Arcieri <tony.arci...@medioh.com> wrote:

> On Wed, Nov 24, 2010 at 5:53 AM, candlerb <b.cand...@pobox.com> wrote:
> > I've not looked at Reia for a while, but I like the new website, and
> > I've just built the latest git master. A bunch of tests fail, but it
> > runs. It's looking very impressive.
>
> Awesome, great work. Is it available on github?

Is what available on github? All I did was build your code from
git://github.com/tarcieri/reia.git

Here are the tests which fail:

$ rake
(in /v/brian/git/reia)
Checking Erlang version... 5.7.4 (ok)
bin/reia test/runner.re
EE..............................E..........EEEEEE.....E.........................

builtins/binary.re ERROR: undef

builtins/boolean.re ERROR: undef

builtins/tuple.re ERROR: undef

builtins/range.re ERROR: undef

builtins/regexp.re ERROR: undef

builtins/numeric.re ERROR: undef

builtins/dict.re ERROR: undef

builtins/fun.re ERROR: undef

core/blocks.re ERROR: undef

core/funrefs.re ERROR: undef

Finished in 0.085041 seconds

78 assertions, 0 failures, 10 errors
rake aborted!
Command failed with status (1): [bin/reia test/runner.re...]
/v/brian/git/reia/Rakefile:92
(See full trace by running task with --trace)

> I've been
> meaning to add an option to reiac ala reiac --erl that would output pretty
> printed Erlang source code.

That would be great. I was just trying to get at the abstract parse
tree so I could erl_pp it.

As far as I can tell, this has to be read from the .beam file, not the
loaded chunk in RAM: see
http://erlang.2086793.n4.nabble.com/Help-with-erl-parse-epp-td2108447.html

Looking for a function which can split a .reb into chunks, I found
reia_internal:load. Unfortunately this loads and executes, so I'd need
to reimplement the logic just to load the abstract code chunk.

This gives something interesting to try in reia itself -
unfortunately, as it uses record types, I think I'll have to deal with
them as tuples (I don't suppose reia supports records?)

$ bin/ire
...
>> (:ok, data) = erl.file.read_file("lib/uuid.reb".to_list()); (data.class(), data.size())
=> (Binary,6120)
### Using my knowledge of the record format
>> (:reia_module, version, filename, base_module) = erl.binary_to_term(data)
...
>> filename
=>
[47,118,47,98,114,105,97,110,47,103,105,116,47,114,101,105,97,47,108,105,98,47,117,117,105,100,46,114,101]
### Not sure how to turn this into a reia string now:
>> String(filename).puts()
NoMethodError: undefined method 'initialize' for String
>> (:reia_string, filename).puts()
/v/brian/git/reia/lib/uuid.re

>> (:ok, (mod, chunks)) = erl.beam_lib.chunks(base_module, [:abstract_code]); mod
=> :'/v/brian/git/reia/lib/uuid'
>> erl.length(chunks)
=> 1
>> (:abstract_code, (vers,forms)) = chunks[0]; nil
=> nil
>> forms.size()
=> 6
>> erl.erl_pp.form(forms[0])
=> [[[[[45,[109,111,100,117,108,101]]],[[40,
[[[39,47,118,47,98,114,105,97,110,47,103,105,116,47,114,101,105,97,47,108,105,98,47,117,117,105,100,39]],
41]]]],[46,10]]]
### Again, would be nice to have a simple way to wrap this iolist in a
String
>> (:reia_string, erl.erl_pp.form(forms[0]))
=> "-module('/v/brian/git/reia/lib/uuid').
"
>> forms.each() { |form| erl.io.fwrite(<["~s"]>, erl.erl_pp.form(form)) }; nil
-module('/v/brian/git/reia/lib/uuid').
-file("/v/brian/git/reia/lib/uuid.re", 1).
-code([{class,
9,
'UUID',
'Object',
[{function,
11,
initialize,
[],
{var,1,'_'},
[{match,11,{ivar,11,ref},{native_call,11,erlang,make_ref,
[]}}]},
...
-parent('/v/brian/git/reia/lib/uuid').
-submodules([{static,
'UUID',
<<70,
79,
82,
49,
0,
0,
10,
52,
66,
69,
65,
77,
65,
116,
111,
109,
0,
0,
0,
...
159,
0,
0,
0>>}]).
toplevel() ->
reia_internal:load_submodule('UUID', module_info(attributes)).

Hmm, I didn't learn as much about the reia compiled code from that as
I thought I would. What are the -code and -submodule sections?

But I did have a useful play with ire talking to erlang. It can be a
bit awkward, but still easier than the erl shell :-)

One thing I learnt to be careful of when mapping erlang to reia is to
remember the : for atoms. It's easy to write:
(reia_module, version, filename, base_module) =
erl.binary_to_term(data)
instead of
(:reia_module, version, filename, base_module) =
erl.binary_to_term(data)

Both work, but the first case would also match things that don't start
with the :reia_module atom.

> Going down the road of having Strings be binaries internally is
> another option. iolists seemed like the POLS for Erlang, though.

If Strings were binaries internally it would be easier to deal with
native erlang code which accepts or returns binaries - such as the
format argument to erl.io.fwrite, or the data returned by
erl.file.file_read

Some methods return iolists anyway, like erl_pp:form, but you still
have to wrap them in (reia_string, ...) to use them as Strings in
reia.

If strings were binaries, and you wanted lazy flattening, you could
always write
mylist = [s1, s2]
explicitly (if a bare binary is also an iolist, this would be
acceptable to functions which take iolists). Or perhaps String could
be a native binary, but you also have an (:iolist, ...) class which
works like String today.

> > I notice that converting a binary to a string doesn't wrap it in a
> > list, is this intentional?
>
> > >> erl.io.write("foo")
> > {reia_string,[<<102,111,111>>]}=> :ok
> > >> erl.io.write(<["foo"]>.to_s())
> > {reia_string,<<102,111,111>>}=> :ok
>
> Yes, my understanding is that a naked binary is still a valid iolist.

Then perhaps for efficiency the string literal "foo" should be
(reia_string,<["foo"]>) instead of (reia_string,[<["foo>]"]) ?

Cheers,

Brian.

[Tony Arcieri]

On Sun, Nov 28, 2010 at 3:47 AM, candlerb <b.ca...@pobox.com> wrote:

Here are the tests which fail:

$ rake
(in /v/brian/git/reia)
Checking Erlang version... 5.7.4 (ok)
bin/reia test/runner.re
EE..............................E..........EEEEEE.....E.........................

78 assertions, 0 failures, 10 errors

Strange. Unfortunately that isn't much of an error to go on. I'll see if I can't make that print out a bit more information.

It sounds like you might be having a similar problem to some other people though.

 

Looking for a function which can split a .reb into chunks, I found
reia_internal:load. Unfortunately this loads and executes, so I'd need
to reimplement the logic just to load the abstract code chunk

I'd eventually like to implement something like Module.to_parsetree() that allows you to retrieve the parse tree for a particular module.

This gives something interesting to try in reia itself -
unfortunately, as it uses record types, I think I'll have to deal with
them as tuples (I don't suppose reia supports records?)

No, Reia doesn't support records. Reia's immutable objects are intended as a replacement for records. As you said, the only option for working with records in Reia is to treat them as naked tuples.

 

$ bin/ire
...
>> (:ok, data) = erl.file.read_file("lib/uuid.reb".to_list()); (data.class(), data.size())
=> (Binary,6120)
### Using my knowledge of the record format
>> (:reia_module, version, filename, base_module) = erl.binary_to_term(data)
...
>> filename
=>
[47,118,47,98,114,105,97,110,47,103,105,116,47,114,101,105,97,47,108,105,98,47,117,117,105,100,46,114,101]
### Not sure how to turn this into a reia string now:

You can do filename.to_string()

 

>> String(filename).puts()
NoMethodError: undefined method 'initialize' for String

String should probably support initialize.

 

### Again, would be nice to have a simple way to wrap this iolist in a
String

List#to_string should work here as well.

 

Hmm, I didn't learn as much about the reia compiled code from that as
I thought I would. What are the -code and -submodule sections?

-code is the original Reia AST

-submodules are separate .beam files for every class or module declared within a .re file. Each class/module is compiled to its own .beam file.

 

But I did have a useful play with ire talking to erlang. It can be a
bit awkward, but still easier than the erl shell :-)

Hehe

 

> Going down the road of having Strings be binaries internally is
> another option. iolists seemed like the POLS for Erlang, though.

If Strings were binaries internally it would be easier to deal with
native erlang code which accepts or returns binaries - such as the
format argument to erl.io.fwrite, or the data returned by
erl.file.file_read

Some methods return iolists anyway, like erl_pp:form, but you still
have to wrap them in  (reia_string, ...) to use them as Strings in
reia.

If strings were binaries, and you wanted lazy flattening, you could
always write
  mylist = [s1, s2]
explicitly (if a bare binary is also an iolist, this would be
acceptable to functions which take iolists). Or perhaps String could
be a native binary, but you also have an (:iolist, ...) class which
works like String today.

I really don't want lazy flattening. Strings as iolists make comparisons hard, preclude pattern matching, and mean that the user can end up triggering a lazy flattening operation repeatedly without knowing it, since right now Reia has no way to memoize the flattened version.

I'm really starting to warm up to the idea that strings should be binaries internally, although the question remains as to whether or not having a separate binary type is still a good idea. I think the answer to that is yes, and that I'd want Reia's strings to be encoding aware and always force UTF-8 internally, whereas binaries can just be arbitrary binary data.

Then perhaps for efficiency the string literal "foo" should be

(reia_string,<["foo"]>) instead of (reia_string,[<["foo>]"]) ?

There's a lot of things Reia could do "for efficiency" that it's not doing right now :) 

[candlerb]

On Nov 29, 9:23 pm, Tony Arcieri <tony.arci...@medioh.com> wrote:
> I'm really starting to warm up to the idea that strings should be binaries
> internally, although the question remains as to whether or not having a
> separate binary type is still a good idea. I think the answer to that is
> yes, and that I'd want Reia's strings to be encoding aware and always force
> UTF-8 internally, whereas binaries can just be arbitrary binary data.

But how would it behave differently, depending on whether the data is
UTF-8 text or arbitrary binary data?

As you probably know, I'm not exactly a fan of the ruby 1.9 world
where every string is tagged with an encoding, with totally
undocumented semantics for every operation which consumes or returns a
string.

The erlang way (eep-0010, eep-0011) seems to be:

* binaries which contain text can contain utf-8 or latin1. (latin1,
being a fixed single-byte encoding, is effectively just binary)
* when the operation needs to know which, you say explicitly at the
point of operation: e.g. "~s" vs "~ts" for format string, "/utf8"
modifier for matching characters in binaries, "unicode" flag to re for
regexp matching.
* if you want to handle any other encoding, you transcode it to utf-8
at the boundary

Hence the binary itself is just a binary and needs no extra dimension,
and the semantics are defined for the functions which need to know if
it's bytes or UTF-8 characters.

Regards,

Brian.

[Tony Arcieri]

On Mon, Dec 6, 2010 at 5:26 AM, candlerb <b.ca...@pobox.com> wrote:

But how would it behave differently, depending on whether the data is

UTF-8 text or arbitrary binary data?

Yes, and beyond that, string would have string-specific methods for working on characters, as opposed to arbitrary binary data.

 

As you probably know, I'm not exactly a fan of the ruby 1.9 world
where every string is tagged with an encoding, with totally
undocumented semantics for every operation which consumes or returns a
string.

I don't like that either. I'd rather all strings be UTF-8. As UTF-8 seems to be the emerging standard for character encoding, it would eliminate all of the transcoding that UTF-16 based languages (e.g. Java) have to go through.

 

The erlang way (eep-0010, eep-0011) seems to be:

* binaries which contain text can contain utf-8 or latin1. (latin1,
being a fixed single-byte encoding, is effectively just binary)
* when the operation needs to know which, you say explicitly at the
point of operation: e.g. "~s" vs "~ts" for format string, "/utf8"
modifier for matching characters in binaries, "unicode" flag to re for
regexp matching.

I'd rather that be automatic, with strings "knowing" they're UTF-8.

--
Tony Arcieri
Medioh! Kudelski

[candlerb]

On Dec 6, 6:31 pm, Tony Arcieri <tony.arci...@medioh.com> wrote:
> > The erlang way (eep-0010, eep-0011) seems to be:
>
> > * binaries which contain text can contain utf-8 or latin1. (latin1,
> > being a fixed single-byte encoding, is effectively just binary)
> > * when the operation needs to know which, you say explicitly at the
> > point of operation: e.g. "~s" vs "~ts" for format string, "/utf8"
> > modifier for matching characters in binaries, "unicode" flag to re for
> > regexp matching.
>
> I'd rather that be automatic, with strings "knowing" they're UTF-8.

I wouldn't, as it leads to all sorts of complexities:

* How do strings get assigned their initial encoding? (String
literals, strings read from stdin, strings read from files, strings
from the O/S like directory listings, ...)
* What happens if string with encoding A is concatenated with a string
of encoding B? What about with the empty string? In general, for every
method which takes string argument(s), how is its behaviour modified
dependent on the encoding of those string(s)?
* For every method which *returns* a string, how is the encoding of
that string chosen?
* Repeat for all system libraries and all third-party libraries

Even if Reia has only a one-bit flag, or two classes (utf8/binary),
these issues all still arise. And yet at the point of interacting with
the operating system, for example when calling read() or readdir(),
strings are just binary data. So all this hidden state has to be
synthesised when data arrives, and thrown away when data leaves.

[Tony Arcieri]

On Mon, Dec 6, 2010 at 2:48 PM, candlerb <b.ca...@pobox.com> wrote:

> I'd rather that be automatic, with strings "knowing" they're UTF-8.

I wouldn't, as it leads to all sorts of complexities:

* How do strings get assigned their initial encoding? (String
literals, strings read from stdin, strings read from files, strings
from the O/S like directory listings, ...)

What does that have to do with how strings are stored internally? Even if I don't stipulate that strings are stored as UTF-8 internally this problem doesn't go away.

I believe the general wisdom here is to assume UTF-8 unless specified otherwise via a mechanism like magic comments.

 

* What happens if string with encoding A is concatenated with a string
of encoding B?

I'm suggesting all strings are stored as UTF-8 internally, so this problem simply doesn't exist. The only encoding allowed is UTF-8. This applies to your remaining questions as well. If you have a string, then internally it's encoded as UTF-8. Period. No other encodings are allowed.

 

Even if Reia has only a one-bit flag, or two classes (utf8/binary),
these issues all still arise.

I'd like to have separate String and Binary data types.

 

And yet at the point of interacting with the operating system, for example when calling read() or readdir(),
strings are just binary data. So all this hidden state has to be
synthesised when data arrives, and thrown away when data leaves.

Anything that does I/O can return binaries that have to be explicitly converted into strings (with a provided encoding), or you could potentially provide this encoding ahead of time and have all binaries processed with the given encoding and transcoded into UTF-8 if necessary. 

[candlerb]

On Dec 6, 9:55 pm, Tony Arcieri <tony.arci...@medioh.com> wrote:
> > * How do strings get assigned their initial encoding? (String
> > literals, strings read from stdin, strings read from files, strings
> > from the O/S like directory listings, ...)
>
> What does that have to do with how strings are stored internally?

Well, you say later that you want to have separate classes for String
and Binary. So it affects how strings are stored internally in the
sense that you have to choose whether a particular datum is a String
or a Binary. That's pretty much equivalent to having a String with a
one-bit internal flag saying "utf-8" or "binary".

> > * What happens if string with encoding A is concatenated with a string
> > of encoding B?
>
> I'm suggesting all strings are stored as UTF-8 internally, so this problem
> simply doesn't exist.

But you're having Strings and Binaries. So the question then becomes:
what happens if you try to concatenate a String with a Binary, or vice
versa?

* If this always fails, that's fine (you need to provide a mechanism
to coerce a String into a Binary, and vice versa).
* If this always succeeds, that's OK, except the problem is then
whether the result is a String or a Binary.
* If you take the ruby 1.9 approach, where it succeeds only sometimes
(e.g. if one of the strings is empty, or contains only ASCII
characters), then you have a dog's breakfast.

> > Even if Reia has only a one-bit flag, or two classes (utf8/binary),
> > these issues all still arise.
>
> I'd like to have separate String and Binary data types.

Sure, that's your one-bit encoding flag. The questions become:

* In the source code, which literals are String, and which are Binary?
(different syntax for the two?)
* When you read from a file or from stdin or from a socket, do you get
a String or a Binary? (different open mode?)
* When you call an OS function like readdir(), does it return a String
or a Binary?
* When you call a function like to_s() or pack(), does it return a
String or a Binary?
* Which of String or Binary is going to be a native erlang binary? Or
neither?
And so on.

These questions all vanish if you just have binaries, and say at the
point of use whether you want it to be treated as utf8 or binary data
(which in many cases doesn't need to be specified anyway, e.g.
concatenation)

Regards,

Brian.

[Tony Arcieri]

On Tue, Dec 7, 2010 at 7:33 AM, candlerb <b.ca...@pobox.com> wrote:

But you're having Strings and Binaries. So the question then becomes:

what happens if you try to concatenate a String with a Binary, or vice
versa?

You should have to specify the encoding of the binary, so it can be transcoded into UTF-8.

The questions become:

* In the source code, which literals are String, and which are Binary?
(different syntax for the two?)

Per the current syntax:

"I'm a string!"

<["I'm a binary!"]> # (this is effectively the same as Erlang's <<>> syntax, except with <[]>)

I'm not sure I really like that syntax though.

 

* When you read from a file or from stdin or from a socket, do you get
a String or a Binary? (different open mode?)

A binary, unless the file was specifically opened so as to return Strings (and configured with a specific encoding)

 

* When you call an OS function like readdir(), does it return a String
or a Binary?

Strings 

 

* When you call a function like to_s() or pack(), does it return a
String or a Binary?

I don't know if I want to provide pack/unpack. I think they can be done with binary pattern matching.

to_s() should, of course, return Strings

 

* Which of String or Binary is going to be a native erlang binary? Or
neither?

Binaries

 

These questions all vanish if you just have binaries, and say at the
point of use whether you want it to be treated as utf8 or binary data
(which in many cases doesn't need to be specified anyway, e.g.
concatenation)

I don't want the questions to all go away and their answers left as an exercise to the end user.

From a purely theoretical perspective strings are a collection of characters, and I want them to have a separate identity and for their methods to work on characters. Binaries are collections of bits.

-- 

Tony Arcieri
Medioh! Kudelski