Extensible Records now in dev-branch! Great for games?

[Evan Czaplicki]

Extensible records now appear in the dev-branch of Elm, so if you sync to the latest version of the compiler on github you can try it out :)

I am pretty excited about this! The paper linked above gives a formal overview, but the syntax in Elm will be a bit different for readability and to avoid some syntactic ambiguities. Here's a general overview of the syntax in Elm (light blue text shows how the expressions are evaluated):

Defining Records. Fields can be polymorphic functions. No need for type signatures or ADTs, just create the record and the types are inferred:

empty = {}

point = { x = 3, y = 4 }

book = { title = "Steppenwolf"

       , author = "Hermann Hesse" }

group1 = { op x y = x + y

         , zero = 0 }

group2 = { op x y = x ++ y

         , zero = [] }

Deleting fields (restriction):

{ point - x } == { y = 4 }

{ book - author } == { title = "Stepenwolf" }

Adding fields (extension):

{ point | z = 12 } == { x = 3, y = 4, z = 12 }

{ book | pages = 150 } == { title = "Steppenwolf", author = "Hermann Hesse", pages = 150 }

Renaming fields:

{ book - title | name = book.title } == { name = "Steppenwolf", author = "Hermann Hesse" }

Replacing values (you can change the type of a field too!):

{ book - title | title = True } == { title = True, author = "Hermann Hesse" }

Prettier and more concise way to replace values. This allows you to batch changes too:

{ point | x <- 0 } == { x = 0, y = 4 }

{ book | title <- "Das Glasperlenspiel" }

{ book | title <- "Harry Potter"

       , author <- "J. K. Rowling" }

(vx,vy) = (10,10)

t = 1

{ point | x <- point.x + vx * t

        , y <- point.y + vy * t } == { x = 13, y = 14 }

You can also do some pattern matching on records when used in function definitions and let-expressions. You can be selective about what fields to access, so it's okay to pattern match on a record with extra fields:

dist {x,y} = sqrt (x^2 + y^2)

dist point == dist { x = 3, y = 4, z = 12 }

Finally you can use arbitrary field accessors. The type of the accessor is inferred from its name and can be used on any record that has that field:

map .author books

map .x points

That's pretty much everything for now! If you give it a try tell me what you think while things are still flexible.

Warning: There is something odd going on with the type-checker, so some type-errors with records don't get caught at the moment. I am hoping to do a general rewrite of the type-checking code to clean up the code and resolve this kind of issue.

Enjoy!
Evan

[Dobes Vandermeer]

Sounds very nice - are there instructions somewhere how best to install from github?  I'm not familiar with cabal or haskell.

[Evan Czaplicki]

No, I should add this to the readme! It should be pretty easy though:

- fork or download project

- navigate to Elm/elm/

- choose how to build:

    • run "cabal install" to install Elm-0.7 on your machine (it takes precedence over the existing version but you can remove it if you want)

    • run "cabal build" to build the executable somewhere in the dist/ directory. This lets you run the compiler without messing with the existing version. Calls to elm will still go to 0.6.0.3, but you can also run the new executable.

That should do it!

Notes: you may want to reinstall elm-server if you install the new compiler ("cabal install elm-server" I believe), no more guarded defs, no more {;;} with let expressions, yes to multi-way ifs:

if | x == y -> EQ

   | x > y -> GT

   | otherwise -> LT

[Dobes Vandermeer]

Now if only there was a simpler way to "lift" signals into a record instead of using a function; i.e. I just wrote:

mkinput t mouseDown mousePosition = {t=t, mouseDown=mouseDown, mousePosition=mousePosition}

input = lift3 mkinput (every 0.1) Mouse.isDown, Mouse.position}

But could that be more like this in some magical kind of way?

input = liftRec {t=(every 0.1), mouseDown=Mouse.isDown, mousePosition=Mouse.position}

[Dobes Vandermeer]

Also here's a little bug report; when I returned a record directly from a function, it generated code with a syntax error:

(can you spot the invisible semicolon?)

function mkinput_1(t_60){

return function(mouseDown_61){

return function(mousePosition_62){

return

{mouseDown : [mouseDown_61],

mousePosition : [mousePosition_62],

t : [t_60],

_ : [true]};};};}

Either moving the opening brace onto the same line as the return or putting an open parenthesis after the return (on the same line) and a close paren after the object is necessary to prevent javascript from parsing that as return ; { ... } which is invalid.

Cheers,

Dobes

[Dobes Vandermeer]

One more idea: could you add a way to "bulk update" a record using another one?  Basically "merge" in a second record as if it were manually copying all the fields over using the <- syntax?

e.g. {a=3,b=7} | {b=3,c=9} == {a=3,b=3,c=9} ?

On Wed, Dec 26, 2012 at 2:57 PM, Evan Czaplicki <eva...@gmail.com> wrote:

[Dobes Vandermeer]

Not sure if this is "really" a bug but it surprised me.

Elm rejects this (parse error):

xga = {

    width=1024,

    height=768

}

But accepts this:

xga = {

         width=1024,

         height=768

 }

Not sure what the rule really is but I thought it would be okay to align the closing brace with the start of the first line.

[Dobes Vandermeer]

Another report: I tried to use pattern matching to read a record's field in a "let" but it didn't take it:

updateGame {t,mouseDown,mousePosition} state =

    let {room} = state

        {bg,items} = room

    in image bg.width bg.height ("images/bg/" ++ bg.image)

Error:

elm: src\Parse\Patterns.hs:(90,3)-(99,26): Non-exhaustive patterns in case

[Evan Czaplicki]

Thanks for the bug reports! Lots of bugs to fix!

For the `liftRec` and `merge` ideas, if you can give them meaningful types and semantics in terms of these extensible records I think they are potentially a good idea. Merge may be possible, but I am pretty sure that liftRec will not be. When would merge be useful in general? What kind of things would people want to put together? Can they not just add one record as a field in the other? How often does this come up?

In general, I am opposed to magic. I also distrust the temptation to add features that look good if they have never appeared in a language or academic paper before. I strongly prefer to have a strong formal basis for language features, either academic or historical. Until there is more formal work and until we see how records are used in the wild, I think it is a bad idea to add a bunch of new semantics to the system. This record system is already more flexible than in any strongly typed functional language I've ever seen (the ML's and Haskell), so it is already a somewhat risky choice. Sorry to be a wet blanket here :/ I really like to see specs, design docs, semantics, types, etc. for this kind of thing!

[Evan Czaplicki]

The let and ; errors should be resolved in dev branch. Even after working with JavaScript for so long, it still finds ways to massively disappoint me.

Looking into the weird whitespace error next!

[Dobes Vandermeer]

On Fri, Dec 28, 2012 at 1:42 AM, Evan Czaplicki <eva...@gmail.com> wrote:

For the `liftRec` and `merge` ideas, if you can give them meaningful types and semantics in terms of these extensible records I think they are potentially a good idea. Merge may be possible, but I am pretty sure that liftRec will not be. When would merge be useful in general? What kind of things would people want to put together? Can they not just add one record as a field in the other? How often does this come up?

For liftRec I believe it would be a better approach than lift3, lift4, etc. because the positional approach is not as robust when you have more than a couple of inputs.  I can convert the positional thing into a record myself, so it's not a big deal.  Maybe a shorthand way of writing a function that converts from positional arguments to a record would do the trick very well, kind of like a "record constructor" of some sort:

mkinput t mouseDown mousePosition = {t=t, mouseDown=mouseDown, mousePosition=mousePosition}

You might be able to write:

mkinput = ({t,mouseDown,mousePosition}) // Works like (,,,) but for building a record instead

For merge I just wanted to make it easier to "compose" a record from a few parts that are themselves a record.  For example if I have some predefined records with x,y,width,height, and functions then I can merge these together into a new object, not unlike prototype-based programming.  As you pointed out, though, I can do this by nesting the common information a bit deeper in the tree.  I'll see if this bugs me more or not, it's definitely more of a convenience than a necessity.

In general, I am opposed to magic. I also distrust the temptation to add features that look good if they have never appeared in a language or academic paper before. I strongly prefer to have a strong formal basis for language features, either academic or historical. Until there is more formal work and until we see how records are used in the wild, I think it is a bad idea to add a bunch of new semantics to the system. This record system is already more flexible than in any strongly typed functional language I've ever seen (the ML's and Haskell), so it is already a somewhat risky choice. Sorry to be a wet blanket here :/ I really like to see specs, design docs, semantics, types, etc. for this kind of thing!

Hey, no problem, thanks for listening to my wild suggestions!  It's a learning process for me as well.

I don't expect instant results, just trying to give some helpful ideas for the future and see what you think.

I have been making a little programming language of my own for making games, that would have good static analysis and checking and probably compile to C++.  I wanted to use mainly records/objects because I find that tuples and argument lists really break down for arity > 2; it's just a pain to keep things in sync and to understand what's what when you don't have names on things.  When I saw this extensible record thing in Elm, I'm excited to see how this kind of system works in practice.  Who knows, if I can get used to the Haskell based syntax and the javascript performance and capabilities are good enough ... Elm might do the trick for my projects.

[Evan Czaplicki]

I didn't understand the {} issue you ran into earlier, but yeah, that is intended behavior for now.

I like the idea of a special record constructor syntax. The trouble is going to be thinking of a way to do it in a way that is not too syntactically ambiguous. I think your suggestion would potentially work, but it's not super clear that it is a function. I'll think more about this because this could be really convenient.

Some languages have a _ syntax like this:

(_ + _) == (\x y -> x + y)

(_^2) == (\n -> n^2)

This would let you say something like { x = _, y = _ } to get the constructor you want. I am not sure if I like this approach because it can be abused pretty easily (using really long expressions), but it could be a good option.

I am excited to hear how your experience with Elm goes :) Hopefully it can suit your needs! If not, please tell me where it fails you so I can try to improve on that! As a side-note, Elm can theoretically be compiled to C++ or whatever other language without any trouble; the hard part would be getting things working with some GUI lib!

[Dobes Vandermeer]

Hi,

I did think about that underscore thing while composing that other email but it's also not super obvious it is a function either.  I guess the underscores may reduce the number of possible interpretations of that syntax so someone unfamiliar with the language could figure it out faster.

I had also wondered whether there was a variation on your ".foo" syntax for having accessors as a standalone function.  Perhaps the inverse of this would be to define functions like this:

.foo: = (\ row v -> {row | foo = v})

.bar: = (\ row v -> {row | bar = v})

.baz: = (\ row v -> {row | baz = v})

mkrec = (.foo: . .bar: . .baz:) {} 

rec = mkrec 1 2 3

Unfortunately those functions won't really compose properly that way ....  So a different operator besides "." would have to be defined - some sort of "folding compose"- for this purpose, it gets a bit clunky I think.

Since it's not something that would see a *ton* of use it might be OK to use a longer syntax for it, perhaps some sort of keyword for it. Or just ignore the problem ...

[Evan Czaplicki]

Maybe something like this:

{\x y z} 3 4 12 == { x = 3, y = 4, z = 12 }

{\title author} "Demain" "Hesse"

lift2 {\x y} Mouse.x Mouse.y

Also, with the new (<~) and (~) syntax, this can become:

{\x y} <~ Mouse.x ~ Mouse.y

I kind of like this, but I want to be cautious about adding new syntax. Taking things out of a language is super hard, so I'd prefer to wait until I've written/seen more record code to know how much of a win this would be and how it could be negative. I'd also like to hear more proposals on syntax for this. I have not seen a feature like this before, so I just want to think this through as much as possible.

A downside here is that it is pretty close to the pattern matching syntax:

\{x,y} -> x + y

[Dobes Vandermeer]

Something to ruminate on I guess.  If it was me I'd throw some thoughts into a ticket on github and see if any more comments come in, or ideas arise, then maybe fix it a couple months later if it still seems like a good idea by then.

[Evan Czaplicki]

Good idea. I added an issue on github the other day.

[Grzegorz Balcerek]

Hi,

> When would merge be useful in general? What kind of things would people want to put together? Can they not just add one record as a field in the other? How often does this come up?

How about this:

One could keep configuration data in records and use one record with default values and another with some (but not all) specific values.

Then by merging both records one would get the result record with the specific values from the second recornd and the defaults for thoses values which are not present in the second record.

Grzegorz

[Evan Czaplicki]

Ah, I like this a lot! Here is an example:

defaults = { color = red, scale = 1, ... }

render  options obj = render' { defaults + options } obj

render' options obj =

  {- actually render things -}

This would let you say:

render {} obj                     -- for defaults

render { color = blue } obj       -- to change just the color

render { scale = 3    } obj       -- to change just the scale

So effectively allowing optional arguments in a fairly clear way! This is very very cool! One downside is that you cannot really check for typos in the optional arguments part (e.g. what if someone says colour instead of color?). Is this what you had in mind?

Without record concatenation, optional args would look like this:

render defaults obj

render { defaults | color <- blue } obj

render { defaults | scale <- 3    } obj

This gives much more type-level guidance about what values you can and cannot change. It is heavier though.

Another cool use for record concatenation is "multiple inheritance"  so one composite record can use functions from each of its parts without modification. This idea came out of this discussion.

[Evan Czaplicki]

Perhaps a batch replace function might be nicer in this case because it would capture the type constraints:

{ defaults <- options }

so every field in the options record must also be in the defaults record. In the case of optional arguments, this would also ensure that the user uses values of the correct type.

[Dobes Vandermeer]

Yes, this is exactly the kind of use case and solution I was thinking of!

[Evan Czaplicki]

Cool, let's try to write up some examples using { a + b } and { a <- b } to see when they are useful and when they are not quite right. If I can see three or four really good examples, I'll be much happier with this idea. My fear is that we are not seeing the crappy parts of these ideas yet. To clear up semantics of these two operations:

origin   = { x = 0, y = 0 }

point    = { x = 3, y = 4 }

velocity = { angle = 0, velocity = 0 }

obj  = { origin + velocity }

obj' = { obj <- point }

One use of records is to create really flexible modules. I am not sure if this helps or hinders. Also, I do not want to allow anything but variable names in the following blanks { _ + _ } and { _ <- _ } because things would be nearly impossible to parse unambiguously without tons of bullshit rules. Better to keep it simple.

[Evan Czaplicki]

Oh, the point of that example was supposed to be that { a + b } is biased towards b, so { origin + point }.x == 3

The idea being that you always add things to the right: { r | x = 0 } So

point = { x = 0, y = 0 }

point = { { {} | x = 0 } | y = 0 }

{ r + point } == { { r | x = 0 } | y = 0 }

So access can be thought of as:

r.x = if r == { r' | x = v } then v else r'.x

This is of course not how it works and this is not valid code, but it works as a mental model.