Any new updates?
Mike Austin
Mike
Bob Nystrom
> Just curious how Magpie is going. You're on to something with pattern
> matching and multi-methods, I hope it can bloom into something.
Yeah, I suppose a Magpie State of the (Tiny) Union is in order since
things have been quiet for a while.
TL;DR:
* Multimethods and patterns are awesome.
* I need to figure out how multimethods are scoped.
* Might ditch the extensible syntax and do Smalltalk-style blocks instead.
* I want to ditch the JVM.
I definitely haven't given up on Magpie. I think about it every day,
and spend some chunk of free time most days hacking on
language-related stuff that will likely eventually work its way back
to Magpie. Most recently, I've been writing a little bytecode-compiler
for a toy language (https://github.com/munificent/bulfinch) so that I
can wrap my head around register-based VMs. That will help as I start
writing a real implementation of Magpie. (The current Java one is
mostly a prototype while I explore the language design.)
Also, I've been playing with Finch for a while. That helps me brush up
on my rusty C++ skills, and it also lets me play with some features
Magpie doesn't have. Right now, I'm trying to get what I learned with
Bulfinch and turn Finch's currently-stack-based bytecode into
something register-based.
So Magpie...
Scoping
Part of the reason progress has slowed on Magpie is that I've hit a
stumbling block. I'm still not sure exactly what the scoping semantics
for multimethods should be (as the last couple of threads here
indicate).
Purely lexical like it is now makes some stuff awesome (safe "monkey
patching", user-defined operators) but other stuff really nasty
(overriding and duck typing). Object-based dispatch like most OOP
languages makes different stuff awesome (overridding, duck typing,
accessing properties) but makes other stuff nasty (symmetric
operators, monkey-patching).
The current semantics don't feel right to me, but I'm not sure what's
"righter" yet.
Extensible Syntax
One thing that's been fun about playing with Finch is it gave me a
chance to write code in a language that has keyword selectors, blocks,
and non-local returns. I found that those three features together make
it *so easy and awesome* to create your own flow-control-like
structures. (Smalltalkers and Rubyists know this already).
For example, here's the C++ code (more or less) for parsing operator
expressions in Finch:
operator {
expr <- self unary
self while-match: Token/Operator do: {|operator|
right <- self unary
expr <-- Expr message-receiver: expr name: operator text args: #[right]
}
expr
}
The `while-match:do:` method is essentially a flow-control structure
that loops repeatedly while consuming matching operator tokens. That
was cake to implement and makes the code really easy to read (I
think).
This is exactly the kind of stuff I wanted Magpie's extensible syntax
to enable but I found myself never using that in practice. Mucking
with the grammar just feels too heavyweight so I never thought to take
it out of the toolbox. In Finch, it's just natural and works.
Having an extensible grammar also makes a bunch of other stuff harder:
tooling, syntax highlighting, compiling. I like that it's really
powerful, and I think it was cool to show that it can be done, but I
don't know if it's really shown enough value to merit its complexity.
So one thing I'm considering is simplifying Magpie to have a fixed
grammar, and then add Smalltalk-style blocks, non-local returns and
keyword selectors. Given that I use 'with' a lot in Magpie (and never
actually use 'fn'), I think blocks are a good fit. Magpie's focus on
methods, I think, makes non-local returns make sense too. Having
blocks and non-local returns really begs for something like keyword
selectors so you can use them like:
if: (blah) then: {
do something
} else: {
something else
}
Taking out all of the syntax extension stuff would make the language
much simpler and make it *way* easier to write an implementation that
compiled to bytecode.
The tricky part is coming up with a syntax that plays nice with
multimethods and (hopefully) records. One thing I've found in the
Magpie code I've written is that I rarely use records/tuples as first
class objects and usually just use them to do overloading, like:
"blah" substring(from: 1, to: 2)
"blah" substring(from: 1, count: 3)
"blah" substring(from: 1)
Keyword selectors in Smalltalk accomplish the same thing without
boxing. I still think having a compact syntax for anonymous composite
objects is important, and *especially* having corresponding
destructuring support for them in pattern matching. But I don't know
if it's still critical to base all method dispatch on them. (Don't get
me wrong, though, I like the conceptual unification this gives Magpie.
But sometimes you can go *too* far down the "everything-is-a-___"
unification path.)
So what I'm currently thinking is:
1. Hack on whatever while my back-brain tries to figure out some
solution to Magpie's scoping problems that makes the basic use cases I
care about work.
2. Replace the extensible syntax craziness with blocks and non-local returns.
3. Write a solid implementation in C++.
This is the path I'm on right now. I am heading down it, but I'm going
very slowly given the limited free time I have. That's OK: there's no
big rush on this.
How does that sound?
- bob
Mike Austin
I guess it'll take a small complete app to test what feels right. Maybe a pluggable text based game engine or something? Pluggable meaning there is a core engine and you just add on top of it.
Extensible Syntax
One thing that's been fun about playing with Finch is it gave me a
chance to write code in a language that has keyword selectors, blocks,
and non-local returns. I found that those three features together make
it *so easy and awesome* to create your own flow-control-like
structures. (Smalltalkers and Rubyists know this already).
For example, here's the C++ code (more or less) for parsing operator
expressions in Finch:operator {
expr <- self unary
self while-match: Token/Operator do: {|operator|
right <- self unary
expr <-- Expr message-receiver: expr name: operator text args: #[right]
}
expr
}The `while-match:do:` method is essentially a flow-control structure
that loops repeatedly while consuming matching operator tokens. That
was cake to implement and makes the code really easy to read (I
think).
begin repeat(IdentifierToken) do |argname|
argnames << argname.float
end end while option(CommaToken)
So much tighter and readable than my previous C++ versions, so I agree blocks are awesome!
This is exactly the kind of stuff I wanted Magpie's extensible syntax
to enable but I found myself never using that in practice. Mucking
with the grammar just feels too heavyweight so I never thought to take
it out of the toolbox. In Finch, it's just natural and works.Having an extensible grammar also makes a bunch of other stuff harder:
tooling, syntax highlighting, compiling. I like that it's really
powerful, and I think it was cool to show that it can be done, but I
don't know if it's really shown enough value to merit its complexity.So one thing I'm considering is simplifying Magpie to have a fixed
grammar, and then add Smalltalk-style blocks, non-local returns and
keyword selectors. Given that I use 'with' a lot in Magpie (and never
actually use 'fn'), I think blocks are a good fit. Magpie's focus on
methods, I think, makes non-local returns make sense too. Having
blocks and non-local returns really begs for something like keyword
selectors so you can use them like:if: (blah) then: {
do something
} else: {
something else
}
if (blah) then (
do something
) else (
something else
)
Taking out all of the syntax extension stuff would make the language
much simpler and make it *way* easier to write an implementation that
compiled to bytecode.The tricky part is coming up with a syntax that plays nice with
multimethods and (hopefully) records. One thing I've found in the
Magpie code I've written is that I rarely use records/tuples as first
class objects and usually just use them to do overloading, like:"blah" substring(from: 1, to: 2)
"blah" substring(from: 1, count: 3)
"blah" substring(from: 1)Keyword selectors in Smalltalk accomplish the same thing without
boxing. I still think having a compact syntax for anonymous composite
objects is important, and *especially* having corresponding
destructuring support for them in pattern matching. But I don't know
if it's still critical to base all method dispatch on them. (Don't get
me wrong, though, I like the conceptual unification this gives Magpie.
But sometimes you can go *too* far down the "everything-is-a-___"
unification path.)
while: {x < 10} do: {
x = x + 1
}
In this case you need to use a block for the text, while if: takes an expression. if: could take a block to make it more orthogonal, I guess, but the point is it pushes the decision onto the programmer. In Ruby, an if and a while block look very similar - it handles the details that one is actually using a lazy expression.
So what I'm currently thinking is:
1. Hack on whatever while my back-brain tries to figure out some
solution to Magpie's scoping problems that makes the basic use cases I
care about work.2. Replace the extensible syntax craziness with blocks and non-local returns.
3. Write a solid implementation in C++.
This is the path I'm on right now. I am heading down it, but I'm going
very slowly given the limited free time I have. That's OK: there's no
big rush on this.How does that sound?
- bob
Bob Nystrom
> I guess it'll take a small complete app to test what feels right. Maybe a
> pluggable text based game engine or something? Pluggable meaning there is a
> core engine and you just add on top of it.
Yeah. I've written some random little sample programs and a more or
less complete lexer and parser so I'm gradually accumulating
experience, but more programs are always better. Strangely enough, I
actually did start writing a little text-based game in Magpie a while
back but didn't put more than an hour or two into it.
>
>>
>> Extensible Syntax
>>
>> One thing that's been fun about playing with Finch is it gave me a
>> chance to write code in a language that has keyword selectors, blocks,
>> and non-local returns. I found that those three features together make
>> it *so easy and awesome* to create your own flow-control-like
>> structures. (Smalltalkers and Rubyists know this already).
>
> And now Scala'ists know too. I'm not the biggest fan of Scala's syntax, but
> it does have that.
Yup. The difference with Scala (from what I understand) is that it
will implicitly create closures based on the type that the method
expects, where with Smalltalk et. al. the block syntax is explicit at
the callsite. At first I considered that a drawback, but after playing
with Finch for a while I actually kind of like it. It's handy to see
at the callsite which things will be eagerly evaluated and which are
lazy (i.e. wrapped in a closure). As long as you have a
lightweight-enough syntax for blocks I find myself liking this.
>> The `while-match:do:` method is essentially a flow-control structure
>> that loops repeatedly while consuming matching operator tokens. That
>> was cake to implement and makes the code really easy to read (I
>> think).
>
> This reminded me a lot of my parser in Ruby, for parsing block argument
> names:
>
> begin repeat(IdentifierToken) do |argname|
> argnames << argname.float
> end end while option(CommaToken)
>
> So much tighter and readable than my previous C++ versions, so I agree
> blocks are awesome!
Agreed! What I really like is combining them with keyword selectors
because then you can define operations that take multiple blocks. Ruby
(and Magpie using 'with') handle the single block case OK, but
multiple blocks are even more awesome. For example, my little Finch
parser also has:
self if-look-ahead: Token/LeftParen then: {
self consume: Token/LeftParen
defines <-- self parse-defines
} else: {
defines <-- #[self parse-define]
}
That if-look-ahead:else: method here is something that would be hard in Ruby.
>> if: (blah) then: {
>> do something
>> } else: {
>> something else
>> }
>
> That's not too bad, although I kind of likes the def/end syntax.
I do too, which is part of the reason I originally went down the
user-defined syntax and keywords route. But lately I'm wondering if
the aesthetic preference is worth the complexity.
> You can emulate this in Io also because arguments can be lazy:
>
> if (blah) then (
> do something
> ) else (
> something else
> )
Right, Io is pretty cool here. But it also has known performance
problems because a compiler can't tell which arguments can be compiled
eagerly and which will be lazy. It's also harder to tell at the
callsite which arguments are eager and which aren't. The extreme
flexibility is something I really dig about Io but it kinda feels
*too* flexible to me.
> I agree. One problem with no keywords is that the syntax gets "fuzzy":
>
> while: {x < 10} do: {
> x = x + 1
> }
>
> In this case you need to use a block for the text, while if: takes an
> expression. if: could take a block to make it more orthogonal, I guess, but
> the point is it pushes the decision onto the programmer. In Ruby, an if and
> a while block look very similar - it handles the details that one is
> actually using a lazy expression.
Right, that part is a bit weird. My hunch is that you get used to it
pretty quickly but this could indeed be a problem. One corner where
Smalltalk really does look bad is short-circuiting logical operators:
Lots of languages:
if ((foo != null) && (foo.bar)) { ... }
Smalltalk/Finch:
if: ((foo != null) and: { foo.bar }) { ... }
One workaround I did for Finch was to just define a couple of extra
flow-control methods like if:and:then: to get rid of some of those
parentheses, but it's a still a bit hairy. For Magpie, I would likely
have special-case && and || operators that do short-circuit. It's not
as clean and simple that way, but logical operators are such a common
case that it may well be worth giving sugar for them.
>> 2. Replace the extensible syntax craziness with blocks and non-local
>> returns.
>
> I'm curious if you're thinking of Smalltalk/Objective-C style. I found it
> hard to implement this style with default arguments, although default
> arguments can be implemented by adding multiple helper methods.
Magpie in general doesn't have default/named arguments. It relies on
overloaded multimethods to accomplish the same thing. (For example,
substring(from:) and substring(from:to:) are just different methods.)
In general, I prefer that because I find it simpler to think about.
Optional args always feel a bit mushy to me.
The only case I know where this goes bad is when you have arguments
that are sort of optional-configuration stuff. If there's lots of
these args you definitely don't want the combinatorial explosion of
having every possible overload.
For that, in Magpie, I'd say either make those options all one big
option object, or it might be possible to define patterns that will
match a record with missing fields, but I haven't thought about that
much.
>> This is the path I'm on right now. I am heading down it, but I'm going
>> very slowly given the limited free time I have. That's OK: there's no
>> big rush on this.
>>
>> How does that sound?
>
> It sounds good!
\o/
- bob