Compare of Scala actors vs Go routines?

[philip]

Hi,

I want to know, which one is going to be best for parallel processing
on current and near future CPU's.
I read that the Go language go routines are more lightweight than a
thread, also Scala claims its threading is more lightweight.

http://java.dzone.com/articles/scala-threadless-concurrent
I read here "shared memory threads are quite heavyweight and incur
severe performance penalties from context switching overheads.". I
remember when I was programming a C++ server in the past, about 8
years ago, we had 1 thread per client, of course it started having
overhead of context switching so we had to drop that idea and go for a
different model.

Also it says in that link ". Hence Scala actors have been designed as
lightweight event objects, which get scheduled and executed on an
underlying worker thread pool that gets automatically resized when all
threads block on long running operations". This seems simular to Go
language?

Regardless, can someone compare and contrast the differences and
benifits of each?
Assuming I am trying to write some software which works well on 8 core
CPU's in the future say 2 or N years from now, should I be using
Scala, should I be using Go? I know this is only one consideration and
that many other factors weigh in the decision, but for discussion I
feel its best to limit to the threading topic.

Thanks! Phil

[Michael]

It would be nice to add Erlang to this shootout. Here's a comparison
between Erlang and Scala: http://ruben.savanne.be/articles/concurrency-in-erlang-scala

You can get a pretty good idea of what Erlang and Scala look like when
doing (very simple) parallel stuff. Anybody want to take that code
and write the Go version? Maybe we can do some quick measurements.

Michael

[Russ Cox]

I can think of a few key differences compared to Go.

Scala runs on top of a JVM.

Erlang doesn't have writable variables.

Those are simple and maybe obvious but they color a lot of
the rest of the language and the way it gets used.

Russ

[Michael Bilca]

Absolutely right.  However, it would be nice (yes, and perhaps unfair) to to a first-order comparison of scalability to 10K - 100K - 1M "green" threads among these languages (and any other languages folks may think of).

I have a project with relative freedom from other requirements.  I need a system that's fast, and scales well to those levels.

I really don't want to guess, but I suspect Go may have the edge, since it seems the "closest-to-the-metal."  

I agree that considerations such as runtime support (VM or not), libraries, frameworks, and many others are necessary for a *real* shootout. A good engineer would look at the whole picture before committing a project to any language. "Surface" language features, such as non-writable variables, the lack of control structures aside from recursion, etc., are important, but I'd be willing to put up with such relatively minor pains to gain super-scalability (although I am partial to recursion :)

If we are clear about the constraints, perhaps we can come up with a test that's meaningful and valid. The hard part is coming up with a test that somewhat isolates and only measures the feature we're interested in (scalability).

My idea would be to spawn a set of threads, and bounce a simple message around them (perhaps in round-robin fashion, or in an explosive one-to-many fashion), and see how they do time-wise and memory-wise. Any other ideas?

Michael

[Scott Solmonson]

Hi Michael-

I'm in your court regarding super-scalability, and that is essentially my area of expertise.
My current thought is that the most scalable model existing now is that of the autonomous actor implementing periodically-updated actions received via an authenticated centralized command and control point.

As a complete Java-as-a-language/syntax-hater, I still have to admit that the VM has very much matured performance-wise since its early days, and I see Scala as a promising way to make it newly enjoyable for me to code in as well as pushing towards the nirvana that is unified interfaces and instruction sets.

At the same time, I reinforce your thoughts that the "surface level" features as you describe are mostly meaningless; instead I focus on how a language interfaces with the "weird" constraints of the mainstream hardware and operating systems that exist these days-

When a language makes it easy to scale and propagate both blocking and non-blocking operations across multiple cores->multiple CPUs->distributed NUMA it makes the grand plan shine like Baby Jesus.; and I like others see GoRoutines as crude but effective, which is definitely the right solution at times.

MUXing blocking and non-blocking OPs in to an OS-level thread pool that can hit every processor as needed is ugly, but a step in the right direction as far as I'm concerned :)

--
NUNQUAM NON PARATUS
V: 408.718.6290

[philip]

> Erlang doesn't have writable variables.

I really don't understand these languages without writable variables,
I mean I do program in them, I program in Scala, but I dont like these
- in the past, I used to just grab a big block of ram and do whatever
I wanted with it. malloc huge block, free it later, then I didn't have
to deal with memory management and I write where I want.

[Michael]

Well, let's clean up our language, and call them write-once variables,
only because someone unfamiliar with this may be reading this thread
and think we've lost our mind. And this makes perfect sense in the
Prolog/Erlang world, which is a very close cousin of algebra. I
actually find this makes my programs a lot more readable, since I'm
forced to name my variables well. For example:

recursiveFunction(Thing) ->
CandidateNewThing = Thing + 1,

if
CandidateNewThing > 20 -> NewThing = 20;
CandidateNewThing <= 20 -> NewThing = CandidateNewThing;
end,

%% do some interesting stuff with NewThing, and tail-recurse:

recursiveFunction(NewThing).

See? Did you miss being able to do Thing++? Not me. Every variable
tells you why it's there. Some call it cumbersome, some love it. Very
few people in-between.

Before we go too far away from the subject, let's get back to writing
a test. I'll throw something up in Erlang today. I'll need help from
you guys to translate it to Scala and Go.

Michael

[Michael]

A small aside - if someone figures out a way to wrap a GPGPU under one
of these "actor" based languages, so I don't have to muddle through
CUDA or OpenCL any more, I'll name my first baby after them, even if
their name is S&!t-for-brains. Yes, I realize that the GPU model of
parallelism is really SIMD (no matter how much NVidia stomps their
feet and says it's MIMD), and that makes it hard to match to the
asynchronous message-passing model under Erlang/Scala/Go(? - I'm just
learning about Go, so keep me honest here).

Maybe Go has a fair shot at this. Hey, I'm allowed to ask for anything
in dreamland :)

[David Flemström]

On Fri, Jan 22, 2010 at 2:38 PM, Michael <mbi...@gmail.com> wrote:

Before we go too far away from the subject, let's get back to writing
a test.  I'll throw something up in Erlang today.  I'll need help from
you guys to translate it to Scala and Go.

Michael

 

I could do the translation to Scala, but since Scala is one of those languages that allows you to do something in many different ways, there should be other translation candidates as well.

FWIW, a list of known weaknesses that the Scala model has, worth considering when creating the benchmark:

• Passing around primitive data like integers is expensive, because of the JVM legacy.
• Since real OS threads are used for each actor (when using the thread-based actor library and not the hybrid-event-based one - as I said, Scala has many actor implementations), there's an overhead to consider when spawning 1000+ actors.

[Michael]

Understood. To match the Erlang model, we should stick to event-based
actors and avoid thread context-switching. I think this also
corresponds well to goroutines/channels.

On the message payload issue... I'm a little stumped. It would be
meaningless to pass payload-less messages. To be useful, the messages
should carry at least the ID of the sending actor. In Erlang, I can
make that an atom by using the built-in method to register processes.
"Registering" a process in erlang binds the atomic name with the
ProcessID. Is there something equivalent in Scala and in Go?

Michael

On Jan 22, 9:31 am, David Flemström <david.flemst...@gmail.com> wrote:
> On Fri, Jan 22, 2010 at 2:38 PM, Michael <mbi...@gmail.com> wrote:
>
> > Before we go too far away from the subject, let's get back to writing
> > a test.  I'll throw something up in Erlang today.  I'll need help from
> > you guys to translate it to Scala and Go.
>
> > Michael
>
> I could do the translation to Scala, but since Scala is one of those
> languages that allows you to do something in many different ways, there
> should be other translation candidates as well.
>
> FWIW, a list of known weaknesses that the Scala model has, worth considering
> when creating the benchmark:
>

>    - Passing around primitive data like integers is expensive, because of
>    the JVM legacy.
>    - Since real OS threads are used for each actor (when using the

[Michael]

OK, here's my first shot. There are four files in the Erlang package
(send me a note for a .zip of all four). The first is the "Makefile":

------------- snip -------------
# Turing Makefile
.SUFFIXES: .erl .beam
.erl.beam:
erlc -W $<
ERL = erl -boot start_clean -sname concurrencytest -setcookie cookie1
MODS = test shepherd sheep
all: compile
${ERL} -pa 'YOUR/PATH/HERE' -s test run
compile: ${MODS:%=%.beam}
clean:
rm -rf *.beam erl_crash.dump
------------- /snip -------------

Careful when you paste the Makefile. The indents MUST be one tab, not
a bunch of spaces. Make is finicky. Also, be sure to tell it where
the code is by modifying the path string (YOUR/PATH/HERE).

Next is "test.erl" It doesn't do much except set some initial
conditions. Think of this as your project file.

------------- snip -------------
%% Author: mbilca
%% Created: Jan 21, 2010
%% Description: test.erl :This is an Erlang concurrency test that uses
one shepherd
%% to spawn many sheep that send messages to each other.
%% The shepherd measures the time it takes for a given number of
messages
%% to randomly make their way through the population.

-module(test).
-export( [run/0] ).

run() ->
io:format("~nStarting Test ...~n"),
NumSheep = 10,
MaxMessages = 100,
io:format("Initializing Shepherd with ~p sheep
that will pass ~p messages...~n", [NumSheep,
MaxMessages]),
register(shepherd, spawn(shepherd, init, [NumSheep,
MaxMessages])).
------------- /snip -------------

Next is the "shepherd.erl":

------------- snip -------------
%% Author: mbilca
%% Created: Jan 21, 2010
%% Description: shepherd.erl: The shepherd spawns the sheep, and sends
the first message,
%% starting the sheep talking to each other
-module(shepherd).

-import( io, [format/1, format/2] ).
-import( lists, [seq/2] ).

-export([init/2]).

init(NumSheep, MaxMessages) ->
SheepAliases = [list_to_atom(integer_to_list(X)) || X <- seq
(1,NumSheep)],
[register(Alias, spawn(sheep, init, [Alias, SheepAliases,
NumSheep, MaxMessages])) || Alias <- SheepAliases],
loop(0, NumSheep, MaxMessages).

loop(START_Timestamp, NumSheep, MaxMessages) ->
receive
{start} ->
format("Starting shepherd...~n"),
{MegaSecs,Secs,MicroSecs} = now(),
START_Timestamp_new = (MegaSecs * 1000000 + Secs) *
1000000 + MicroSecs,
'1' ! {'fire', '2', 0}, %% kick off the party
loop(START_Timestamp_new, NumSheep, MaxMessages);
{stop} ->
{MegaSecs,Secs,MicroSecs} = now(),
STOP_Timestamp = (MegaSecs * 1000000 + Secs) * 1000000 +
MicroSecs,
ElapsedTime = STOP_Timestamp - START_Timestamp,
ElapsedSecs = ElapsedTime / 1000000,
format("Elapsed time for ~p sheep with ~p messages: ~p
seconds~n", [NumSheep, MaxMessages, ElapsedSecs]);
{save} ->
void
end.
------------- /snip ------------

Last is the "sheep.erl"
------------- snip -------------
%% Author: mbilca
%% Created: Jan 21, 2010
%% Description: sheep.erl: The only thing the sheep know to do is to
receive a message from other sheep,
%% increment the NumMessagesSoFar by one, and send a new message along
to a new randomly chosen sheep.
-module(sheep).
-import(io, [format/2]).
-export([init/4]).

init(MyAlias, SheepAliases, NumSheep, MaxMessages) ->
format("Starting sheep #~p~n", [MyAlias]),
loop(MyAlias, SheepAliases, NumSheep, MaxMessages).

loop(MyAlias, SheepAliases, NumSheep, MaxMessages) ->
receive
{'fire', FromAlias, NumMessagesSoFar} ->
format("RECEIVE: ~p <<< ~p; Message # ~p~n", [MyAlias,
FromAlias, NumMessagesSoFar]),
RandomTarget = list_to_atom(integer_to_list(random:uniform
(NumSheep))),
NewNumMessagesSoFar = NumMessagesSoFar + 1,
if
NewNumMessagesSoFar =< MaxMessages ->
begin
format(" SEND: ~p >>> ~p~n", [MyAlias,
RandomTarget]),
RandomTarget ! {'fire', MyAlias,
NewNumMessagesSoFar}
end;
true ->
begin
format(" Trying to stop ~p~n", [MyAlias]),
stop(MyAlias, SheepAliases)
end
end,
loop(MyAlias, SheepAliases, NumSheep, MaxMessages);
{'die'} ->
format(" ... sheep ~p dying!~n", [MyAlias])
end.

stop(MyAlias, SheepAliases) ->
format(" ... sheep ~p stopping.~n", [MyAlias]),
%% the first sheep that reaches MaxMessages commits mass murder:
[X ! {'die'} || X <- SheepAliases -- [MyAlias]],
shepherd ! {stop}.
------------- /snip ------------

We'll call this the babbling sheep test. They don't do anything
interesting except pass one message around randomly until MaxMessages
is hit. We'll be able to vary NumSheep and MaxMessages to make the
test runnable.

This is very raw, and not ready to actually do scaling tests. It is
offered so we can agree on the strategy first.

In the real test, we would have to turn off all the printouts, at
least. They're in there so you get an idea of what is happening under
the hood. This is why it starts with 10 sheep and only goes to 100
messages. Also, the Erlang runtime default to max 32K processes. This
can be overridden, though.

Let's first talk about whether this approach is a valid test of
scalability, since there are some things in here that go beyond pure
scalability (using random numbers for example). To David's point, the
messages have to carry some things to be useful. I think the above is
a realistic enough piece of code -- I don't know if I could simplify
it much and still retain it's usefulness.

If you think of better (simpler or not) ways to do it, great, but
let's make sure the Scala and Go versions do the same thing, and are
similar in approach.

So, David volunteered to do the Scala bit, any Go takers?

Regards,
Michael

[Michael]

Forgot to mention one thing:

After you make the project (just type make in the src directory), it
will automatically start an Erlang runtime instance. You'll see the
sheep being built, but the messaging won't start until you tell the
shepherd to start, like so:

shepherd ! {start}.

That ought to do it.

Michael

[Taru Karttunen]

Excerpts from Michael's message of Fri Jan 22 21:16:44 +0200 2010:

> OK, here's my first shot. There are four files in the Erlang package
> (send me a note for a .zip of all four). The first is the "Makefile":
>
> ------------- snip -------------
> # Turing Makefile
> .SUFFIXES: .erl .beam
> .erl.beam:
> erlc -W $<
> ERL = erl -boot start_clean -sname concurrencytest -setcookie cookie1
> MODS = test shepherd sheep
> all: compile
> ${ERL} -pa 'YOUR/PATH/HERE' -s test run
> compile: ${MODS:%=%.beam}
> clean:
> rm -rf *.beam erl_crash.dump
> ------------- /snip -------------

When benchmarking Erlang please note that you will have to try out a
few switches to get best performance.

For compiling: bytecode vs native
For running: whether to tell erlang use multiple cores, smp, etc

The flags needed for best performance will vary with the hardware used.

- Taru Karttunen

[Michael]

Thanks, Taru.

We'll have to do our best to match these OS-level concerns across the
languages, and also across multiple machines. Who knows, we may
discover some interesting variances between two-cores and, say, eight
hyper-thread-cores. We'll complete the matrix as we go, once a few of
us compile the three contenders on whatever machines we have
available. I'll run my tests on a Core2Duo OSX 10.6. I hope someone
with a Corei7 four core will volunteer. And an AMD-64 person :)

Michael

[atomly]

One very clear advantage of Erlang out of the box is that it's
basically completely painless to scale beyond one machine as well.
It's really no harder to send a message to a process on a different
machine-- the runtime does all the heavy lifting.

--
:: atomly ::

[ ato...@atomly.com : www.atomly.com : http://blog.atomly.com/ ...
[ atomiq records : new york city : +1.347.692.8661 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...

[Michael Bilca]

Hello all.  Sorry for the absence - too many projects, too little time.

For completeness, below is the equivalent Go version of the Erlang code above.  Suffice it to say that I have my answer, but I don't want to publish numbers because it would not really be fair (given all the valid reasons enumerated on this thread). I'd still love to see a detailed shoot-out. Still, you can run the code yourself in your environment, and see for yourself. If David F. is still around, I'd love to see the Scala flavor. Any criticism is welcome.

I did my best to make sure that the two versions follow the same pattern.

There are three files: test.go, shepherd.go, and sheep.go (equivalent to test.erl, shepherd.erl, and sheep.erl above). Again, as before, just copy the code between the ---snip--- and ---/snip--- lines into their own files with the given names, and compile.  Uncomment the print statements in sheep.go to look under the hood.

I put test.go in its own directory in my go distribution:  ~/go/src/pkg/scalingTest/test.go  Note that its makefile includes Make.cmd.

I put shepherd.go and sheep.go in ~/go/src/pkg/sheep/*.go    Notice that this Makefile includes Make.pkg 

Hope you find it useful.

Regards,
Michael

test.go Makefile first:

------------- snip ------------- 

# File:  ($GOROOT)/src/pkg/scalingTest/Makefile 

# Go Turing Makefile  Author: Michael Bilca 2010

include $(GOROOT)/src/Make.$(GOARCH)

TARG=scalingTest

GOFILES=\

test.go

include $(GOROOT)/src/Make.cmd

------------- /snip ------------- 

The test.go file isn't very interesting - it just sets up the initial conditions. It's mostly here so it parallels the Erlang version.

------------- snip --------------

/* File:  ($GOROOT)/src/pkg/scalingTest/test.go  

 * Author: mbilca

 * Created: Feb 12, 2010

 * Description: This is an Go concurrency test that uses one shepherd to spawn

 * many sheep that send messages to each other.

 * 

 * The time is measured with the UNIX time utility:

 * time ./scalingtest

 * This measures the time it takes for a given number of messages to make their

 * way throught the population. Incidentally, each sheep sends the message to the

 * next one instead of a random one, because the random number generator kept 

 * blowing up after a while - no time to dig into it...

 */

package main

import "fmt" 

import "sheep/sheep"

func main() {

fmt.Printf("\n-------------------------------------------------------------")

fmt.Printf("\nStarting test...\n")

numSheep    := int(100000)

maxMessages := int(1000000)

fmt.Printf("Initializing Shepherd with %v sheep that will pass %v messages...\n", numSheep, maxMessages) 

sheep.InitShepherd(numSheep, maxMessages)

}

------------- /snip ------------- 

Here's the Makefile in the sheep package directory:

------------- snip ------------- 

# File:  ($GOROOT)/src/pkg/sheep/Makefile 

# Go Turing Makefile  Author: Michael Bilca 2010

include $(GOROOT)/src/Make.$(GOARCH)

TARG=sheep/sheep

GOFILES=\

sheep.go\

shepherd.go\

include $(GOROOT)/src/Make.pkg

------------- /snip ------------- 

Now the shepherd:

------------- snip --------------

/* File:  ($GOROOT)/src/pkg/sheep/shepherd.go 

 * Author: mbilca

 * Created 2/12/2010

 * Description: The shepherd spawns the requested number of sheep and sends the first

 * message, starting the sheep talking to each other

 */

package sheep

import "fmt"

type MessageT struct {

phrase string

fromID int

numMessagesSoFar int

}

func InitShepherd(numSheep, maxMessages int) {

fmt.Printf("\nStarting shepherd...\n")

fmt.Printf("\nID:MESSAGE\t\t\tfromID\t>>\tthisID\t>>\ttargetID\tnumSoFar\n")

fmt.Printf("-------------------------------------------------------------------------------------------------\n")

channelArray := make([]chan MessageT, numSheep, numSheep)

finalAnswerChannel := make(chan int)

// create the channels

for id := int(0); id < numSheep; id++ {

                channelArray[id] = make(chan MessageT) // no buffer

        }

// spawn the sheep

for id := int(0); id < numSheep; id++ {

go Sheep(id, numSheep, channelArray, finalAnswerChannel, maxMessages)

}

message := new(MessageT)

message.phrase = "baa"

message.fromID = 99 

message.numMessagesSoFar = 0

channelArray[0] <- *message // start the babbling

x := <-finalAnswerChannel // wait for final answer

fmt.Printf("Done! %v messages sent and received\n", x)

}

------------- /snip -------------

And the sheep:

------------- snip --------------

/* File:  ($GOROOT)/src/pkg/sheep/sheep.go

 * Author: mbilca

 * Created 2/12/2010

 * Description: The only thing the sheep do is receive a message from other sheep,

 * increment the numMessagesSoFar field in the message by one, check if maxMessages was

 * reached, if so send a stop message back to the shepherd, otherwise send the message 

 * along to another sheep from the population.

 */

package sheep

import "fmt"

// the sheep use the channel indexed by their own id number as their input channel

func Sheep(myID, numSheep int, chArray []chan MessageT, finalAnswerCh chan int, maxMessages int) {

//fmt.Printf("Initializing sheep ID=%v...\n", myID)

for {

message := <-chArray[myID]

//fmt.Printf("%v received message %v\n", myID, message)

message.numMessagesSoFar = message.numMessagesSoFar + 1

if message.numMessagesSoFar == maxMessages { // send stop message to shepherd

finalAnswerCh <- message.numMessagesSoFar

}

//target := int(rand.Float() * float(numSheep)) // couldn't get this to work - kept blowing up before 1M messages

target := ( myID + 1 ) % numSheep  // instead, each sheep targets the next one (with wrapping @ numSheep)

/*

// only print out every mod N message

if message.numMessagesSoFar % 100000 == 0 {

fmt.Printf("%v:%v\t%v\t>>\t%v\t>>\t%v\t\tnumSoFar = %v\n", myID, message, message.fromID, myID, target, message.numMessagesSoFar)

}*/

message.fromID = myID

message.phrase = "baa"

chArray[target] <- message

}

}

------------- /snip ------------- 

[Michael Bilca]

Sorry, the indentation went to heck because the tabs were stripped by the forum.  Here it is, with spaces instead of tabs.  Don't forget, Make *insists* on tabs, so put them back in the Makefiles (before these three lines: test.go, sheep.go, shepherd.go.