[ANN] Glow - A distributed compute system in pure Go, similar to Spark, Flint, etc

[ChrisLu]

I am trying to get more interests, ideas, contributions on this fresh project.

Please take a look at this hello_world.go and think about a minute:

https://play.golang.org/p/1xPT-LIQsV

This piece of code can run locally or distributedly. 

In local mode, it will in parallel goroutines, without idiomatic but boring goroutine and wait groups. 

To run in distributed mode, setup the cluster, and just use "hello_world -glow".

This is the execution plan: https://github.com/chrislusf/glow_examples/blob/master/word_count/testBasicMapReduce.svg

Check here first to have more feeling of the code and execution plans.

https://github.com/chrislusf/glow_examples/tree/master/word_count

And check here about setting up the glow cluster.

https://github.com/chrislusf/glow

Warning! This is freshly baked and bugs will happen!

[ChrisLu]

It seems Java/Scala/C++ is taking all the funs of distributed computing, with Hadoop, Spark, Storm, Meso, YARN, Kafka, Zookeeper, etc.

Glow is trying to do it via Go, and make distributed computing simple for Go.

Here is how to setup the glow cluster:

  // fetch and install via go, or just download it from somewhere
  go get github.com/chrislusf/glow
  // start a master on one computer
  glow master
  // run one or more agents on computers
  glow agent --dir . --max.executors=16 --memory=2048 --master="localhost:8930" --port 8931

And to run the code in distributed mode, just run it via

  hello_world -glow

This should be simple enough to make your Go code run across the cluster.

Chris

[Pablo Rozas-Larraondo]

Thanks Chris, it looks really neat and clean. I was also asking myself why the whole big data/map reduce open source ecosystem has been built around the JVM. For me Go is like the perfect language to build these kind of platforms.

I think this is very interesting, great work! I'll give it a try and will let you know.

Cheers,

Pablo