Channels and Networks together

[Junaid Farooq]

Hi eveyone, Very new to #GoLang. have been doing Elixir / Ruby / PHP and etc stuff for past 4 years.

I was listening to the talk meet Go Lang. in which Rob Pike mentioned that, He and Ken talked a lot about getting Channels and networks together, but they still didnt, and he mentioned erlang has done that but it has different model structure.

The talk was very long time ago, what is the current status of this issue?

Go lang has done something in this? Channels and Networks together ? 

[Mandolyte]

Not to my knowledge. But if your goal is distributed computing, then take a look at https://github.com/chrislusf/glow

[Caleb Doxsey]

The challenge with the design of channels, is they need to be reliable to work properly, and networks are anything but reliable. Channels only really make sense within a single process.

That said, there have been libraries developed which help make working with distributed systems easier

The context package provides a uniform mechanism for defining deadlines/timeouts, cancellation and request-scoped variables. With a set of context-aware APIs it makes it really easy to implement the functionality necessary to deal with unreliable systems.

For example I can make an RPC call, which then calls a backend system, which then calls a database. With a context, I could cancel the RPC call, and it would automatically cancel the backend system call, which would, in turn, cancel the database call. This works for deadlines too.

Also worth looking into is gRPC, a generic RPC framework which has a lot of useful features. In particular, gRPC supports streams, which allows you to send well-defined messages in a channel-like fashion. There's also been a lot of thought put into features like load-balancing, retries, various error conditions, exponential backoff, etc... Things that are necessary to build reliable systems.

Don't take this as saying channels aren't useful. They still very much are, just within a single process. They are orthogonal to the communication that happens between processes.

Maybe an example might help:

I work on some systems that communicate via queues. They take in messages from one queue, do some processing, then write additional messages to another queue. A pipeline is a good model for a system like this. We could have 3 stages:

type message1 struct{}

type message2 struct{}

func startStage1(out chan<- message1, stop chan struct{})

func startStage2(in <-chan message1, out chan<- message2, stop chan struct{})

func startStage3(in <-chan message2, stop chan struct{})

The first stage would use a consumer library to consume from a Queue. For example from kafka:

func startStage1(out chan<- message1, stop chan struct{}) {

c, _ := kafka.NewConsumer(&kafka.ConfigMap{})

for {

select {

case <-stop:

return

case ev := <-c.Events():

switch e := ev.(type) {

case *kafka.Message:

out <- e

}

}

}

}

 

So we're generating messages. A process stage:

func startStage2(in <-chan message1, out chan<- message2, stop chan struct{}) {

for {

var msg1 message1

select {

case <-stop:

return

case msg1 = <-in:

}

// do your processing here

var msg2 message2

select {

case <-stop:

return

case out <- msg2:

}

}

}

And the producer would use the producer side of Kafka. 

This approach is easy to write and understand, but it's also very powerful:

• you can add multiple workers for a stage, for example 4 processors instead of 1, to fully utilize your machines cores, or leave it at 1 if you need the consistency
  
• you can use buffered channels between the stages to keep things moving smoothly
• you avoid having to use locks, because no state is shared
• the separation of concerns between each stage makes it easy to understand the code
• failures on the producer side apply back-pressure to the pipeline, so we don't consume more than we can handle (useful for a distributed queue like kafka where there are many consumers), and we don't run out of memory and cause a cascading failure, etc...

Hopefully that demonstrates one way to use channels with a distributed system. See https://blog.golang.org/pipelines for a deeper look.

Having worked with similar systems in Java, I can say the approach was nearly identical, with things like mailboxes and the like. It's just nice having the functionality built into the language as a first-class system, and having a robust scheduler backing it, so you don't have to be too concerned about creating too many goroutines, or fairness in selecting work.

On Wednesday, December 14, 2016 at 1:03:41 PM UTC-5, Junaid Farooq wrote: