thread.join equivalent in Go

[i3dmaster]

Quote from spec:

"When main.main() returns, the program exits. It does not wait for
other (non-main) goroutines to complete."

func compute() {
result := make(chan int);
go func() {
result<-do_number_crunching_compute();
}();
report(result);
close(result);
}

func main() {
go compute();
do_something_else();
// Now, how I can make sure the program exists only after the
compute is done?
}

What's the Go way to do something like join in thread programming?

PS: a naive thought is to make another chan just to signal the main
thread, and add the last statement in main() like "<-wait;" But I am
not sure if that's the idiomatic way in Go...

[Kevin Gisi]

I asked people about this yesterday, and you're absolutely right. A termination channel is the way to go.

-Kevin W. Gisi
ke...@kevingisi.com | (916) 538-4474
http://www.kevingisi.com

[Yongjian Xu]

wow.. ok. Wonder how scalable this solution would be. Imagine if we write some big parallel computing programs utilizing a big number of goroutines, then I'd have to create term chan for each and place them all at the end of the main... That's quite some boilerplates I have to keep up with... 

[Kevin Gisi]

In that type of situation, you could hypothetically have one term channel, and simply wait until 10,000 items come through the tube.

A bit of an oversimplification, but the point is that often you can share the channel resources between lots of goroutines.

-Kevin W. Gisi
ke...@kevingisi.com | (916) 538-4474
http://www.kevingisi.com

[Russ Cox]

On Sun, Nov 29, 2009 at 16:47, Yongjian Xu <i3dm...@gmail.com> wrote:
> wow.. ok. Wonder how scalable this solution would be. Imagine if we write
> some big parallel computing programs utilizing a big number of goroutines,
> then I'd have to create term chan for each and place them all at the end of
> the main... That's quite some boilerplates I have to keep up with...

Kevin's solution is better (one channel), but even if you had one
channel per goroutine, it wouldn't need a lot of boilerplate, just
a for loop.

Russ

[Esko Luontola]

On Nov 30, 2:51 am, Kevin Gisi <ke...@kevingisi.com> wrote:
> In that type of situation, you could hypothetically have one term channel,
> and simply wait until 10,000 items come through the tube.

I just did that recently and it works nicely. I keep a count of how
many goroutines are running - increasing the count when a goroutine is
started and decrementing it when one of them finishes:

http://github.com/orfjackal/gospec/blob/22caf9a3114408b1a63c47f311ac468359059111/src/gospec/runner.go#L63-77

[atomly]

If somebody had some spare time, it might be nice to build some
concurrency tools in Go like Latches, Semaphores, Barriers, etc...
Should be relatively straightforward...

--
:: atomly ::

[ ato...@atomly.com : www.atomly.com : http://blog.atomly.com/ ...
[ atomiq records : new york city : +1.917.442.9450 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...

[Russ Cox]

On Mon, Nov 30, 2009 at 17:02, atomly <ato...@atomly.com> wrote:
> If somebody had some spare time, it might be nice to build some
> concurrency tools in Go like Latches, Semaphores, Barriers, etc...
> Should be relatively straightforward...

Go encourages a different way of thinking about concurrent
programming. There are going to be a few low-level things
like mutexes in package sync for use by libraries that can't
help it, but at the application level, we hope people will use
channels and communication to synchronize their programs
instead of these kinds of low-level shared memory primitives.
Our experience with other languages has been that explicit
communication is far easier to reason about and get right
than protocols that operate on shared memory.

This is so important that we have a mantra:

Do not communicate by sharing memory
Instead, share memory by communicating.

Russ

[atomly]

On Tue, Dec 1, 2009 at 3:49 AM, Russ Cox <r...@golang.org> wrote:
> Go encourages a different way of thinking about concurrent
> programming.  There are going to be a few low-level things
> like mutexes in package sync for use by libraries that can't
> help it, but at the application level, we hope people will use
> channels and communication to synchronize their programs
> instead of these kinds of low-level shared memory primitives.
> Our experience with other languages has been that explicit
> communication is far easier to reason about and get right
> than protocols that operate on shared memory.
>
> This is so important that we have a mantra:
>
> Do not communicate by sharing memory
> Instead, share memory by communicating.

I definitely agree with this.. It's hard to go back to my day job
doing highly concurrent code in Java after working in Go and Erlang on
my personal projects. I still think it'd be worthwhile, though, to
have some abstractions for certain types of behavior. A barrier, for
example, is a useful construct, and basically what the OP wanted--
building one in the "Go idiom" should be trivial. Perhaps it's not so
much a package that I'm talking about building, but "patterns," as
they like to say in the Java world.

[Bob Cunningham]

This is highly reminiscent of some work I did over 20 years ago on "data flow" computer architectures, which today are the foundation for DSP, GPGPU, FPGA and other digital signal processing. The general notion actually pre-dates digital processing, as it was the fundamental architecture for the analog computers used for sonar and radar processing. Much radar processing is still performed in the analog domain, because digital systems still aren't fast enough.

In high-throughput environments, it can be very effective to have the data "flow" as streams through the code, the flow of data branching and merging as needed, rather than having the code "get" and "put" the data. At an extreme, it may be better to bring the code to the data, rather than the other way around, which is the concept behind embedded database actions/functions, where it can be far more effective to compute some data values within the database itself rather than perform a fetch-compute-store cycle. It is also similar to parallel pipelined CPU architectures, where a given processor element (ALU, FPU, etc.) "fires" only when data is presented to it, and is idle otherwise (saving power).

For me, the various features and restrictions of Go and goroutines make much more sense when viewed from a fine-grained cascaded/parallel signal processing perspective, rather than a traditional coarse-grained task/thread-oriented parallel processing perspective.

Current languages and tools often force programs to be designed to make optimal use of large-scale system capabilities/resources, with little inherent flexibility when moved to systems with different capabilities. Go may encourage programming toward the smallest capabilities, letting the Go runtime manage how goroutines are aggregated and mapped to the system resources actually present. That's a difficult problem that should not need to be uniquely solved by each programmer for each program for each system. How many goroutines per thread? How many threads per core? Make "enough" goroutines, and let the Go runtime handle the rest.

Only recently has the Linux kernel begun to manage "task affinity" to run related programs on the same core, and unrelated programs (or program instances) on different cores. Go will need to do similar work for goroutines, and possibly for threads, in close cooperation with the OS.

Ultimately, the only thing a Go programmer may need to care about is the relative cost of a local function call versus the cost of using a channel and a goroutine in the same thread. And that will only matter when cores are few and/or when algorithms are inherently serial (Amdahl's Law). In that case, the goal will be to minimize overhead when forced to run on a single core, and maximize performance when many cores/sockets are available.

But why should a Go programmer even need to care about that? I would strongly favor a Go runtime optimization that, when connected goroutines are scheduled into the same thread, will automatically collapse simple goroutine invocations into function calls (convert parallel to serial). This is a *much* easier problem than the reverse (automatic parallelization of function calls)! It would also enable a given Go program to run as efficiently as possible on both a single-core phone and on a 100 core workstation.

When starting a Go program, it should be made possible to pass "hints" to the Go runtime to support this optimization, based on previous patterns of execution (information I'd put into the resource fork of the program file, or into the ELF file structure).

If Go had such capabilities, I'd probably call far few functions, and make much greater use of channels and goroutines.


-BobC

[Magnus Hemmendorff]

How about adding more flexibility of launching threads? Sometimes it's
nice to use thread servers. Sometimes it's nice to set thread
priority. Sometimes thread.join is better than channels.

Flexibility is more important than saving 2 lines of code for calling
a go routine.

I suggest Go introduces thread objects and objects describing
invocation. The latter would be a combination of delegate and functor.
http://en.wikipedia.org/wiki/Function_object
I guess a functor/delegate can be implemented as a struct comprising:
1. A function pointer, 2. A sequence of bytes to use as a call
stack, and 3. A list of objects referenced by the call stack, in
order to inhibit the Garbage collector from deleting them

I believe such functor/delegate objects would simplify the use of
thread servers, and other methods for thread invocation. Most of all,
it would facilitate GUI programming.

[Ian Lance Taylor]

I'm not opposed to flexibility, but what you are describing sounds
rather complicated to use. I think you would need some reasonable
compelling advantages.

Ian

[Dimiter "malkia" Stanev]

For GUI programming wouldn't the channel system be enough? I dunno...

I'm trying to thnink in Go as if there is no way to have threads in
Go, for example the "goroutines" might be even running out of the
process (although they don't), or even on a different machine.

On Dec 1, 3:09 am, Magnus Hemmendorff <magnu...@gmail.com> wrote:
> How about adding more flexibility of launching threads? Sometimes it's
> nice to use thread servers. Sometimes it's  nice to set thread
> priority. Sometimes thread.join is better than channels.
>
> Flexibility is more important than saving 2 lines of code for calling
> a go routine.
>
> I suggest Go introduces thread objects and objects describing

> invocation. The latter would be a combination of delegate and functor.http://en.wikipedia.org/wiki/Function_object

[Magnus Hemmendorff]

Flexibility, delegates and functors do not have to be complicated.
Just take a look at delegates in python GUI programming.

Go would still be simple using the following syntax for functors,
delegates and go routines:
Introduce "makeFunctor" as a new operator in the language.
Assume we want to call myFunction(1, 2) in a new thread.

Simple use:
go( makeFunctor( myFunction, 1, 2 ) )

Flexibility is available when we need it:
myFunctor := makeFunctor( myFunction, 1, 2 );
myThreadObject := go(myFunctor);
myThreadObject.setPriority(2);
myThreadObject.join();

or:
myThreadServer.enqueue( myFunctor );

To invoke the functor in the same thread:
myFunctor()

Delegates:
The functor can also be delegate for invoking myObject.myMethod(1,2)

A delegate taking two arguments may be defined using the following
syntax:
myFunctor := makeFunctor( myObject.myMethod );
go( myFunctor(1,2) );

or for GUIs:
myButton.onClick = myFunctor

An advanced possibility is a mixture of a delegate taking one argument
when calling a function with two arguments:
myFunctor := makeFunctor( myObject.myMethod, 1, ... );
go(myFunctor(2));

[Devon H. O'Dell]

2009/11/29 Yongjian Xu <i3dm...@gmail.com>:

> wow.. ok. Wonder how scalable this solution would be. Imagine if we write
> some big parallel computing programs utilizing a big number of goroutines,
> then I'd have to create term chan for each and place them all at the end of
> the main... That's quite some boilerplates I have to keep up with...

It's really not that bad. See the `goroutines stalling' thread for a
solution to this issue I posted earlier.

--dho

[Ian Lance Taylor]

Magnus Hemmendorff <magn...@gmail.com> writes:

> Go would still be simple using the following syntax for functors,
> delegates and go routines:
> Introduce "makeFunctor" as a new operator in the language.
> Assume we want to call myFunction(1, 2) in a new thread.
>
> Simple use:
> go( makeFunctor( myFunction, 1, 2 ) )

How is this different from
go func() { MyFunction(1, 2) }()
?

> Flexibility is available when we need it:
> myFunctor := makeFunctor( myFunction, 1, 2 );
> myThreadObject := go(myFunctor);
> myThreadObject.setPriority(2);
> myThreadObject.join();
>
> or:
> myThreadServer.enqueue( myFunctor );

The idea of myThreadObject seems entirely independent of the idea of
makeFunctor. I agree that some sort of goroutine control is likely to
be useful at some point, but it does not seem very high priority at
present. And I'm not sure that join will ever make sense for Go.

Ian

[K Wang]

simple way is make another channel

func compute(ch chan int){

      // do your stuff

      ch <-1

} 

func main() { 

   ch:= make( chan int)
  go compute(ch); 
  do_something_else(); 
   <- ch //HERE will exit only ch is done
} 

[Stannis Kozlov]

I've been looking for a solution for .join like method in Go. Finally solution came with "sync":

https://golang.org/pkg/sync/#WaitGroup