go-routine Id..

[Sunil S Nandihalli]

Hello everybody, 

 I would like to have an id on a per goroutine basis.. When I googled I found this  old post  ...http://groups.google.com/group/golang-nuts/browse_thread/thread/d071b208eae1bae2/5a219b8901389e15

I would like to know if things have changed since. I need it for the purpose of logging from various goroutines. 

Thanks again,

Sunil.

[John Asmuth]

Nothing has changed - you can still do this on your own by creating an ID when different goroutines are spawned, possibly as a parameter to the function.

[Sunil S Nandihalli]

Thanks John

[Torben Weis]

2011/4/18 John Asmuth <jas...@gmail.com>

Nothing has changed - you can still do this on your own by creating an ID when different goroutines are spawned, possibly as a parameter to the function.

Is there a reason for not implementing it or has nobody volunteered yet?

Thread local data used to be very useful from time to time.

For example, in a web server I create a Go routine for each user request.

This go routine calls several functions. To each of these functions I have to pass the session ID which is quite nasty on the long run, especially when calling through libraries which are not under my control. Attaching the session ID to the go routine would simplify the issue a lot.

The go-routine-local-data should be a map[string]interface{}, where the string is the name of a library.

This way there are no collisions if two independent libs try to attach data to the same go routine.

Initially the map is zero, thus, there is no significant memory impact when the feature is not used.

Torben

[Russ Cox]

> Is there a reason for not implementing it or has nobody volunteered yet?
> Thread local data used to be very useful from time to time.

Yes, it is great for breaking C programs when they use
multiple threads in a way that the library they call did not
anticipate. Just say no.

> For example, in a web server I create a Go routine for each user request.
> This go routine calls several functions. To each of these functions I have
> to pass the session ID which is quite nasty on the long run, especially when
> calling through libraries which are not under my control. Attaching the
> session ID to the go routine would simplify the issue a lot.

Until you want to create a second goroutine to help with the request.

Russ

[Russ Cox]

There are already goroutine-local variables: they are called
function arguments, function return values, and local variables.

Russ

[Dmitriy Vyukov]

On Thu, Apr 21, 2011 at 5:37 PM, Russ Cox <r...@google.com> wrote:
> There are already goroutine-local variables: they are called
> function arguments, function return values, and local variables.

Why then goroutine creation, memory allocation, garbage collection,
etc do not use explicit contexts passed via arguments? ;)

[Torben Weis]

2011/4/21 Russ Cox <r...@google.com>

> Is there a reason for not implementing it or has nobody volunteered yet?
> Thread local data used to be very useful from time to time.

Yes, it is great for breaking C programs when they use
multiple threads in a way that the library they call did not
anticipate.  Just say no.

There are always solutions to shoot yourself in the foot. That's no argument.

I agree that go-routine-local-data might be a bad thing to do for a library programmer, because of the problems you mentioned.

A lib has no full control of the program flow. Agreed.

However, for an application programmer this is a totally different story.

If the app programmer does not know that some wired thread is running through his code then most likely he has messed up locking anyway and unexpected go-routine-local-data is the least of his problems.

Next, for debugging purposes it is very useful to add such a go-routine ID to the log output.

 

> For example, in a web server I create a Go routine for each user request.
> This go routine calls several functions. To each of these functions I have
> to pass the session ID which is quite nasty on the long run, especially when
> calling through libraries which are not under my control. Attaching the
> session ID to the go routine would simplify the issue a lot.

Until you want to create a second goroutine to help with the request.

Of course a sensitive solution to go-routine-local-data would inherit the data from its parent go routine.

Torben

 

Russ

--
---------------------------
Prof. Torben Weis
Universitaet Duisburg-Essen
torbe...@gmail.com

[Torben Weis]

2011/4/21 Russ Cox <r...@google.com>

There are already goroutine-local variables: they are called
function arguments, function return values, and local variables.

Funny :-) In theory you are right, in practice it does not help.

Imagine a web server calling into a HTML template library (written by somebody else) and this lib is calling back into the database layer of the web server.

Chances are that the template lib will make it hard or impossible to pass along some context information.

Torben

 

Russ

[Canopée]

Why do you think this library (which cannot handle specific parameters)
will use the same goroutine for the callback ?

Le 21/04/2011 15:57, Torben Weis a �crit :

[Russ Cox]

> Funny :-) In theory you are right, in practice it does not help.
> Imagine a web server calling into a HTML template library (written by
> somebody else) and this lib is calling back into the database layer of the
> web server.

So far I don't see the problem. Presumably the template library
is being given an interface or function value to call, and in either
case you can construct one containing the appropriate context.

Russ

[Torben Weis]

2011/4/21 Canopée <cano.p...@gmail.com>

Why do you think this library (which cannot handle specific parameters)
will use the same goroutine for the callback ?

As long as it calls back from a child-go-routine, the data is preserved (see my previous mail) and everything is fine.

If the lib is using some pool of go-routines to callback from then it won't work.

I do not claim that the technique is fool proof. But which multi-threading technique is?

When reusing a 3rd party library in an application then its documentation should state whether it calls back asynchronously from different threads (go routines), because then I have to get my locking right. If the lib does such a thing (I assume most don't), go-routine-local-data is no solution. Otherwise it does the job.

Torben

[Torben Weis]

So far I don't see the problem.  Presumably the template library
is being given an interface or function value to call, and in either
case you can construct one containing the appropriate context.

- True, if I create a new instance of the template engine for every incoming request.

I would simply attach a closure to call back into, or I wrap the DB interface in an object that stores the context for me.

- False, if I call into the same engine from multiple requests in parallel, because the template engine is designed as a thread-safe singleton.

I think your question boils down to: Is there a use case that cannot be solved without go-routine-local-data?

In the end there is none, because with enough effort (including changing the library) you can pass context data along.

Thus, there is a trade-off between three things

a) Keep the runtime simple

b) Avoid techniques that may be misused to shoot holes in people's feet

c) Provide easy to use idioms for common tasks to simplify programming

IMHO carrying context along a call chain is an often recurring problem that can be greatly simplified by go-routine-local-data.

Regarding the danger, I assume that by using a map (as outlined before) and by inheriting go-routine-local-data the risk for people's is limited.

Obviously, it adds yet another concept to the runtime and thus complexity.

In the end it is a matter of taste.

Torben

Russ

[Russ Cox]

Go's use of threads has broken a surprising number of
C libraries that turned out to have hidden thread-local state.
Go is not going to make the same mistake.

Russ

[Ian Lance Taylor]

Torben Weis <torbe...@gmail.com> writes:

> Of course a sensitive solution to go-routine-local-data would inherit the
> data from its parent go routine.

I've thought about that before, and it is attractive in some ways. In
Go you need that inheritance across goroutines. However, once you add
it, you can see that you are not actually talking about goroutine-local
data. You are talking about dynamic scoping, similar to dynamic scoping
in LISP implementations. Dynamic scoping is a powerful technique, but
most languages do not offer it, because most programmers find it
confusing.

The argument about difficulties in passing context is only meaningful in
Go in very special circumstances: those in which for some reason the
callee needs to call back to specific names in the caller. Most people
do not build library callbacks that way. When a library callback does
not require a specific name, then it is trivial in Go to create a
closure which includes all the context information you might want.

Ian

[roger peppe]

the only place i really miss having a goroutine id available
is for log messages. it's much easier to work out
what was going on if you can separate the log messages
into individual sequential streams.

obviously it's possible to pass around a context,
but when searching for a bug through several third
party packages, adding a context is often much too
much work.

if it was possible to allow just this without letting
it be abused for thread-local storage, i'd definitely be in favour,
but i'm not sure it is.

[Torben Weis]

> the only place i really miss having a goroutine id available
> is for log messages. it's much easier to work out
> what was going on if you can separate the log messages
> into individual sequential streams.
>
> obviously it's possible to pass around a context,
> but when searching for a bug through several third
> party packages, adding a context is often much too
> much work.
>
> if it was possible to allow just this without letting
> it be abused for thread-local storage, i'd definitely be in favour,
> but i'm not sure it is.

Even if the go-routine ID is assigned by the runtime (i.e.
applications cannot influence the ID), then a "clever" coder can still
build a map that maps the ID to thread-local data and this app will
suffer from the confusion that Russ fears.

Thus, it is either all or nothing. Given the discussion so far it
looks more like "nothing" :-(
especially since it is hard to prove that the future harm resulting
from this feature is harmless.

Seems that the design rule for Go is: If you feel like passing around
context then do it explicitly. Do not try to attach it to a go-routine
implicitly.

Torben

> On 21 April 2011 21:06, Ian Lance Taylor <ia...@google.com> wrote:
>> Torben Weis <torbe...@gmail.com> writes:
>>
>>> Of course a sensitive solution to go-routine-local-data would inherit the
>>> data from its parent go routine.
>>
>> I've thought about that before, and it is attractive in some ways.  In
>> Go you need that inheritance across goroutines.  However, once you add
>> it, you can see that you are not actually talking about goroutine-local
>> data.  You are talking about dynamic scoping, similar to dynamic scoping
>> in LISP implementations.  Dynamic scoping is a powerful technique, but
>> most languages do not offer it, because most programmers find it
>> confusing.
>>
>> The argument about difficulties in passing context is only meaningful in
>> Go in very special circumstances: those in which for some reason the
>> callee needs to call back to specific names in the caller.  Most people
>> do not build library callbacks that way.  When a library callback does
>> not require a specific name, then it is trivial in Go to create a
>> closure which includes all the context information you might want.
>>
>> Ian
>>
>

--

[bflm]

On Apr 21, 11:03 pm, Torben Weis <torben.w...@gmail.com> wrote:
> Seems that the design rule for Go is: If you feel like passing around
> context then do it explicitly. Do not try to attach it to a go-routine
> implicitly.

I don't see it as a design rule. If the (only) "reason" to introduce a
goroutine ID is a broken design of some third party package (lacking a
way to properly pass to it a required context) then a "remedy" via
twisting the language is just plain wrong.

[Torben Weis]

2011/4/21 bflm <befeleme...@gmail.com>:

First of all, it is not the only use case as discussed before.

Second, I sincerely wish you a job where you can always reject using
some existing code base because it is not designed the way it could
have been in a perfect world.

Torben

[Russ Cox]

> Second, I sincerely wish you a job where you can always reject using
> some existing code base because it is not designed the way it could
> have been in a perfect world.

We can do that. We're working on a new language.
If we had to accept everything that already existed
we'd still be using C++.

Russ

[Paul Borman]

When you start talking about needing thread ID's, thread local storage, often shared memory is not far behind.  I suggest reading "The Problem with Threads":

    http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

The authors of the Go language are not new to concurrent programming.  I have also done quite a bit of research in concurrent programming (symmetric, asymmetric, and distributed).  Haore's "Communicating Sequential Processes (CSP)"

    http://www.usingcsp.com/cspbook.pdf

Is also a good read.  I think Go's channels combined with Go Routines really provide the right facilities to write robust code.  If you mistakenly think of them as threads and try to do with Go what others have done with threads in other languages I am glad you will find it difficult since the reason it is difficult is the original paradigm was flawed.'

If you want to use C or C++, please do so!

    -Paul

[Paul Borman]

[Torben Weis]

> We can do that.  We're working on a new language.
> If we had to accept everything that already existed
> we'd still be using C++.

Sure. However, a good language supports reuse of code instead of
enforcing a redesign.

For example GO's interfaces allow me to take some object that has
never explicitly implemented some interface and cast it to this
interface nevertheless.
That's extremely cool, because I do not have to change the code of
this other object to explicitly mention the new interface.
One could even argue that it is dangerous, because the object might
implement the interface syntactically, but the semantics are
different.
Nevertheless, I guess nobody who has ever written Go would like to
loose this feature.

Passing data along a call chain implicitly (i.e. by attaching it to a
go-routine) is a tool that follows the same goal: Make it possible to
use code that is there and that has not explicitly being designed for
this purpose. And yes, there is a chance that something goes wrong.

I fully accept if the Go designers say the benefit does not value the
risk of something going wrong.
Most choices in new languages are not really ground breaking new.
Most of them are old choices, but new languages have the freedom to
choose a different answer.

BTW: If passing context along in local variables is the best way of
doing it, why does UNIX feature a function to obtain the PID? You
could pass it to the main function and from there to the place where
you need it. This way your code does not get confused if the PID
suddenly changes after a fork :-)

My last mail on this topic, promised.

Cheers
Torben

> Russ
>

[Paul Borman]

On Thu, Apr 21, 2011 at 4:20 PM, Torben Weis <torbe...@gmail.com> wrote:

Sure. However, a good language supports reuse of code instead of
enforcing a redesign.

Reuse of what code?  Go is a new language.  I think Russ' point is that we are getting a fresh start here and we should not have to accept libraries written with one paradigm (say pre-concurrent programming) and now be forced into a new paradigm.

    -Paul

[roger peppe]

On 22 April 2011 00:20, Torben Weis <torbe...@gmail.com> wrote:
> BTW: If passing context along in local variables is the best way of
> doing it, why does UNIX feature a function to obtain the PID?

unix provides getpid so you can do things with a process
without its consent - you can kill it, set its priority, query its
statistics etc.

in Go you can do none of those things.

[John Asmuth]

On Thursday, April 21, 2011 7:20:34 PM UTC-4, Torben wrote:

BTW: If passing context along in local variables is the best way of
doing it, why does UNIX feature a function to obtain the PID? 

Not to knock unix, which is a wonderful thing... the fact that unix does something one way does not mean it's the only or even the best way. It's just what seemed like a good idea back in the 50s *ducks* or whenever it was created. 

[Michael Uhlenberg]

You can add the following to go/src/pkg/runtime/proc.c:

void runtime.Goid(int32 ret)
{
    ret = g->goid;
    FLUSH(&ret);
}
Take runtime.Goroutines as template. I don't know (yet) how to enter the centered dot after runtime.

In your go code you can then simply say (after importing the runtime package)  println("my goid:", Goid())

[Hotei]

+1 to Paul'ls comment

I've converted a fair number of smallish C / C++ programs (about 1,000 LOC each) and found that re-factoring them to use Go paradigms - while not always easy - in the end made the code much easier to re-use elsewhere (in other Go programs) and slightly more robust in the long run.  Part of the benefit resulted from the "what you see is _exactly_ what it does" aspect of go code and part was the ability to factor out more/larger chunks of repetitive stuff.  

I think go is mature enough now to need a 'feature-freeze'.  If in a few years things look different there's always time for version 1.1, but we need 1.0 first (soon please! :-)

[Michael Uhlenberg]

You must apparently also add
func Goid() int32
to debug,go and
void    runtime·Goid(int32);
to runtime.h, all in src/pkg/runtime.