global goroutine data / thread local storage?

[Alex Mills]

Since by default all http requests coming to a go http server are on their own goroutine, I am wondering if there is a way to have some sort of "global" variable that is local to a goroutine if that makes sense, something like this, where "gork" is the namespace for variables available anywhere within a goroutine:

func PrintLoggedInUser(){

    log.Println(gork.loggedInUser)

}

go func(){

     var loggedInUser = "abc"

     PrintLoggedInUser()

}()

go func(){

     var loggedInUser = "def"

     PrintLoggedInUser()

}()

why? i am looking to log the current logged in user id, without having to manually pass that id to every log call.

log.Warn("foo")  // will log:  "foo", "logged in user id:", abc

log.Error("bar")  // will log:  "bar", "logged in user id:", abc

but for now, I have to pass the id manually:

log.Warn(id, "foo")  // will log:  "foo", "logged in user id:", abc

log.Error(id, "bar")  // will log:  "bar", "logged in user id:", abc

[Joop Kiefte]

As far as I know, that is exactly what the Context package and customs are meant for.

Joop Kiefte - Chat @ Spike

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/07fb2e73-14a2-4559-a7d6-2010acbc7c51n%40googlegroups.com.

[burak serdar]

On Wed, Sep 23, 2020 at 6:17 PM Alex Mills <al...@channelmeter.com> wrote:
>
> Since by default all http requests coming to a go http server are on their own goroutine, I am wondering if there is a way to have some sort of "global" variable that is local to a goroutine if that makes sense, something like this, where "gork" is the namespace for variables available anywhere within a goroutine:
>
>
> func PrintLoggedInUser(){
> log.Println(gork.loggedInUser)
> }
>
> go func(){
> var loggedInUser = "abc"
> PrintLoggedInUser()
> }()
>
> go func(){
> var loggedInUser = "def"
> PrintLoggedInUser()
> }()
>
>
> why? i am looking to log the current logged in user id, without having to manually pass that id to every log call.

Usually, this kind of information is placed in the http request
context, so all the function down the call chain can have access to
the variables in the context. You have to pass that context to the
call chain.

>
>
> log.Warn("foo") // will log: "foo", "logged in user id:", abc
> log.Error("bar") // will log: "bar", "logged in user id:", abc
>
> but for now, I have to pass the id manually:
>
> log.Warn(id, "foo") // will log: "foo", "logged in user id:", abc
> log.Error(id, "bar") // will log: "bar", "logged in user id:", abc
>
>
>

[Alex Mills]

Not sure about that. Context can store a map. I could map:

request => userid

but then I would have to pass the request to the logging statements, so that would put me back at square one.

There appears to be a way to get a reference on the goroutine id:

http://blog.sgmansfield.com/2015/12/goroutine-ids/

so if I could have a global map of:

goroutineid => userid

then in the current goroutine, I could look in the global map to see if the id is in there?

if the id is not in there, it's not really a fatal problem, so it can be fuzzy.

so before it looks like:

func Warn(r *http.Request, ...args){

   var id = context.Get(r)

   fmt.Println("WARN: logged in user id", args...)

}

the new func would look like:

var m = map[string]string{}

func Warn(..args){

   var gid = getGoroutineId()

   var loggedInUserId = m[gid]

   fmt.Println("WARN: logged in user id:", loggedInUserId, args...)

}

make sense? But I wonder if it's performant to using this get the goroutine id heavily:

func getGID() uint64 {
   b := make([]byte, 64)
   b = b[:runtime.Stack(b, false)]
   b = bytes.TrimPrefix(b, []byte("goroutine "))
   b = b[:bytes.IndexByte(b, ' ')]
   n, _ := strconv.ParseUint(string(b), 10, 64)
   return n
}

-alex

[Ian Lance Taylor]

On Wed, Sep 23, 2020 at 5:46 PM Alex Mills <al...@channelmeter.com> wrote:
>
> There appears to be a way to get a reference on the goroutine id:
>
> http://blog.sgmansfield.com/2015/12/goroutine-ids/

But as you can see by reading that blog article, that is almost a joke.

Go considers these things to be better handled explicitly, which is
why people are telling you to use a context.Context value. And, yes,
you'll want to use a Context aware logging package.

In Go it's trivial to create new goroutines, and as soon as you do
that any goroutine-local-variable scheme falls apart. So Go has
consistently chosen to not provide that capability, and similarly to
not provide goroutine IDs. It's an intentional choice by the
language. There have been a number of discussions about this in the
past on this mailing list.

Ian

[Alex Mills]

Not a joke, in terms of performance, if you access the goroutine id via the C library call?

My only concern would be if goroutine id's were reused, if not it would work.

[Alex Mills]

But I tried compiling this program:

https://github.com/davecheney/junk/tree/master/id

and I cant get it to compile/run, the runtime.h file seems MIA

-alex

[Dan]

On Wed, 2020-09-23 at 18:12 -0700, Alex Mills wrote:
> But I tried compiling this program:
> https://github.com/davecheney/junk/tree/master/id
>
> and I cant get it to compile/run, the runtime.h file seems MIA
>
> -alex

There is this: https://github.com/kortschak/goroutine

Don't use it; do take people's advice here and explicitly pass the
information you need to log.

[Ian Lance Taylor]

On Wed, Sep 23, 2020 at 6:02 PM Alex Mills <al...@channelmeter.com> wrote:
>
> Not a joke, in terms of performance, if you access the goroutine id via the C library call?

I'm not sure what C library call you mean.

> My only concern would be if goroutine id's were reused, if not it would work.

Goroutine IDs are not reused. (Well, a goroutine ID is just a 64-bit
integer so it could technically wrap around and be reused in that way,
but it seems unlikely.)

Ian

[Alex Besogonov]

Inheritable goroutine-locals would actually work just fine in Go. Moreover, Go actually has them in the form of pprof labels.

[Ian Lance Taylor]

On Fri, Sep 25, 2020 at 10:35 AM Alex Besogonov
<alex.be...@gmail.com> wrote:
>
> Inheritable goroutine-locals would actually work just fine in Go. Moreover, Go actually has them in the form of pprof labels.

What should happen if one goroutine changes an inherited
goroutine-local variable?

Ian

> On Wednesday, September 23, 2020 at 5:55:50 PM UTC-7 Ian Lance Taylor wrote:
>>
>> On Wed, Sep 23, 2020 at 5:46 PM Alex Mills <al...@channelmeter.com> wrote:
>> >
>> > There appears to be a way to get a reference on the goroutine id:
>> >
>> > http://blog.sgmansfield.com/2015/12/goroutine-ids/
>>
>> But as you can see by reading that blog article, that is almost a joke.
>>
>> Go considers these things to be better handled explicitly, which is
>> why people are telling you to use a context.Context value. And, yes,
>> you'll want to use a Context aware logging package.
>>
>> In Go it's trivial to create new goroutines, and as soon as you do
>> that any goroutine-local-variable scheme falls apart. So Go has
>> consistently chosen to not provide that capability, and similarly to
>> not provide goroutine IDs. It's an intentional choice by the
>> language. There have been a number of discussions about this in the
>> past on this mailing list.
>>
>> Ian
>

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f78f2d12-89d2-494d-983a-a462f0124d82n%40googlegroups.com.

[Alex Besogonov]

If you mutate the value, then only the current goroutine is affected (not even its existing children). Basically, the goroutine-local context is copied on new goroutine launch.

That's how pprof labels work right now, btw.

[Alex Besogonov]

On Friday, September 25, 2020 at 4:43:45 PM UTC-7 Ian Lance Taylor wrote:

On Fri, Sep 25, 2020 at 10:35 AM Alex Besogonov
<alex.be...@gmail.com> wrote:
>
> Inheritable goroutine-locals would actually work just fine in Go. Moreover, Go actually has them in the form of pprof labels.
What should happen if one goroutine changes an inherited
goroutine-local variable?

A Go-style goroutine-local system can be thought as a context.Context that is transparently assigned to a goroutine and its children. Thus if you want to change the value, you'd do something like "c := magic.GetCurrentContext(); c = c.WithValue(...); magic.SetContext(c);".

This would affect only the current goroutine going forward, but not its children.

[roger peppe]

On Sat, 26 Sep 2020, 02:36 Alex Besogonov, <alex.be...@gmail.com> wrote:

On Friday, September 25, 2020 at 4:43:45 PM UTC-7 Ian Lance Taylor wrote:

On Fri, Sep 25, 2020 at 10:35 AM Alex Besogonov
<alex.be...@gmail.com> wrote:
>
> Inheritable goroutine-locals would actually work just fine in Go. Moreover, Go actually has them in the form of pprof labels.
What should happen if one goroutine changes an inherited
goroutine-local variable?

A Go-style goroutine-local system can be thought as a context.Context that is transparently assigned to a goroutine and its children. Thus if you want to change the value, you'd do something like "c := magic.GetCurrentContext(); c = c.WithValue(...); magic.SetContext(c);".

This would affect only the current goroutine going forward, but not its children.

What happens if you want to invoke an operation by passing its arguments to an existing goroutine that does work on the current goroutine's behalf (aka a worker pool)?

I don't think this pattern is that uncommon, and it would break the inheritable-variables model. 

 

> On Wednesday, September 23, 2020 at 5:55:50 PM UTC-7 Ian Lance Taylor wrote:
>>
>> On Wed, Sep 23, 2020 at 5:46 PM Alex Mills <al...@channelmeter.com> wrote:
>> >
>> > There appears to be a way to get a reference on the goroutine id:
>> >
>> > http://blog.sgmansfield.com/2015/12/goroutine-ids/
>>
>> But as you can see by reading that blog article, that is almost a joke.
>>
>> Go considers these things to be better handled explicitly, which is
>> why people are telling you to use a context.Context value. And, yes,
>> you'll want to use a Context aware logging package.
>>
>> In Go it's trivial to create new goroutines, and as soon as you do
>> that any goroutine-local-variable scheme falls apart. So Go has
>> consistently chosen to not provide that capability, and similarly to
>> not provide goroutine IDs. It's an intentional choice by the
>> language. There have been a number of discussions about this in the
>> past on this mailing list.
>>
>> Ian
>
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f78f2d12-89d2-494d-983a-a462f0124d82n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7bc62278-5cbc-47fe-abd2-36172e4e8520n%40googlegroups.com.

[Ian Lance Taylor]

On Fri, Sep 25, 2020 at 6:29 PM Alex Besogonov <alex.be...@gmail.com> wrote:
>
> If you mutate the value, then only the current goroutine is affected (not even its existing children). Basically, the goroutine-local context is copied on new goroutine launch.
>
> That's how pprof labels work right now, btw.

Labels are not variables. Variables have values that can be changed.

For example, if we had goroutine local variables then code like

goroutine var G int32

func F(s []P) {
for _, p := range s {
G += p.SlowMethod()
}
}

might get changed to

func F(s []P) {
for _, p := range s {
p := p
go func() {
atomic.Add(&G, p.SlowMethod())
}()
}
}

but the two functions would work very differently.

Ian

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ee6daca3-6e85-458e-95fd-877e8af4368dn%40googlegroups.com.

[Ian Lance Taylor]

On Fri, Sep 25, 2020 at 6:36 PM Alex Besogonov <alex.be...@gmail.com> wrote:
>
> On Friday, September 25, 2020 at 4:43:45 PM UTC-7 Ian Lance Taylor wrote:
>>
>> On Fri, Sep 25, 2020 at 10:35 AM Alex Besogonov
>> <alex.be...@gmail.com> wrote:
>> >
>> > Inheritable goroutine-locals would actually work just fine in Go. Moreover, Go actually has them in the form of pprof labels.
>> What should happen if one goroutine changes an inherited
>> goroutine-local variable?
>
> A Go-style goroutine-local system can be thought as a context.Context that is transparently assigned to a goroutine and its children. Thus if you want to change the value, you'd do something like "c := magic.GetCurrentContext(); c = c.WithValue(...); magic.SetContext(c);".
>
> This would affect only the current goroutine going forward, but not its children.

In general, Go's attitude is that explicit is better than implicit.
If you rigorously stick to the pattern you describe, it's the same as
passing the context.Context around explicitly (except that passing the
Context around explicitly is more flexible).

Ian

[Alex Besogonov]

On Saturday, September 26, 2020 at 5:41:54 AM UTC-7 rog wrote:

A Go-style goroutine-local system can be thought as a context.Context that is transparently assigned to a goroutine and its children. Thus if you want to change the value, you'd do something like "c := magic.GetCurrentContext(); c = c.WithValue(...); magic.SetContext(c);".

This would affect only the current goroutine going forward, but not its children.

What happens if you want to invoke an operation by passing its arguments to an existing goroutine that does work on the current goroutine's behalf (aka a worker pool)?

I don't think this pattern is that uncommon, and it would break the inheritable-variables model. 

Worker pools with long-lived goroutines are actually very rare in Go, because launching a new goroutine is easy. A worker pool that is created in-place (for example, to run parallel processing on data) would work just fine.

[Alex Besogonov]

On Saturday, September 26, 2020 at 12:33:40 PM UTC-7 Ian Lance Taylor wrote:

On Fri, Sep 25, 2020 at 6:29 PM Alex Besogonov <alex.be...@gmail.com> wrote:
> If you mutate the value, then only the current goroutine is affected (not even its existing children). Basically, the goroutine-local context is copied on new goroutine launch.
> That's how pprof labels work right now, btw.
Labels are not variables. Variables have values that can be changed.

Most people don't need fully mutable goroutine-specific variables, but something like an automatic context carrier. It's immutable, so this would mostly solve the mutation problem.

So in your example you'll have to write:

for _, p := range s { threadCtx = threadCtx.WithValue(p.SlowMethod(threadCtx.get()));}

 

This makes it clear that there's causal relationship between the current value and the next one, so the loop can't be naïvely parallelized. Of course, a very determined programmer can still shoot themselves in the foot (by reading and replacing the automatic context inside p.SlowMethod), but the footgun here is pretty explicit.

[Alex Besogonov]

On Saturday, September 26, 2020 at 12:34:37 PM UTC-7 Ian Lance Taylor wrote:

> A Go-style goroutine-local system can be thought as a context.Context that is transparently assigned to a goroutine and its children. Thus if you want to change the value, you'd do something like "c := magic.GetCurrentContext(); c = c.WithValue(...); magic.SetContext(c);".
> This would affect only the current goroutine going forward, but not its children.
In general, Go's attitude is that explicit is better than implicit.
If you rigorously stick to the pattern you describe, it's the same as
passing the context.Context around explicitly (except that passing the
Context around explicitly is more flexible).

Well, I ran a simple regexp grep over my existing code base. Out of 8153 functions I have 7632 functions that have context.Context as a parameter. So basically in my Go code every function has to have about half a line of extra fluff (context and the error return).

I'm ambivalent about goroutine-specific variables. Context is not a terribly bad solution, I can personally live with it just fine. Unfortunately, there are _STILL_ APIs that don't have context. I was bitten by JSON marshalling and SQL. And apparently newer APIs are being designed by the Golang team that don't accept a context (because it doesn't look nice).