package-level structs vs package-level vars
114 views
Skip to first unread message
B Carr
Aug 4, 2019, 5:19:16 PM8/4/19
to golang-nuts
Concept observation. Could use some amplification from the congregants, please...
I'm learning about Go.
My webserver program had lots of package level vars and only two, small structs. I set a dozen webpages to refresh very busy data calculations every 6 seconds. Over the course of an hour, looking at the pprof output for 'alloc' & 'heapAlloc' it looked like the GC was very busy and collected >6 GiB of stuff. It looked like it would have continued increasing into infinity. Even during that time, the GC rarely went over 10ms active times.
I got rid of all the package level vars and put them inside of structs and pass around the struct values to the functions as needed. Using the same test setup above, I set a dozen webpages to refresh very busy data calculations every 6 seconds. Over the course of an hour, looking at the pprof output for 'alloc' & 'heapAlloc' it looked like the GC just loafed along and collected <10 MiB of stuff. And that number was pretty level. The GC was showing mostly zero nanosecond times.
I didn't do any timing testing, but the speediness of all the webpages felt about the same in both test cases.
I have a notion about what I'm seeing. Anyone care to increase my level of understanding? Are my interpretations about what pprof is showing me accurate?
(ps - a big thank you to Burak Serdar who, in a previous thread, showed me something I hadn't considered and put me on the path to making my webserver program successfully concurrent!)
TIA,
Bucky
burak serdar
Aug 4, 2019, 11:41:39 PM8/4/19
to B Carr, golang-nuts
On Sun, Aug 4, 2019 at 11:19 AM B Carr <buc...@gmail.com> wrote:
>
> Concept observation. Could use some amplification from the congregants, please...
>
> I'm learning about Go.
>
> My webserver program had lots of package level vars and only two, small structs. I set a dozen webpages to refresh very busy data calculations every 6 seconds. Over the course of an hour, looking at the pprof output for 'alloc' & 'heapAlloc' it looked like the GC was very busy and collected >6 GiB of stuff. It looked like it would have continued increasing into infinity. Even during that time, the GC rarely went over 10ms active times.
>
> I got rid of all the package level vars and put them inside of structs and pass around the struct values to the functions as needed. Using the same test setup above, I set a dozen webpages to refresh very busy data calculations every 6 seconds. Over the course of an hour, looking at the pprof output for 'alloc' & 'heapAlloc' it looked like the GC just loafed along and collected <10 MiB of stuff. And that number was pretty level. The GC was showing mostly zero nanosecond times.
The GC activity will depend on what goes to the heap. I can't say I
>
> Concept observation. Could use some amplification from the congregants, please...
>
> I'm learning about Go.
>
> My webserver program had lots of package level vars and only two, small structs. I set a dozen webpages to refresh very busy data calculations every 6 seconds. Over the course of an hour, looking at the pprof output for 'alloc' & 'heapAlloc' it looked like the GC was very busy and collected >6 GiB of stuff. It looked like it would have continued increasing into infinity. Even during that time, the GC rarely went over 10ms active times.
>
> I got rid of all the package level vars and put them inside of structs and pass around the struct values to the functions as needed. Using the same test setup above, I set a dozen webpages to refresh very busy data calculations every 6 seconds. Over the course of an hour, looking at the pprof output for 'alloc' & 'heapAlloc' it looked like the GC just loafed along and collected <10 MiB of stuff. And that number was pretty level. The GC was showing mostly zero nanosecond times.
understand your program structure from your description, but it sounds
like in the first case you have global variables, but in the second
case you started declaring variables in your functions and passed them
around. If that's really the case, then GC will have less to do in the
second case as most of the memory will be allocated on the stack, and
GC will not be involved in cleaning them up.
>
> I didn't do any timing testing, but the speediness of all the webpages felt about the same in both test cases.
>
> I have a notion about what I'm seeing. Anyone care to increase my level of understanding? Are my interpretations about what pprof is showing me accurate?
>
> (ps - a big thank you to Burak Serdar who, in a previous thread, showed me something I hadn't considered and put me on the path to making my webserver program successfully concurrent!)
>
> TIA,
>
> Bucky
>
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/fb49d5bb-f795-4ca3-af5c-d883aabcf1af%40googlegroups.com.
Robert Engels
Aug 5, 2019, 12:07:04 AM8/5/19
to burak serdar, B Carr, golang-nuts
Just because you declare in functions and pass around does not have anything to do if it will be on the heap or the stack. It depends on if the compiler can detect if the object does not “escape”.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMV2RqrEB7nOOx-CytFXYiTu7s5PosPZP%3D1ugJsE6MxK6KtwJA%40mail.gmail.com.
burak serdar
Aug 5, 2019, 12:16:59 AM8/5/19
to Robert Engels, B Carr, golang-nuts
On Sun, Aug 4, 2019 at 6:06 PM Robert Engels <ren...@ix.netcom.com> wrote:
>
> Just because you declare in functions and pass around does not have anything to do if it will be on the heap or the stack. It depends on if the compiler can detect if the object does not “escape”.
Right. However, if the variables are declared globally (as I
>
> Just because you declare in functions and pass around does not have anything to do if it will be on the heap or the stack. It depends on if the compiler can detect if the object does not “escape”.
understand it is the first case in the post), then there is no chance
that they will be on stack.
Robert Engels
Aug 5, 2019, 12:25:55 AM8/5/19
to burak serdar, B Carr, golang-nuts
I’m pretty sure they will be in the data section, for non interface/pointer types which is even better than the stack.
B Carr
Aug 5, 2019, 2:14:19 AM8/5/19
to golang-nuts
burak serdar
Aug 5, 2019, 2:52:41 AM8/5/19
to B Carr, golang-nuts
On Sun, Aug 4, 2019 at 8:14 PM B Carr <buc...@gmail.com> wrote:
>
>
> Would that be where structs live as well?
Any global variable, struct or not, would be in the data section. For
>
>
> Would that be where structs live as well?
interfaces/pointers, the variable itself will be in the data section,
but where it points may be in the heap.
Any variable you declare within a function that escapes will be in
heap. These are things such as having a global cache, and cache
elements allocated within a function. Something similar to this is
what I understood when I read your description in your first post.
Any variable you declare within a function that is shared between
goroutines escapes, and that will be in the heap. There are other ways
a variable can escape.
If a variable is declared in a function and it doesn't escape, then it
will be on stack.
>
> On Sunday, August 4, 2019 at 6:25:55 PM UTC-6, Robert Engels wrote:
>>
>> I’m pretty sure they will be in the data section, for non interface/pointer types which is even better than the stack.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/96d1c669-a65c-48ad-9ac1-17663e20c416%40googlegroups.com.
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
Robert Engels
Aug 5, 2019, 3:03:49 AM8/5/19
to burak serdar, B Carr, golang-nuts
I would assume so. That is the advantage of Go compared to others in that is has value objects. Someone else can confirm.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMV2RqoVmnY%2B-n1-jgtdbq9TizaciAm3e7Wp%3DwENr3NZbtBnmw%40mail.gmail.com.
B Carr
Aug 5, 2019, 2:47:22 PM8/5/19
to golang-nuts
Is it safe to say that environment variables (accessible to all goroutines) also live in the data segment?
burak serdar
Aug 5, 2019, 2:52:41 PM8/5/19
to B Carr, golang-nuts
On Mon, Aug 5, 2019 at 8:47 AM B Carr <buc...@gmail.com> wrote:
>
> Is it safe to say that environment variables (accessible to all goroutines) also live in the data segment?
No.
>
> Is it safe to say that environment variables (accessible to all goroutines) also live in the data segment?
When you say "environment variables", do you mean OS env variables, or
the variables visible to a goroutine?
If you're talking about the variables visible to a goroutine, where
the variable lives depends on how it is declared and allocated.
If the variable is allocated at compile time (like a global var/struct
initialized with a literal) it lives in data segment.
If the variable is allocated dynamically using new(), or it is
declared in a function and it escapes, it lives on the heap.
>
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
B Carr
Aug 5, 2019, 3:19:50 PM8/5/19
to golang-nuts
On Monday, August 5, 2019 at 8:52:41 AM UTC-6, burak serdar wrote:
On Mon, Aug 5, 2019 at 8:47 AM B Carr <buc...@gmail.com> wrote:
>
> Is it safe to say that environment variables (accessible to all goroutines) also live in the data segment?
No.
When you say "environment variables", do you mean OS env variables, or
the variables visible to a goroutine?
If you're talking about the variables visible to a goroutine, where
the variable lives depends on how it is declared and allocated.
If the variable is allocated at compile time (like a global var/struct
initialized with a literal) it lives in data segment.
If the variable is allocated dynamically using new(), or it is
declared in a function and it escapes, it lives on the heap.
Oh. I'm talking about env variables which a goroutine sets by using os.Setenv() which to me implies OS env variables. Are those env variables accessible by all goroutines regardless of which goroutine set the env variable? Would such env variables be available to newer goroutines that spin up? I think the answer to this is "Yes", based on a response I got to another thread.
I'm apparently being too black/white in my thinking about where things get put into memory and your explanations are helping to get my mind right.
burak serdar
Aug 5, 2019, 3:33:08 PM8/5/19
to B Carr, golang-nuts
On Mon, Aug 5, 2019 at 9:20 AM B Carr <buc...@gmail.com> wrote:
>
>
>
> On Monday, August 5, 2019 at 8:52:41 AM UTC-6, burak serdar wrote:
>>
>> On Mon, Aug 5, 2019 at 8:47 AM B Carr <buc...@gmail.com> wrote:
>> >
>> > Is it safe to say that environment variables (accessible to all goroutines) also live in the data segment?
>>
>> No.
>>
>> When you say "environment variables", do you mean OS env variables, or
>> the variables visible to a goroutine?
>>
>> If you're talking about the variables visible to a goroutine, where
>> the variable lives depends on how it is declared and allocated.
>>
>> If the variable is allocated at compile time (like a global var/struct
>> initialized with a literal) it lives in data segment.
>>
>> If the variable is allocated dynamically using new(), or it is
>> declared in a function and it escapes, it lives on the heap.
>
>
> Oh. I'm talking about env variables which a goroutine sets by using os.Setenv() which to me implies OS env variables. Are those env variables accessible by all goroutines regardless of which goroutine set the env variable? Would such env variables be available to newer goroutines that spin up? I think the answer to this is "Yes", based on a response I got to another thread.
Those env variables belong to the process. All gorountines share them.
>
>
>
> On Monday, August 5, 2019 at 8:52:41 AM UTC-6, burak serdar wrote:
>>
>> On Mon, Aug 5, 2019 at 8:47 AM B Carr <buc...@gmail.com> wrote:
>> >
>> > Is it safe to say that environment variables (accessible to all goroutines) also live in the data segment?
>>
>> No.
>>
>> When you say "environment variables", do you mean OS env variables, or
>> the variables visible to a goroutine?
>>
>> If you're talking about the variables visible to a goroutine, where
>> the variable lives depends on how it is declared and allocated.
>>
>> If the variable is allocated at compile time (like a global var/struct
>> initialized with a literal) it lives in data segment.
>>
>> If the variable is allocated dynamically using new(), or it is
>> declared in a function and it escapes, it lives on the heap.
>
>
> Oh. I'm talking about env variables which a goroutine sets by using os.Setenv() which to me implies OS env variables. Are those env variables accessible by all goroutines regardless of which goroutine set the env variable? Would such env variables be available to newer goroutines that spin up? I think the answer to this is "Yes", based on a response I got to another thread.
If you're setting env variables from goroutines, you might want to
think about thread safety of setenv.
>
> I'm apparently being too black/white in my thinking about where things get put into memory and your explanations are helping to get my mind right.
>
> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e064b3a9-19f1-48ca-8fab-e438a737b889%40googlegroups.com.
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
B Carr
Aug 5, 2019, 3:41:18 PM8/5/19
to golang-nuts
On Monday, August 5, 2019 at 9:33:08 AM UTC-6, burak serdar wrote:
On Mon, Aug 5, 2019 at 9:20 AM B Carr <buc...@gmail.com> wrote:
>
> Oh. I'm talking about env variables which a goroutine sets by using os.Setenv() which to me implies OS env variables. Are those env variables accessible by all goroutines regardless of which goroutine set the env variable? Would such env variables be available to newer goroutines that spin up? I think the answer to this is "Yes", based on a response I got to another thread.
Those env variables belong to the process. All gorountines share them.
If you're setting env variables from goroutines, you might want to
think about thread safety of setenv.
Oh yes. I'm using a mutex.
Thank you.
Reply all
Reply to author
Forward
0 new messages