Memory management int Golang runtime

[tomas....@gmail.com]

Hi golang-dev,

I'm playing around with the Go runtime. To learn more about it I'm trying to count netpoll open operations per P but I have a problem. I've added counters to the P struct and I atomically increment them, but once in a while they seem to be zeroed and I have no clue why. Why can this happen? Do I need to have something special in mind while writing code in this area? Am I doing something totally wrong?

I'd be more than grateful for any tips.

Cheers!

Tomas

[Dave Cheney]

Hi Tomas,

Can you share your code?

Thanks

Dave

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[tomas....@gmail.com]

Sure!

Lines added to src/runtime/netpoll_epoll.go:

func netpollopen(fd uintptr, pd *pollDesc) int32 {
        _p_ := getg().m.p.ptr()
        lock(&sched.lock)
        atomic.Xadd(&_p_.n_polls, 1)
        unlock(&sched.lock)
        // ...

I've also added n_polls uint32 to the p struct and adequate print to the shedtrace function. 

My example program just opens multiple TCP sockets with net.Listen method, each on different goroutine. Output from shedtrace shows that the counters are incremented as they should but at some point they are zeroed. I don't know either when nor why :(

Do I miss something obvious? 

Thanks a lot!

Tomas

[Dave Cheney]

It's hard to say, could you push your code to a branch so others can
try to reproduce the issue?

[tomas....@gmail.com]

The whole thing is here: https://github.com/tomaslither/go.

Example main is in the root directory: netpoller_problem.go

In the standard output can be noticed, that some P's have increased value for "netpollCounter=..." which is zeroed later.

Thank you for your time and patience.

Tomas

[tomas....@gmail.com]

Does anyone know anything regarding such a problem? :(

How can I even approach debugging? Where to look for documentation, how can I use GDB with Go runtime?

I'd be more than grateful for any clues.

Tomas

[Austin Clements]

Tomas, your code looks reasonable to me. There's no reason these counters should ever be zeroed. Can you give an example of what the output looks like that you're not expecting?

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

[Alexander Menzhinsky]

Are you sure that they're really zeroed?
Probably you get results from different goroutines, if I'm not mistaken at least the garbage collector is running in a separate one.

And my second guess is int overflow.

[tomas....@gmail.com]

Variable p.netpollCounter is iterated each time a netpoll is being opened. Example code opens 3 sockets using different goroutines, so the expected output from schedtrace should look, for example, as follows (may vary depending on P's allocation):

P1: ... netpollCounter=1

P2: ... netpollCounter=0

P3: ... netpollCounter=0

P4: ... netpollCounter=0

P1: ... netpollCounter=1

P2: ... netpollCounter=1

P3: ... netpollCounter=0

P4: ... netpollCounter=0

P1: ... netpollCounter=1

P2: ... netpollCounter=2

P3: ... netpollCounter=0

P4: ... netpollCounter=0

P1: ... netpollCounter=1

P2: ... netpollCounter=2

P3: ... netpollCounter=0

P4: ... netpollCounter=0

P1: ... netpollCounter=1

P2: ... netpollCounter=2

P3: ... netpollCounter=0

P4: ... netpollCounter=0

And so on. 

At the moment the variables are iterated correctly, but at the end the output is 0:

P1: ... netpollCounter=1

P2: ... netpollCounter=0

P3: ... netpollCounter=0

P4: ... netpollCounter=0

P1: ... netpollCounter=1

P2: ... netpollCounter=1

P3: ... netpollCounter=0

P4: ... netpollCounter=0

P1: ... netpollCounter=1

P2: ... netpollCounter=2

P3: ... netpollCounter=0

P4: ... netpollCounter=0

P1: ... netpollCounter=0

P2: ... netpollCounter=0

P3: ... netpollCounter=0

P4: ... netpollCounter=0

P1: ... netpollCounter=0

P2: ... netpollCounter=0

P3: ... netpollCounter=0

P4: ... netpollCounter=0

...

I have no idea why it happens :(

[Peter Waller]

I don't have time to run the code at this moment, but two suggestions come to mind:

Print the value of the pointer &_p_.netpollCounter - check that it's as you expect and consistent - where you need it to be.

If that all lines up, I would suggest trying to arrange a memory breakpoint on that pointer using GDB to find what is setting it to zero.

Interested to hear how it goes :)