find memory leak in go programs

[Vasiliy Tolstov]

Hello. How can i find code part, that contains memory leak?
What instruments i must use, and may somebody can check my code in
http://code.google.com/p/wiff/source/browse/#git%2Fwiff for leaking
memory?
And how can i minimize memory usage in my case? Now i have about 9-10
Mb in resident memory size and about 98 in virt...

--
Vasiliy Tolstov,
Clodo.ru
e-mail: v.to...@selfip.ru
jabber: va...@selfip.ru

[Lars Pensjö]

I suppose it is not really memory leaks you mean, as there is garbage collection. However, there is an excellent tool to find how much memory is "wasted" from different places in your program.

Use  pprof.WriteHeapProfile() and gopprof.

[Mike Samuel]

Go can leak memory. The naive queue implementation below expands an
array on Put() and after N calls to Take(), there will be N elements
of that array that are inaccessible via the public API, and so which
should be considered garbage.

Memory leaks can creep in wherever the GC's definition of garbage
(based on reachability) differs from the programmer's definition of
garbage based on utility and accessibility via APIs.


type Queue struct {
start int
els []int
}

func (q *Queue) Put(i int) {
// Adds to end.
q.els = append(q.els, i)
}

func (q *Queue) Take() (i int) {
if len(q.els) == q.start {
panic("empty")
}
i, q.start = q.els[q.start], q.start + 1
return
}

[Steven Blenkinsop]

That's not a memory leak. The memory will be collected after put has to reallocate the underlying array. Certainly this won't necessarily happen right away, but it's still not a leak. The memory can be freed eventually through use. You could always make an accumulator that can be grown, but not shrunk or accessed via it's public interface, but that's not really a leak either, just poor interface design.

You can, however, leak a goroutine along with any memory it uses by having it loop forever without communication, or block on a channel that is no longer held by any other goroutine.

Also, go currently has a conservative garbage collector, which means it's possible for the collector to overlook some garbage because it thinks there might be a pointer to it when there isn't.

[Mike Samuel]

2011/10/9 Steven Blenkinsop <stev...@gmail.com>:

> On Sunday, October 9, 2011, Mike Samuel <mikes...@gmail.com> wrote:
>> Go can leak memory.  The naive queue implementation below expands an
>> array on Put() and after N calls to Take(), there will be N elements
>> of that array that are inaccessible via the public API, and so which
>> should be considered garbage.
>>
>> Memory leaks can creep in wherever the GC's definition of garbage
>> (based on reachability) differs from the programmer's definition of
>> garbage based on utility and accessibility via APIs.
>>
>>
>> type Queue struct {
>>        start int
>>        els   []int
>> }
>>
>> func (q *Queue) Put(i int) {
>>        // Adds to end.
>>        q.els = append(q.els, i)
>> }
>>
>> func (q *Queue) Take() (i int) {
>>        if len(q.els) == q.start {
>>                panic("empty")
>>        }
>>        i, q.start = q.els[q.start], q.start + 1
>>        return
>> }
>>
>
> That's not a memory leak. The memory will be collected after put has to
> reallocate the underlying array. Certainly this won't necessarily happen
> right away, but it's still not a leak. The memory can be freed eventually

Sure, but then there will be another array with the same amount of
space that cannot be used by any public API, and continuous uses of
the public API can cause the amount of wasted space to grow without
bound. There will be O(calls-to-take) useless memory until the queue
becomes garbage.

> through use. You could always make an accumulator that can be grown, but not
> shrunk or accessed via it's public interface, but that's not really a leak
> either, just poor interface design.

> You can, however, leak a goroutine along with any memory it uses by having
> it loop forever without communication, or block on a channel that is no
> longer held by any other goroutine.
>
> Also, go currently has a conservative garbage collector, which means it's
> possible for the collector to overlook some garbage because it thinks there
> might be a pointer to it when there isn't.

My quibble is with defining "memory leak" based on garbage collection
at all. Not only does that definition not extend to languages without
GC that everyone agrees have memory leaks, it does not match developer
intuition.

A similarly naive queue of interface{}s that did not assign nil on
Take would maintain a pointer to something that is not accessible via
the public API. This seems like a leak to me.

I think we should be careful to take developer intuition into account
before saying that certain classes of bugs do not affect certain kinds
of systems.

[Steven Blenkinsop]

It's only a memory leak if the memory becomes no longer accessible or recoverable to the program. Neither is the case here. Firstly, the program can still access the memory from within the package, the code just chooses to perpetuate memory it knows it isn't going to use. Also, you can still get it collected by dropping the reference to the queue.  This is poor use of memory, but not a leak. There is a distinction, even though you don't seem to be making it. Yes, you can still have growing memory from poor memory use. Still a bug, just a different one.

[Mike Samuel]

2011/10/9 Steven Blenkinsop <stev...@gmail.com>:

You can keep asserting this definition and ignoring my arguments as to
why it is a poor definition but mere assertions are not going to make
it a good definition.

[ron minnich]

Ah. But what if we discover that Steven's definition has been the
commonly understood definition in the field for almost a half century?
Would it be acceptable to you then?

ron

[Mike Samuel]

2011/10/9 ron minnich <rmin...@gmail.com>:

If the poor schlubs who send emails titled "please help me find the
memory leak in my code?" are included "in the field", then yes.

[Mike Samuel]

On Oct 9, 8:51 pm, Mike Samuel <mikesam...@gmail.com> wrote:
> > Ah. But what if we discover that Steven's definition has been the
> > commonly understood definition in the field for almost a half century?
> > Would it be acceptable to you then?
>
> If the poor schlubs who send emails titled "please help me find the
> memory leak in my code?" are included "in the field", then yes.

Sorry, this was unnecessarily flip.
I acknowledge that specialists need to have precise definitions and
that non-specialists like me should make an effort to learn the
terminology and use it correctly. I have no right to foist my
definitions on a group of specialists, and I should appreciate it when
specialists take the time to clarify.

My concern is that "system X does not suffer Y" is likely to be
inspire misplaced confidence when non-specialists have a different
understanding of the term Y than the specialists who give the advice.

Apologies all round for derailing the thread.

[Vasiliy Tolstov]

Thanks for all suggestions. I'm try to use memprofile and solve some
issues. ALso i simplify some code parts. Buw memory leaks are small,
but not dissapeared.
Can somebody helps me to find location in code that eats memory...?
After some tests, i get , that this function eats after some time
memory (code.google.com/p/wiff/):
func config(conf Config, _data chan Data) {
var err os.Error
var r *http.Response
var line string
var match []string
var data Data
var pattern *regexp.Regexp =
regexp.MustCompile("([a-zA-Z0-9.]+)=\"?([^\"]+)\"?")
var rb *bufio.Reader
var lerr os.Error
var cmd *exec.Cmd
var val url.Values
var mem Xmem
var disk Xdisk

val = make(url.Values)
for {
r, err = http.Get(conf.Uri)
if err != nil || r.StatusCode != 200 {
time.Sleep(conf.Pullint * 1000000000)
continue
}

lerr = nil
rb = bufio.NewReader(r.Body)
for lerr == nil {
line, lerr = rb.ReadString('\n')
match = pattern.FindStringSubmatch(line)
if lerr != os.EOF && lerr != nil {
break
}
if lerr == os.EOF {
break
}
if match == nil {
break
}
match[1] = strings.Trim(match[1], "\n\t\r")
match[2] = strings.Trim(match[2], "\n\t\r")
switch match[1] {
case "userpw":
data.Userpw = match[2]
break
case "rootpw":
data.Rootpw = match[2]
break
case "memmin":
data.Memmin, _ = strconv.Atoui64(match[2])
data.Memmin = data.Memmin * 1024 * 1024
break
case "memmax":
data.Memmax, _ = strconv.Atoui64(match[2])
data.Memmax = data.Memmax * 1024 * 1024
break
case "memint":
data.Memint, _ = strconv.Atoi64(match[2])
if data.Memint == 0 || data.Memint == 1000 {
data.Memint = 1
}
break
case "memhold":
data.Memhold, _ = strconv.Atoui64(match[2])
data.Memhold = data.Memhold * 1024 * 1024
break
case "memblk":
data.Memblk, _ = strconv.Atoui64(match[2])
data.Memblk = data.Memblk * 1024 * 1024
break
case "pullint":
conf.Pullint, _ = strconv.Atoi64(match[2])
break
case "cmdline":
if data.Cmdline != match[2] && match[2] != "" {
data.Cmdline = match[2]
cmd = exec.Command("/bin/sh", "-c", data.Cmdline)
cmd.Start()
}
break
}
}
match = nil
rb = nil
line = ""
if data.Memmax < data.Memmin || data.Memmax == 0 {
data.Memmax = data.Memmin
}

err = Memory(&mem)
if err == nil {
val.Set("mem_used", fmt.Sprintf("%v", mem.mem_used))
}
err = Disk(&disk)
if err == nil {
val.Set("disk_used", fmt.Sprintf("%v", disk.disk_used))
}
r, err = http.PostForm(conf.Uri, val)
if err == nil {
r.Body.Close()
} else {
fmt.Printf(err.String())
}
val.Del("mem_used")
val.Del("disk_used")

r.Body.Close()
_data <- data
if tr, ok := http.DefaultTransport.(*http.Transport); ok {
tr.CloseIdleConnections()
}
time.Sleep(conf.Pullint * 1000000000)
}
return

[⚛]

On Oct 10, 1:51 am, Mike Samuel <mikesam...@gmail.com> wrote:
> Memory leaks can creep in wherever the GC's definition of garbage
> (based on reachability) differs from the programmer's definition of
> garbage based on utility and accessibility via APIs.
>

> [Copied from elsewhere]:

> I acknowledge that specialists need to have precise definitions and
> that non-specialists like me should make an effort to learn the
> terminology and use it correctly. I have no right to foist my
> definitions on a group of specialists, and I should appreciate it when
> specialists take the time to clarify.

I sort-of agree with your viewpoint that "Memory leaks can creep in

wherever the GC's definition of garbage (based on reachability)
differs from the programmer's definition of garbage based on utility
and accessibility via APIs".

However, the issue is that under your definition of garbage, GC
becomes an intractable problem. In other words, it is impossible to
implement the garbage collector. This is because you are allowing
*arbitrary computation* to decide what garbage is and what garbage
isn't. You can *not* implement such a garbage collector - not because
you (or me) don't like it, but because it is impossible. You just need
to accept the fact that it indeed is impossible.

The definition of GC has to draw "a line" somewhere. This line itself
gives meaning to the term "garbage". The usual definition draws this
line at memory. This is easy-enough for most programmers to
understand, and because of this easy-enoughness it is the right
definition from a practical viewpoint. A GC algorithm deals (by
definition) only with the contents of memory which are existing at a
particular moment in time - dealing with computation is (on purpose)
left out of the picture.

You wrote that "I have no right to foist my definitions [of GC] on a
group of specialists". The truth is that the definition you presented
in this discussion is *not* a definition. It isn't.

Of course, if you want to individually change the definition of what
GC is and what isn't (to shift the line somewhere elsewhere), you
*can* do that. But you have to do it in a precise way.

[Lars Pensjö]

I did a similar thing, but memory is a little vague. I would recommend to use the tool "valgrind". It may be available only in Linux?

That tool will show you exactly what lines in the source code that allocates memory.

[Vasiliy Tolstov]

2011/10/10 Lars Pensjö <lars....@gmail.com>:

valgrind dies in my case after try to run my binary

[John Asmuth]

On Monday, October 10, 2011 5:18:26 AM UTC-4, ⚛ wrote:This is because you are allowing 

*arbitrary computation* to decide what garbage is and what garbage
isn't. You can *not* implement such a garbage collector - not because
you (or me) don't like it, but because it is impossible. You just need
to accept the fact that it indeed is impossible.

Just to emphasize this point - he doesn't mean "really hard" or "intractable". Impossible is the correct word. Such a garbage collector would solve the halting problem.

[Vasiliy Tolstov]

2011/10/10 Vasiliy Tolstov <v.to...@selfip.ru>:

> 2011/10/10 Lars Pensjö <lars....@gmail.com>:
>> I did a similar thing, but memory is a little vague. I would recommend to
>> use the tool "valgrind". It may be available only in Linux?
>> That tool will show you exactly what lines in the source code that allocates
>> memory.
>
> valgrind dies in my case after try to run my binary
>
> --

i'm doing some profiling and get:
Total: 768.0 MB
586.5 76.4% 76.4% 586.5 76.4% runtime.malg
85.5 11.1% 87.5% 110.0 14.3% runtime/pprof.WriteHeapProfile
20.5 2.7% 90.2% 20.5 2.7% bufio.NewReaderSize
14.5 1.9% 92.1% 25.0 3.3% regexp.(*Regexp).FindStringSubmatch
12.5 1.6% 93.7% 12.5 1.6% bufio.NewWriterSize
10.5 1.4% 95.1% 10.5 1.4% io.Copy
7.0 0.9% 96.0% 7.0 0.9% bytes.(*Buffer).grow
7.0 0.9% 96.9% 7.0 0.9% regexp/syntax.(*compiler).inst
6.5 0.8% 97.7% 6.5 0.8% regexp.progMachine
5.0 0.7% 98.4% 5.0 0.7% runtime.convT2E

Is that possible to minimize memory usage ?
P.S. I'm add some bad code lines and memory does not leak now.

[Jonathan Amsterdam]

You are all getting in a tizzy over nothing.

"Memory leak" has no precise definition. "Polynomial time." "Turing-
complete." Now those are terms with precise definitions. "Memory leak"
is a term of art -- everyone pretty much uses it the same way in the
clear cases, but it's fuzzy around the edges.

Now to garbage collection people, "memory leak" does have a precise
meaning. OF COURSE it does to them, otherwise they couldn't claim that
a GC'd runtime has no memory leaks, for the reason mentioned -- it's
undecidable to figure out what memory the program WON'T touch; the
best you can do is figure out what it CAN'T touch. GC people also call
the user's program the "mutator," but no one's suggesting that
everyone else do so. They use words in a way that makes sense for
their field.

But that doesn't help the working programmer, who observes his
program's memory consumption grow without bound. Working programmers
call that a memory leak. OF COURSE they do. Their program is running
out of memory! They care whether that is due to a bug in the
implementation (e.g. conservative GC) or a bug in their code, but they
don't care what labels you attach to those two alternatives.

Moral: stop having arguments about things about which there is no fact
of the matter, and help the working programmers with their real
problems.

[⚛]

On Oct 10, 5:14 pm, Vasiliy Tolstov <v.tols...@selfip.ru> wrote:
> 2011/10/10 Vasiliy Tolstov <v.tols...@selfip.ru>:
>
> > 2011/10/10 Lars Pensjö <lars.pen...@gmail.com>:

> >> I did a similar thing, but memory is a little vague. I would recommend to
> >> use the tool "valgrind". It may be available only in Linux?
> >> That tool will show you exactly what lines in the source code that allocates
> >> memory.
>
> > valgrind dies in my case after try to run my binary
>
> > --
>
> i'm doing some profiling and get:
> Total: 768.0 MB
>    586.5  76.4%  76.4%    586.5  76.4% runtime.malg
>     85.5  11.1%  87.5%    110.0  14.3% runtime/pprof.WriteHeapProfile
>     20.5   2.7%  90.2%     20.5   2.7% bufio.NewReaderSize
>     14.5   1.9%  92.1%     25.0   3.3% regexp.(*Regexp).FindStringSubmatch
>     12.5   1.6%  93.7%     12.5   1.6% bufio.NewWriterSize
>     10.5   1.4%  95.1%     10.5   1.4% io.Copy
>      7.0   0.9%  96.0%      7.0   0.9% bytes.(*Buffer).grow
>      7.0   0.9%  96.9%      7.0   0.9% regexp/syntax.(*compiler).inst
>      6.5   0.8%  97.7%      6.5   0.8% regexp.progMachine
>      5.0   0.7%  98.4%      5.0   0.7% runtime.convT2E
>
> Is that possible to minimize memory usage ?
> P.S. I'm add some bad code lines and memory does not leak now.

Can you submit those "bad code lines"?

[Dave Cheney]

Hi,

Are you using 8g or 6g ? The former does have some known problems,
which Dmitry is looking at solving.

Cheers

Dave

[Vasiliy Tolstov]

2011/10/10 Dave Cheney <da...@cheney.net>

Hi,

Are you using 8g or 6g ? The former does have some known problems,
which Dmitry is looking at solving.

Cheers

I'm use 6g

 

[Mike Samuel]

On Oct 10, 7:52 am, John Asmuth <jasm...@gmail.com> wrote:
> Just to emphasize this point - he doesn't mean "really hard" or
> "intractable". Impossible is the correct word. Such a garbage collector
> would solve the halting problem.

To clarify, I was not claiming that language implementors should try
to produce languages without memory-leaks according to this
definition.

I was asserting that my definition is what naive developers think of
when they hear "programs written in language X cannot leak memory."

[Dmitry Vyukov]

On Mon, Oct 10, 2011 at 11:53 PM, Dave Cheney <da...@cheney.net> wrote:

Hi,

Are you using 8g or 6g ? The former does have some known problems,
which Dmitry is looking at solving.

I do not actively looking at it now:

https://groups.google.com/d/topic/golang-dev/UQldoCAfK14/discussion

[Dave Cheney]

Sorry, my mistake, my information was out of date, and not relevant as
the OP is using 6g.

[Vasiliy Tolstov]

I'm found that this code, if that it runs from for loop (many times), grow rss memory size:

package main

type Xmem map[string]uint64

func main() {

  for ; ; {

  mem := make(Xmem)

  Memory(mem)

  }

}

func Memory(mem Xmem) os.Error {

  var pattern *regexp.Regexp = regexp.MustCompile("([a-zA-Z0-9_]+): *([0-9]+) kB")

  var match []string

  var value uint64

  var line string

  var lerr os.Error

  var r *bufio.Reader

  lerr = nil

  f, err := os.Open("/proc/meminfo")

  if err != nil {

    return err

  }

  defer f.Close()

  r = bufio.NewReader(f)

  for lerr == nil {

    line, lerr = r.ReadString('\n')

    if lerr != os.EOF && lerr != nil {

      return err

    }

    if lerr == os.EOF {

      break

    }

    match = pattern.FindStringSubmatch(line)

    if match != nil {

      value, err = strconv.Atoui64(match[2])

      if err != nil {

        value = 0

      }

      switch match[1] {

      case "MemTotal":

        mem["total"] = value * 1024

        break

      case "MemFree":

        mem["free"] = value * 1024

        break

      case "Buffers":

        mem["buffers"] = value * 1024

        break

      case "Cached":

        mem["cached"] = value * 1024

        break

      }

    }

  }

  line = ""

  r = nil

  match = nil

  pattern = nil

  mem["free"] = mem["free"] + mem["buffers"] + mem["cached"]

  mem["used"] = mem["total"] - mem["free"]

  return nil

}

What i need to do, to not rgowing memory usage?

[⚛]

Unfortunately, I am unable to reproduce the memory leak on my machine
(32-bit x86).

On Oct 12, 11:54 am, Vasiliy Tolstov <v.tols...@selfip.ru> wrote:
> I'm found that this code, if that it runs from for loop (many times), grow
> rss memory size:
>
> package main

> [cut]

>
> What i need to do, to not rgowing memory usage?

There are multiple options how to detect which part is causing the
memory leak (and to optimize the code):

- move the "regexp.MustCompile" to file-scope

- replace "map[string]uint64" with "[]uint64". Use predefined integer
constants instead of textual map keys, for example:

const (
MEM_TOTAL = iota
MEM_FREE
)
type Xmem []uint64
...
func Memory(mem Xmem) os.Error {
...
mem[MEM_TOTAL] = value * 1024
...
}

- try "strings.NewReader(predefinedString)" instead of "os.Open" and
"bufio.NewReader", and see what happens

[Vasiliy Tolstov]

Thanks for answer.

2011/10/12 ⚛ <0xe2.0x...@gmail.com>

[tux21b]

Russ has fixed a map related memory leak recently. If I remember correctly, the last element wasn't freed or something like that. Anyway, this fix is already included in r60.2 so, make sure to use a recent version of Go.

[Kyle Lemons]

Russ has fixed a map related memory leak recently. If I remember correctly, the last element wasn't freed or something like that. Anyway, this fix is already included in r60.2 so, make sure to use a recent version of Go.

The issue was that the map included a pointer one past the end of the map, and so it would pin in memory whatever object was placed right after the map in the heap.