Memory use problem on FreeBSD

[John Graham-Cumming]

I have a long running Go program that allocates a large amount of
memory over time. On FreeBSD 64 bit I am seeing a problem where once I
am done with the memory the RES size as reported by top never
decreases.

For example, after my program has finished processing requests I stop
all requests and let CPU drop to 0% and then leave it so that GC and
scavenging has time to work. Then I look at the RES size in top and
dump out a memory profile for pprof. In one example, I have RES of
15666M (i.e. 15GB) and pprof showing me:

(pprof) top10
Total: 32.7 MB

This is using Go version: devel +40ba4d4e4672 Tue Nov 13 10:45:30 2012
-0800

Are there any known problem with Go not returning memory to the OS on
that version (or the bleeding edge in general)? I do not see memory
problems like this on Linux. Although the nature of my test
environment means I can't do precisely the same test on Linux and
FreeBSD because FreeBSD as it a client site.

John.

[John Graham-Cumming]

On Nov 19, 9:20 pm, John Graham-Cumming <jgrah...@gmail.com> wrote:
> I have a long running Go program that allocates a large amount of
> memory over time. On FreeBSD 64 bit I am seeing a problem where once I
> am done with the memory the RES size as reported by top never
> decreases.

Hmm. I think I might have found the answer to my question in
mem_freebsd.c:

void
runtime·SysUnused(void *v, uintptr n)
{
USED(v);
USED(n);
// TODO(rsc): call madvise MADV_DONTNEED
}

It appears that it does not tell the OS that the memory is no longer
needed. Am I correct?

John.

[Dave Cheney]

Hi John,

I was just chasing that down. You are correct that implementing this
method should activate the scavenger. I'll prepare a CL.

Cheers

Dave

> --
>
>

[John Graham-Cumming]

On Nov 19, 11:47 pm, Dave Cheney <d...@cheney.net> wrote:
> I was just chasing that down. You are correct that implementing this
> method should activate the scavenger. I'll prepare a CL.

It looks like for xBSD (where x is Free, Net and Open) runtime.madvise
is not implemented at all.

John.

[John Graham-Cumming]

I made an experimental patch to Go on FreeBSD amd64 by adding the following to sys_freebsd_amd64.s:

TEXT runtime?madvise(SB),7,$0                                                                                                            

        MOVQ    8(SP), DI                                                                                                                

        MOVQ    16(SP), SI                                                                                                               

        MOVQ    24(SP), DX                                                                                                               

        MOVQ    $75, AX // madvise                                                                                                       

        SYSCALL                                                                                                                          

        CMPQ    AX, $0xfffffffffffff001                                                                                                  

        JLS     2(PC)                                                                                                                    

        MOVL    $0xf1, 0xf1  // crash                                                                                                    

        RET                          

And changing mem_freebsd.c so that it actually gets called. That worked.

John.

[Dave Cheney]

Yup, that is all that is needed. Do you want to propose a CL ?

Cheers

Dave

> --
>
>

[John Graham-Cumming]

On Tuesday, November 20, 2012 3:46:37 AM UTC, Dave Cheney wrote:

Yup, that is all that is needed. Do you want to propose a CL ?

What's needed is a CL for all the BSDs and both 32 and 64 bit. I don't have all those systems to test against. 

John.

 

[Devon H. O'Dell]

2012/11/19 John Graham-Cumming <jgra...@gmail.com>:

Just for reference (in case this isn't what you've done) I'm pretty
sure that MADV_FREE is what we want on FreeBSD. (I don't know about
OpenBSD or NetBSD). MADV_DONTNEED doesn't mean the same thing on Linux
and FreeBSD.

--dho

> John.
>
>
> --
>
>

[Dave Cheney]

I'd start with FreeBSD, then do the others in a later CL, generally
there is someone on the list who has hardware available to test.

> --
>
>

[Dave Cheney]

Yup, that matches my research,

see: https://codereview.appspot.com/6856066/

> --
>
>

[Steven Hartland]

It may be unrelated but there was a kernel memory leak in fadvise which was fixed by:-

http://svnweb.freebsd.org/base?view=revision&revision=234661

 

    Regards

    Steve

--
 
 

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postm...@multiplay.co.uk.

[John Graham-Cumming]

On Tuesday, November 20, 2012 3:51:47 AM UTC, Dave Cheney wrote:

Yup, that matches my research,

see: https://codereview.appspot.com/6856066/

Are you planning a CL to fix this problem?  My experiments with FreeBSD indicate that MADV_FREE is the way to go, but I am not enough of an expert on the BSD family to do a CL and feel confident that I have fully understood the solution.

John.

 

[Devon H. O'Dell]

2012/11/20 John Graham-Cumming <jgra...@gmail.com>:

I think Dave was going to let you do it. We can review and make
suggestions if you'd like to continue, otherwise I'm sure Dave can
continue?

--dho

> John.
>

[Dave Cheney]

Yup. That was the plan, but if you would rather have me do it, please let me know. 

--

[John Graham-Cumming]

On Tuesday, November 20, 2012 8:08:10 PM UTC, Dave Cheney wrote:

Yup. That was the plan, but if you would rather have me do it, please let me know.

I'll prepare a patch and submit for FreeBSD amd64 which I have tested. Others can expand to the other platforms.

John.

 

[Dave Cheney]

SGTM.

--
 
 

[John Graham-Cumming]

Done. Patch is here: http://codereview.appspot.com/6850081

John.

[Devon H. O'Dell]

FWIW i386 should basically just be the same code; the assembler should
be simple enough to crib from elsewhere.

--dho

2012/11/20 Dave Cheney <da...@cheney.net>:

[John Graham-Cumming]

On Tuesday, November 20, 2012 9:20:06 PM UTC, Devon H. O'Dell wrote:

FWIW i386 should basically just be the same code; the assembler should
be simple enough to crib from elsewhere.

Agreed. Similarly for NetBSD and OpenBSD. I'm sure they are all very similar. I just wasn't comfortable submitting code that I had not tested.

John.