Memory usage

[Gareth Owenson]

Hi all

We have a go application that processes large numbers of documents.  It behaves like it has a memory leak, eventually exhausting system ram after a few hours.  I've setup pprof and looked at the various outputs, and that is adamant that the application is only using some 50MB of ram, yet Linux top reports it using much more (and growing).

I've tried setting madvdontneed to no avail.

Any help on better ways to identify the cause would be greatly appreciated.

best

Gareth

[Kurtis Rader]

If you're using cgo to link with C/C++ code then a memory leak in that API is the first place to start looking..

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/409a950d-b270-4ef7-a965-ea3d450e74f5n%40googlegroups.com.

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Miha Vrhovnik]

And if you are using a cgo and didn't replace the standard malloc with jemaloc you are probably suffering from memory fragmentation or whatever the hell happens.

We had to restart the processors every 1000 iterations because of that. There was no memory leak according to valgrind. (about 300kb), yet the RSS was constantly rising. Calling exactly the same C api with a C shim the RSS was constant about 100M. And api was extremely simplified all large memory allocations happened on the C side. pprof showed small amount of used memory yet the RSS was in gigabytes.

BR;

Miha

[Gareth Owenson]

Thanks both, immensely helpful in narrowing the problem.

You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/lGcsCbY282I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0707e376-d6d7-40ff-9803-e5534c98b62dn%40googlegroups.com.