Heap fragmentation due to protobuffs serialization/de-serialization

[Devanand Reddy]

Hi there,

We are using protocol buffers extensively in our application. We have a suspect that over the time, due to excessive  marshaling/un-marshaling these two operations are causing heap fragmentation. One factor I am looking at, to come to this conclusion is HeapIdle. In my tests, though HeapInUse is coming back to my expectation, HeapIdle is staying there forever. On another note, our protobuff operations consume any where between 700-900mb, given the size of the data we are trying to serialize. 

I am attaching a few graphs here.

Please advise whether this is a sign for concern.

Thanks.

[Ian Lance Taylor]

Heap fragmentation is not normally a concern with the current Go
runtime. The heap is implemented to that large pages are divided up
into blocks that are all the same size. This means that the usual
definition of fragmentation--a small block prevents coalescing of
large blocks--can not occur.

Your graphs seem to show your memory use is in a steady state, which
is a good thing. When there are operations that temporarily use a lot
of memory, it's normal for HeapIdle to be high--the memory is there
ready and waiting for temporary memory use spikes.

Ian

[Dave Cheney]

It looks ok to me, the svg.released number is growing which suggests that fragmentation is not an issue.