Why golang allocated memory space increases?

[steve tang]

chunkCrc32Hash := crc32.NewIEEE()

chunkBuf := bufferpool.GetInstance().Get()

//回收buffer

defer bufferpool.GetInstance().Put(chunkBuf)

writer := io.MultiWriter(chunkBuf, chunkCrc32Hash)

_, copyErr := io.Copy(writer, chunkResp.RawResponse.Body)

if copyErr != nil && copyErr != io.EOF {

   logmgr.ErrorLogger(ctx, "fail to write chunk data to crc32 or buffer, req: %s, err: %s", originReq.PrettyPrint(), copyErr)

   return false, exception.OEFException(nil, http.StatusInternalServerError, CopyChunkError, "internal server exception")

}

In above codes,  it writes chunk data to buffer pool, and the chunk data size is 4MB.  When I used golang tool pprof to export memory pprof, there about 8MB space has been allocated.  Could somebody help me why golang allocated memory space increases?

[robert engels]

If the pool is a sync.Pool:

Any item stored in the Pool may be removed automatically at any time without
    18  // notification. If the Pool holds the only reference when this happens, the
    19  // item might be deallocated.

So placing an object in the pool does not guarantee it won’t be collected - causing a future allocation.

<20200326-111844.png>

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/6dbe93e6-9bb7-4112-87cb-085e68d9d393%40googlegroups.com.
<20200326-111844.png>

[steve tang]

Thanks,  I used sync.Pool to allocate a byte.Buffer,  and response.Body  isn't constant size in program, I find golang will grow the byte.Buffer space against to initial space. I changed code as follows:

type SyncPool struct{ Pool sync.Pool }

func (s *SyncPool) Get(n int) []byte {
   //比n大，不需要slice增长时，使用buffer pool
   if b, ok := s.Pool.Get().([]byte); ok && cap(b) >= n {
      return b[:n]
   } else {
      //miss size
      s.Put(b)
      return make([]byte, n)
   }
}

func (s *SyncPool) Put(b []byte) {
   s.Pool.Put(b)
}

const FourMB = 4 * 1024 * 1024

var pool = &SyncPool{
   sync.Pool{
      New: func() interface{} {
         return make([]byte, FourMB)
      },
   },
}

On Thursday, March 26, 2020 at 12:22:01 PM UTC+8, robert engels wrote:

If the pool is a sync.Pool:

Any item stored in the Pool may be removed automatically at any time without
    18  // notification. If the Pool holds the only reference when this happens, the
    19  // item might be deallocated.

So placing an object in the pool does not guarantee it won’t be collected - causing a future allocation.

On Mar 25, 2020, at 10:22 PM, steve tang <tsg...@gmail.com> wrote:

chunkCrc32Hash := crc32.NewIEEE()

chunkBuf := bufferpool.GetInstance().Get()

//回收buffer

defer bufferpool.GetInstance().Put(chunkBuf)

writer := io.MultiWriter(chunkBuf, chunkCrc32Hash)

_, copyErr := io.Copy(writer, chunkResp.RawResponse.Body)

if copyErr != nil && copyErr != io.EOF {

   logmgr.ErrorLogger(ctx, "fail to write chunk data to crc32 or buffer, req: %s, err: %s", originReq.PrettyPrint(), copyErr)

   return false, exception.OEFException(nil, http.StatusInternalServerError, CopyChunkError, "internal server exception")

}

In above codes,  it writes chunk data to buffer pool, and the chunk data size is 4MB.  When I used golang tool pprof to export memory pprof, there about 8MB space has been allocated.  Could somebody help me why golang allocated memory space increases?

<20200326-111844.png>

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

[Tamás Gulácsi]

1. sync.Pool MUST use pointers (*[]byte)

2. what is `chunkBuf := bufferpool.GetInstance().Get()` ? it must be an io.Writer, but you should Reset/Truncate it when you Put it back to the pool!

[Jakob Borg]

On 26 Mar 2020, at 09:51, Tamás Gulácsi <tgula...@gmail.com> wrote:

sync.Pool MUST use pointers (*[]byte)

I've seen this a lot but I confess I don't understand it. A []byte is essentially a fat pointer, what does it matter if we put that or a *[]byte into the pool? I understand that putting a []byte will result in an allocation of a new []byte, but probably the purpose is to preserve and avoid reallocating the backing array, which might be large.

Is there something fundamental about the sync.Pool that means that a put *must* be zero-allocation for it to function? Likely the calling code around the puts/gets are not zero-allocation either.

//jb

[Henrik Johansson]

Isn't this because of the GC tracking these and treating then as effectively weak references to borrow a Java term?

If they are not pointers they are not tracked by the GC and I guess they could all be removed at every scan?

Just guessing though, I haven't in any way checked it.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/DB9E8E73-10B0-451B-BE66-13AF2930BB74%40kastelo.net.