Memory Leak in bytes.Buffer String() method

func replacePositionalPlaceholders(sql, prefix string) (string, error) {
buf := &strings.Builder{}
defer buf.Reset()

i := 0
for {
p := strings.Index(sql, "?")
if p == -1 {
break
}

if len(sql[p:]) > 1 && sql[p:p+2] == "??" { // escape ?? => ?
buf.WriteString(sql[:p])
buf.WriteString("?")
if len(sql[p:]) == 1 {
break
}
sql = sql[p+2:]
} else {
i++
buf.WriteString(sql[:p])
fmt.Fprintf(buf, "%s%d", prefix, i)
sql = sql[p+1:]
}
}

buf.WriteString(sql)
return buf.String(), nil
}

======

This didn't help either, from the results below

(pprof) top20

Showing nodes accounting for 220.09MB, 90.37% of 243.55MB total

Dropped 126 nodes (cum <= 1.22MB)

Showing top 20 nodes out of 135

flat flat% sum% cum cum%

33.97MB 13.95% 13.95% 33.97MB 13.95% strings.(*Builder).Write

29.07MB 11.94% 25.89% 29.57MB 12.14% github.com/newrelic/go-agent/v3/newrelic.(*txnTrace).witnessNode

28.01MB 11.50% 37.39% 28.01MB 11.50% github.com/newrelic/go-agent/v3/newrelic.(*spanAttributeMap).add

24.34MB 9.99% 47.38% 24.34MB 9.99% strings.(*Builder).WriteString

18.52MB 7.60% 54.98% 18.52MB 7.60% go.uber.org/zap/buffer.NewPool.func1

11.01MB 4.52% 59.50% 60.59MB 24.88% github.com/newrelic/go-agent/v3/newrelic.endExternalSegment

10.01MB 4.11% 63.61% 13.51MB 5.55% github.com/newrelic/go-agent/v3/newrelic.newTxn

10MB 4.11% 67.72% 10MB 4.11% github.com/newrelic/go-agent/v3/newrelic.segmentEnd.spanEvent

7MB 2.88% 73.73% 7MB 2.88% github.com/newrelic/go-agent/v3/newrelic.(*slowQueries).insertAtIndex

7MB 2.88% 76.60% 50.02MB 20.54% github.com/newrelic/go-agent/v3/newrelic.endDatastoreSegment

6.51MB 2.67% 79.28% 6.51MB 2.67% github.com/DataDog/datadog-go/statsd.newStatsdBuffer

6MB 2.46% 81.74% 6.50MB 2.67% github.com/newrelic/go-agent/v3/newrelic.addUserAttribute

5.50MB 2.26% 84.00% 5.50MB 2.26% github.com/newrelic/go-agent/v3/newrelic.getStackTrace

3MB 1.23% 85.23% 3MB 1.23% github.com/newrelic/go-agent/v3/newrelic.newSlowQueries

3MB 1.23% 86.47% 3MB 1.23% go.uber.org/zap.(*Logger).clone

3MB 1.23% 87.70% 3MB 1.23% github.com/newrelic/go-agent/v3/newrelic.externalMetricKey.scopedMetric

2.50MB 1.03% 88.72% 30.51MB 12.53% github.com/newrelic/go-agent/v3/newrelic.(*spanAttributeMap).addString

2MB 0.82% 90.37% 2MB 0.82% encoding/hex.EncodeToString

(pprof) peek Write

Showing nodes accounting for 243.55MB, 100% of 243.55MB total

----------------------------------------------------------+-------------

flat flat% sum% cum cum% calls calls% + context

----------------------------------------------------------+-------------

33.97MB 100% | fmt.Fprintf

33.97MB 13.95% 13.95% 33.97MB 13.95% | strings.(*Builder).Write

----------------------------------------------------------+-------------

22.34MB 91.78% | github.com/XXXXX/squirrel.replacePositionalPlaceholders

2MB 8.22% | net/url.(*URL).String

24.34MB 9.99% 23.94% 24.34MB 9.99% | strings.(*Builder).WriteString

----------------------------------------------------------+-------------

1.16MB 100% | encoding/json.compact

0 0% 23.94% 1.16MB 0.48% | bytes.(*Buffer).Write

1.16MB 100% | bytes.(*Buffer).grow

----------------------------------------------------------+-------------

6.51MB 100% | github.com/DataDog/datadog-go/statsd.New

0 0% 23.94% 6.51MB 2.67% | github.com/DataDog/datadog-go/statsd.newWithWriter

6.51MB 100% | github.com/DataDog/datadog-go/statsd.newBufferPool

----------------------------------------------------------+-------------

12.01MB 100% | go.uber.org/zap.(*SugaredLogger).log

0 0% 23.94% 12.01MB 4.93% | go.uber.org/zap/zapcore.(*CheckedEntry).Write

12.01MB 100% | go.uber.org/zap/zapcore.(*ioCore).Write

----------------------------------------------------------+-------------

12.01MB 100% | go.uber.org/zap/zapcore.(*CheckedEntry).Write

0 0% 23.94% 12.01MB 4.93% | go.uber.org/zap/zapcore.(*ioCore).Write

12.01MB 100% | go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry

----------------------------------------------------------+-------------

1MB 100% | golang.org/x/net/http2.(*Framer).readMetaFrame

0 0% 23.94% 1MB 0.41% | golang.org/x/net/http2/hpack.(*Decoder).Write

1MB 100% | golang.org/x/net/http2/hpack.(*Decoder).parseHeaderFieldRepr

----------------------------------------------------------+-------------

(pprof)

Please help how to debug this more

Regards

Sarath Prabath Redlapalli Jaya

unread,

Aug 30, 2020, 11:27:33 AM8/30/20

to golang-nuts

The memory usage of containers are ever increasing and has grown from 100 MB to ~ 1 GB in couple of hours. Not able to understand where the gap is, even though the the builder is being reset. Would keeping a sync.Pool of builders help here?

Regards

Ian Lance Taylor

unread,

Aug 30, 2020, 11:55:05 AM8/30/20

to Sarath Prabath Redlapalli Jaya, golang-nuts

On Sun, Aug 30, 2020 at 8:25 AM Sarath Prabath Redlapalli Jaya
<sar...@okcredit.in> wrote:
>
> Hi, we've been observing high memory usage in the following code
> https://github.com/Masterminds/squirrel/blob/master/placeholder.go#L113
>
> We measured from pprof of heap
>
> (pprof) top20
>
> Showing nodes accounting for 360.07MB, 91.31% of 394.32MB total
>
> Dropped 142 nodes (cum <= 1.97MB)
>
> Showing top 20 nodes out of 139
>
> flat flat% sum% cum cum%
>
> 101.21MB 25.67% 25.67% 101.21MB 25.67% bytes.(*Buffer).String

This tells you where the memory was allocated. If you are trying to
track down a memory leak, you need to look into what is keeping this
memory live in the heap. The fact that it was allocated here is a
start. But your next step is to look at what calls to this function
are allocating memory, and what is happening to that memory to keep it
alive. The pprof program can help tell you where the calls are coming
from, but then you'll have to look at what happens to those strings
and why they are being kept live.

Ian

Ian Lance Taylor

unread,

Aug 30, 2020, 11:55:35 AM8/30/20

to Sarath Prabath Redlapalli Jaya, golang-nuts

On Sun, Aug 30, 2020 at 8:28 AM Sarath Prabath Redlapalli Jaya
<sar...@okcredit.in> wrote:
>
> The memory usage of containers are ever increasing and has grown from 100 MB to ~ 1 GB in couple of hours. Not able to understand where the gap is, even though the the builder is being reset. Would keeping a sync.Pool of builders help here?

No, because the problem is not the builders. It's what is happening
to the strings after they have been built.

Ian

Jake Montgomery

unread,

Aug 30, 2020, 11:58:44 AM8/30/20

to golang-nuts

Sorry if this is a bit obvious. I did not dive into your code. But I did notice that replacePositionalPlaceholders returns buf.String(). If you are certain that this is where the memory 'leak' is originating, then the most likely scenario is that the returned string is never being collected. Is it possible that somehow there is a reference being kept to the string, or an object containing the string?

Reply all

Reply to author

Forward