Deallocate memory consumption of slice.

[Gowtham Raj]

I have a sample go snippet which pulls 50k rows from source DB and inserts into target DB. https://go.dev/play/p/IVANZyVUbkm

While doing so I create a empty slice and add operations to it. When the length of slice is > 10K, I do a bulk write to DB.

The problem here is this script eats more memory as it runs, I suspect issue is with the slice which gets appended periodically. I try to do a copy as suggested in few articles and reset it but it never gives up on the memory and the script consumes more and more as it runs.

Any idea as to who to release the memory of slice every 10k docs ?

I have reset the slice, but memory is not returned back.

[Marvin Renich]

* Gowtham Raj <rgowt...@gmail.com> [240826 11:31]:

Copying the slice to modelsCopy does nothing but allocate more memory,
which will be freed when the garbage collector gets around to it. Take
out this code.

The statement

models = models[:0]

does not free the memory allocated for the slice's backing array, but it
does cause future appends to reuse slice elements from the beginning of
the array. What you probably want is

models = []mongo.WriteModel{}

which will mark the old backing array for collection and create a new
slice.

...Marvin

[robert engels]

But why would you want to allocate a new array? This would only be required if the MongoDB call were async, and the array cannot be reused.

If you keep allocating new arrays you are putting needless pressure on the GC.

-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/Zsyqz8zgxvf8cjYE%40basil.wdw.

[Gowtham Raj]

I had models = []mongo.WriteModel{} before , then tried with models = nil, and models[:0]. All same result.

For  []mongo.WriteModel{} , I expected that memory gets freed up, but it does not. Everytime the append runs to mongo.WriteModel{} memory consumption increases.

My doubt is if the models object created as global and not within the for loop is causing any issue -?

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/Vw04RLUpRcc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Marvin Renich]

* robert engels <ren...@ix.netcom.com> [240826 12:24]:

> But why would you want to allocate a new array? This would only be required if the MongoDB call were async, and the array cannot be reused.
>
> If you keep allocating new arrays you are putting needless pressure on the GC.
>

True. But I was going on the assumption that the amount of memory
allocated for the 10001 interface values, each holding 50000 records,
swamped the memory used by the slice's backing array. Marking all of
the interface values for collection before going back for another time
through the loop was my goal.

Since mongo.WriteModel is an interface, the memory for each element of
model (the interface value) is kept until that element is overwritten,
even when you have done «model = model[:0]».

The OP can mark each of those elements to be garbage collected and at
the same time avoid reallocating the slice's backing array by assigning
nil to each element, and then reslicing model:

for i := range(len(model)) {
model[i] = nil
}
model = model[:0]

...Marvin

[robert engels]

I looked at the mongo db driver code - it allocates array storage as least as large as the supplied models.

I am guessing you are creating garbage faster than the GC, and the pacing thinks it has plenty of memory.

I would use debug.SetMemoryLimit() to set the desired limit and see what happens.

https://pkg.go.dev/runtime/debug#SetMemoryLimit

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAE-6Q_LOLc1is5ybZbW3D2jVvnnc4QerQZt4aWXz2RMeqOHZow%40mail.gmail.com.

[robert engels]

true, but I don’t think that would lead to continually increasing memory - after the first iteration all entries are being overwritten anyway.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/Zsy1BqMsb4Z9ytgB%40basil.wdw.