[vim/vim] Memory leak with timer and lambda (#8439)
Daniel Steinberg
Describe the bug
A timer with a lambda that references a local variable leaks memory.
To Reproduce
Detailed steps to reproduce the behavior:
- Run
vim --clean(orgvim --clean, etc.) - Edit
example.vim
" Show PID, for monitoring memory echom getpid() function! Function(timer) abort let l:cmd = 'let x = 2 + 2' call timer_start(1, {-> execute(l:cmd)}) endfunction let timer = timer_start(20, 'Function', {'repeat': -1})
- Source
example.vim
source example.vim
- Monitor memory
watch 'pmap <PID> | grep total'- Describe the error
Memory continuously increases.
Expected behavior
Memory would remain constant.
Environment (please complete the following information):
- Vim version:
$ vim --version
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Feb 15 2021 12:29:39)
Included patches: 1-2434
Modified by team...@tracker.debian.org
Compiled by team...@tracker.debian.org
- OS: Ubuntu 21.04
- Terminal: Konsole 20.12.3
Additional context
The memory leak does not occur when the executed command is not stored in a variable.
function! Function(timer) abort call timer_start(1, {-> execute('let x = 2 + 2')}) endfunction
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
lacygoill
Bram Moolenaar
Daniel Steinberg wrote:
> **Describe the bug**
>
> A timer with a lambda that references a local variable leaks memory.
>
> Detailed steps to reproduce the behavior:
> 2. Edit `example.vim`
>
> ```vim
> " Show PID, for monitoring memory
> echom getpid()
>
> function! Function(timer) abort
> let l:cmd = 'let x = 2 + 2'
> call timer_start(1, {-> execute(l:cmd)})
> endfunction
>
> let timer = timer_start(20, 'Function', {'repeat': -1})
>
> 3. Source `example.vim`
>
> ```vim
> source example.vim
> ```
>
> 4. Monitor memory
>
> ```sh
> watch 'pmap <PID> | grep total'
>
> 5. Describe the error
>
> Memory continuously increases.
>
> **Expected behavior**
>
> Memory would remain constant.
>
> **Environment (please complete the following information):**
> - Vim version:
>
> ```
> $ vim --version
> VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Feb 15 2021 12:29:39)
> Included patches: 1-2434
> Compiled by ***@***.***
> ```
>
> - OS: Ubuntu 21.04
> - Terminal: Konsole 20.12.3
>
> **Additional context**
>
> The memory leak does not occur when the executed command is not stored
> in a variable.
>
> function! Function(timer) abort
> call timer_start(1, {-> execute('let x = 2 + 2')})
> endfunction
I cannot reproduce it. When I run the script under valgrind it does not
report any leak (compiled with -DEXITFREE). I also tried writing a test
with this mechanism and run it under valgrind, didn't report a problem
either.
Possibly it's not a real leak but inefficient use of memory. Can't
guess how.
--
"Space is big. Really big. You just won't believe how vastly hugely mind-
bogglingly big it is. I mean, you may think it's a long way down the
road to the chemist, but that's just peanuts to space."
-- Douglas Adams, "The Hitchhiker's Guide to the Galaxy"
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// \\\
\\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
Daniel Steinberg
I encountered this issue using an earlier version of Vim.
After compiling the latest version from source, I still encounter the continuously increasing memory usage (monitored with watch 'pmap <PID> | grep total').
The Valgrind leak summaries differ:
Leak summary using pre-installed Vim on my system:
==123893== LEAK SUMMARY:
==123893== definitely lost: 0 bytes in 0 blocks
==123893== indirectly lost: 0 bytes in 0 blocks
==123893== possibly lost: 342,051 bytes in 5,601 blocks
==123893== still reachable: 5,800,465 bytes in 23,704 blocks
==123893== suppressed: 0 bytes in 0 blocks
Leak summary with Vim compiled from source:
==123454== LEAK SUMMARY:
==123454== definitely lost: 520 bytes in 1 blocks
==123454== indirectly lost: 132 bytes in 6 blocks
==123454== possibly lost: 0 bytes in 0 blocks
==123454== still reachable: 77,961 bytes in 873 blocks
==123454== suppressed: 0 bytes in 0 blocks
Christian Brabandt
definitely lost: 520 bytes in 1 blocks
There are some bytes lost, but that's clearly not your issue. This really does not look like a memory leak.
Can you share the complete valgrind log?
Daniel Steinberg
"Can you share the complete valgrind log?"
👍 Sure. Attached are both logs, for pre-installed Vim and Vim compiled from source. For the latter, I uncommented LEAK_CFLAGS = -DEXITFREE in src/Makefile, based on the instructions in :help valgrind. This was presumbably not done for the pre-installed Vim, based on there being no -DEXITFREE shown in :version output.
valgrind-compiled-vim.txt
valgrind-pre-installed-vim.txt
"Possibly it's not a real leak but inefficient use of memory."
"This really does not look like a memory leak."
I've changed the title and description accordingly.
Christian Brabandt
Okay, I did not see any leaks that come from Vim directly. I skimmed through it and it looks all the leaks where caused by some libraries (libXt, libX11 etc). I do wonder, if you stop the timer, does perhaps garbage collector skip in and releases some memory later?
Daniel Steinberg
"I do wonder, if you stop the timer, does perhaps garbage collector skip in and releases some memory later?"
When I run :call timer_stop(timer) to stop the outer timer, the memory that grew from earlier is not released.
I also tried retaining the inner timers (those created in Function that run once) in an array and later calling timer_stop on all the IDs in the array. That also had no effect on the memory that grew from earlier.
Bram Moolenaar
You can see a big difference in "still reachable" between the run with and without EXITFREE. That means it's not a real leak, but Vim does keep a lot of things in memory. It might indeed be that that the garbage collector is not triggered, or that there is a dependency cycle. This may relate to register_closure() and current_funccal->fc_funcs
Daniel Steinberg
For the title and description, I mentioned timers, as that was the context under which I noticed the memory increase, and I thought there was a relationship.
However, the following steps also result in unreleased memory, without the use of timers.
To Reproduce
- Run
vim --clean(orgvim --clean, etc.) - Edit
example.vim
" Show PID, for monitoring memory echom getpid()
function! Function() abort let l:string = 'hello world!' return {-> l:string} endfunction
- Source
example.vim
:source example.vim
- Monitor memory
$ watch 'pmap <PID> | grep total'- Call the function many times
:for _ in range(1, 500000) | call Function() | endfor
- Describe the error
Vim's memory usage increases to over 1GB.
Bram Moolenaar
What happens if you delete the function: :delfunc Function
Daniel Steinberg
"What happens if you delete the function:
:delfunc Function"
Memory usage remains the same (slightly above 1GB in the example above) after running :delfunc Function. I also tried :call garbagecollect(), which also did not effect the memory usage.
Daniel Steinberg
I've simplified the example further below. It's not necessary to use a string local variable, nor return the lambda from Function.
To Reproduce
- Run
vim --clean(orgvim --clean, etc.) - Edit
example.vim
" Show PID, for monitoring memory echom getpid()
function! Function() abort let x = 0 let Y = {-> x} endfunction
- Same as above
...
Bram Moolenaar
> I've simplified the example further below. It's not necessary to use a
>
> **To Reproduce**
>
> 1. Run `vim --clean` (or `gvim --clean`, etc.)
> 2. Edit `example.vim`
>
> ```vim
> echom getpid()
>
> let x = 0
> let Y = {-> x}
> endfunction
>
> 3. Same as above
> ...
This clearly is a circular dependency: the local variables of Function()
are used by the lambda, and the lambda is stored in a local variable.
Thus the lambda can't be freed because it is referred to by a local
variable, and the local variables cannot be freed because they are
referred to be the lambda.
Not sure how to solve this. We should somehow detect that "Y" isn't
used.
--
The only backup you need is the one that you didn't have time for.
(Murphy)
/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// \\\
\\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
—
Christian Brabandt
> Am 24.06.2021 um 22:18 schrieb Bram Moolenaar ***@***.***>:
>
>
>
> > I've simplified the example further below. It's not necessary to use a
> > string local variable, nor return the lambda from `Function`.
> >
> > **To Reproduce**
> >
> > 1. Run `vim --clean` (or `gvim --clean`, etc.)
> > 2. Edit `example.vim`
> >
> > ```vim
> > echom getpid()
> >
> > let x = 0
> > let Y = {-> x}
> > endfunction
> > ```
> >
> > 3. Same as above
> > ...
>
> This clearly is a circular dependency: the local variables of Function()
> are used by the lambda, and the lambda is stored in a local variable.
> Thus the lambda can't be freed because it is referred to by a local
> variable, and the local variables cannot be freed because they are
> referred to be the lambda.
>
> Not sure how to solve this. We should somehow detect that "Y" isn't
> used.
>
> --
> The only backup you need is the one that you didn't have time for.
> (Murphy)
>
> /// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
> /// \\\
> \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
> \\\ help me help AIDS victims -- http://ICCF-Holland.org ///
> —
Daniel Steinberg
"Perhaps just mention to avoid circular dependencies and if in doubt simply use unlet! to free local variables that are no longer needed."
When using unlet!, the memory is released for the simplified example from above (without the return).
function! Function() abort let x = 0 let Y = {-> x
} unlet! Y endfunction
I also tried it with an adapted version of the function shown in :help closure (adapted to create and call Bar many times), which returns the closure from the function, which results in the memory not being released.
function Foo(arg) let i = 3 return {x -> x + i - a:arg} endfunction for _ in range(100000) let Bar = Foo(4) call Bar(6) unlet! Bar endfor
The memory only increases the first time running. That is, if I re-run the for loop, I don't see a subsequent increase in memory.
Bram Moolenaar
Daniel Steinberg wrote:
> > "Perhaps just mention to avoid circular dependencies and if in doubt
> > simply use unlet! to free local variables that are no longer
> > needed."
>
> from above (without the `return`).
>
> ```vim
> function! Function() abort
> let x = 0
> let Y = {-> x}
> unlet! Y
> endfunction
> ```
>
> I also tried it with an adapted version of the function shown in
> `:help closure` (adapted to create and call `Bar` many times), which
> returns the closure from the function, which results in the memory not
> being released.
>
> function Foo(arg)
> let i = 3
> return {x -> x + i - a:arg}
> endfunction
>
> for _ in range(100000)
> let Bar = Foo(4)
> call Bar(6)
> unlet! Bar
> endfor
> ```
>
> The memory only increases the first time running. That is, if I re-run
> the `for` loop, I don't see a subsequent increase in memory.
code. And most systems don't release (virtual) memory obtained before,
so the size won't go down. Thus this is normal.
--
"You know, it's at times like this when I'm trapped in a Vogon airlock with
a man from Betelgeuse and about to die of asphyxiation in deep space that I
really wish I'd listened to what my mother told me when I was young!"
"Why, what did she tell you?"
"I don't know, I didn't listen!"
-- Arthur Dent and Ford Prefect in Douglas Adams'
"The Hitchhiker's Guide to the Galaxy"
/// \\\
\\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///
—
Bram Moolenaar
Had another look at this, and I see two separate problems.
First is for the original case: the timer calls the function over and over again. In this case memory will be freed as soon as you type something, so that the main loop is encountered and the garbage collector is invoked. You can see this by sourcing the script, wait until the number starts going up, then type some command repeatedly, e.g. "jk". You will see the number stops rising and may also go down, until you stop typing for a while, then it goes up again.
I would consider this normal behavior. We have had problems before calling the garbage collector when not in the main loop, since it's very difficult to track everything that might be in use somewhere halfway a command.
The second is the case where we have the circular reference: A local variable holds a reference to a closure, and the closure uses a variable in the function context. It's very difficult to detect that this cycle has no outside references. It might be possible, but quite hard to implement and test. I'll add a note in the help to avoid this. It normally would not happen anyway, the example is not useful code.
Bram Moolenaar
Closed #8439.
Daniel Steinberg
Thanks!