How to limit amount of memory used by vim?

[Igor]

Hi,

using Vim 7.4 on Linux server I have opened large log file and I tried to perform some log formatting using ":%s" command.

It looks like Vim froze (probably it didn't, it just worked hard), but like after a minute Linux server got out of memory and Linux start killing the biggest memory consumption processes including one very important server. From Linux logs I see vim consumed 50 GB of memory, that is what triggered Linux to get out of memory.

Just wondering is there a way to set in Vim, how much memory is Vim allowed to use? If Vim exceeds that amount the last action performing is just canceled with error.

Regards

[Dominique Pellé]

Igor wrote:

> Just wondering is there a way to set in Vim, how much memory is Vim allowed to use?

No, but there are ways to reduce memory usage. E.g.

* turn off syntax highlighting: syntax off
* turn off undo: `:set undolevels=0`
* tun off spell checking `:set nospell`
* turning off clipboard: by starting vim with 'vim -X'
* check :help 'maxmem' and :help 'maxmempattern' which
may also help but it won't prevent Vim from using plenty of
memory if it needs.
* probably other settings can help to reducing memory?

> using Vim 7.4 on Linux server

That's a very old version of Vim BTW. Some memory leaks have been
fixed since that release thanks to better tooling nowadays (valgrind, asan,
fuzzers). So perhaps upgrading Vim can help reducing memory usage.

The command below shows memory leaks which were fixed but most of
them were in new features fixed shortly after they were introduced or in
in rare code path, sometimes found by fuzzing. So most of the
leaks won't affect you in practice. You just need one memory
leak bug to sometimes increase memory usage though.

$ grep 'memory leak' version8.txt version9.txt
version8.txt:Problem: Valgrind reports memory leak for job that is not freed.
version8.txt:Problem: Setting really_exiting causes memory leaks to
be reported.
version8.txt:Solution: Add test with viminfo lines in bad syntax.
Fix memory leak.
version8.txt: function. Fix memory leak.
version8.txt:Problem: Pending timers cause false memory leak reports.
version8.txt:Problem: More memory leaks when using partial, e.g.
for "exit-cb".
version8.txt:Problem: test_partial has memory leaks reported.
version8.txt:Problem: Test_job_start_fails() is expected to report
memory leaks, making
version8.txt: Fix memory leak.
version8.txt:Solution: Fix memory leak. Fix crash. Add a few more
tests. (Christian
version8.txt: Fix memory leak.
version8.txt:Problem: MS-Windows: memory leak when creating terminal fails.
version8.txt:Problem: Various memory leaks reported by asan.
version9.txt:Many memory leaks, invalid memory accesses and crashes
have been fixed.
version9.txt:Solution: Improve parsing, fix memory leak, add tests.
(Ozaki Kiichi,
version9.txt:Problem: Various memory leaks.
version9.txt:Problem: Various memory leaks in Vim9 script code.
version9.txt:Problem: Vim9: memory leaks.
version9.txt:Problem: Vim9: memory leak when finding common type.
version9.txt:Solution: Check index for being valid. Fix memory leak.
version9.txt:Problem: Vim9: memory leak when using lambda.
version9.txt:Problem: Vim9: memory leak when using nested function.
version9.txt:Problem: Vim9: memory leak in disassemble test.
version9.txt:Problem: Vim9: another memory leak.
version9.txt:Problem: Vim9: memory leak in script test.
version9.txt:Problem: Vim9: memory leak in expr test.
version9.txt:Problem: Vim9: memory leak when script var has wrong type.
version9.txt:Problem: Possible memory leak when file expansion fails.
version9.txt: memory leak. Avoid unnecessary copy/free.
version9.txt:Problem: Vim9: memory leak in allocated types.
version9.txt:Problem: Vim9: memory leak when using nested global function.
version9.txt:Problem: Vim9: memory leak when using nested global function.
version9.txt:Problem: Vim9: memory leak when storing a value fails.
version9.txt:Problem: Vim9: memory leak when using multiple closures.
version9.txt:Problem: Vim9: memory leak with heredoc that isn't
executed. (Dominique
version9.txt:Problem: Vim9: memory leaks when using nested function.
version9.txt:Problem: Vim9: memory leak if "if" condition is invalid.
version9.txt:Problem: Vim9: memory leak when using function reference.
version9.txt:Problem: Vim9: another memory leak when using function
reference.
version9.txt:Problem: Vim9: memory leaks reported in assign test.
version9.txt:Problem: Vim9: memory leak when throwing empty string.
version9.txt:Problem: vim9: memory leak when compiling lambda fails.
version9.txt:Problem: Vim9: memory leak when using add().
version9.txt:Problem: Win32: memory leak when encoding conversion fails.
version9.txt:Problem: Vim: memory leak when :execute fails.
version9.txt:Problem: Vim9: memory leak when statement is truncated.
version9.txt:Problem: Vim9: memory leak when literal dict has an
error and when an
version9.txt:Problem: Vim9: memory leak when parsing lambda fails.
version9.txt:Problem: Vim9: memory leak when parsing nested parenthesis.
version9.txt:Problem: Vim9: memory leak with catch in skipped block.
version9.txt:Problem: Coverity warns for memory leak.
version9.txt:Solution: Clear function growarrays. Fix memory leak.
version9.txt:Problem: Vim9: memory leak when failing on locked variable.
version9.txt:Problem: Vim9: memory leak when calling :def function fails.
version9.txt:Problem: Vim9: memory leak with blob range error.
version9.txt:Problem: Vim9: memory leak when using :s with expression.
version9.txt:Problem: Vim9: memory leak when using \=expr in :substitute.
version9.txt:Problem: Vim9: memory leak when using searchpair().
version9.txt:Problem: Vim9: memory leak when using inline function.
version9.txt:Problem: Vim9: memory leak when lambda has an error.
version9.txt:Problem: Vim9: memory leak
version9.txt:Problem: Vim9: memory leak when debugging a :def function.
version9.txt:Problem: Vim9: memory leak when compilation fails.
version9.txt:Problem: Coverity reports a memory leak.
version9.txt:Problem: Vim9: memory leak when using lambda.
version9.txt:Problem: Vim9: memory leak when add() fails.
version9.txt:Problem: Vim9: memory leak when concatenating to an
imported string.
version9.txt:Problem: Vim9: memory leak when function reports an error.
version9.txt:Problem: Lua: memory leak when adding dict item fails.
version9.txt: Clear the type for flattennew(). Avoid a
memory leak when
version9.txt:Problem: Vim9: memory leak when and/or fails.
version9.txt:Problem: Vim9: memory leak when compiling :elseif fails.
version9.txt:Problem: Coverity reports a memory leak.
version9.txt:Problem: Coverity reports a possible memory leak.
version9.txt:Problem: Vim9: memory leak when text after a nested function.
version9.txt:Solution: Add more tests. Fix uncovered memory leak.
(Yegappan Lakshmanan,
version9.txt:Problem: Vim9: memory leak when exporting function in
autoload script.
version9.txt:Problem: MS-Windows: memory leak in :browse.
version9.txt: Fix memory leaks. (Ken Takata, closes #9754)
version9.txt:Solution: Add an optional argument to give errors. Fix
memory leak when

Regards
Dominique

[Bram Moolenaar]

With a ":%s" command all lines are saved in memory for undo. Vim may
at some point fail to allocate memory and then stop with an error
message. However, because of the way Linux manages memory it will
happily give more memory to Vim than it actually has. There is a system
setting to change that behavior. I recall a lively discussion between
Linux and BSD users about this stragety (BSD works just fine, Linux can
freeze).

In Vim you can disable undo:

setlocal ul=-1


--
"I don’t know how to make a screenshot" - Richard Stallman, July 2002
(when asked to send a screenshot of his desktop for unix.se)

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// \\\
\\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Tim Chase]

On 2022-07-06 11:00, Bram Moolenaar wrote:
> because of the way Linux manages memory it will happily give more
> memory to Vim than it actually has. There is a system setting to
> change that behavior. I recall a lively discussion between Linux
> and BSD users about this stragety (BSD works just fine, Linux can
> freeze).

I believe the setting is controlled by the vm.overcommit_memory
sysctl value. Setting it to 2 instructs the kernel to never
overcommit, letting vim die when it tries to grab more RAM than
available.

https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

Also, if the file is that large and if the :s// isn't overly complex
(e.g. using sub-replace-\= or other vim-specific functionality), it
might be less stressful on the system to use sed(1) instead

sed 's/old/new/g' input.log > output.log

which should run in a fairly fixed amount of RAM.

-Tim

[Steve Litt]

Dominique Pellé said on Wed, 6 Jul 2022 11:51:02 +0200

>Igor wrote:
>
>> Just wondering is there a way to set in Vim, how much memory is Vim
>> allowed to use?
>
>No, but there are ways to reduce memory usage. E.g.
>
>* turn off syntax highlighting: syntax off

[snip excellent advice about Vim using less RAM]

Igor, this probably isn't responsive to you because you're running a
server, but RAM can also be saved by limiting browser tabs and using a
light WM/DE (Window Manager/Desktop Environment) such as Openbox or
LXDE (*NOT* Xfce, it's not light). Also, go easy on the VM's you're
running.

SteveT

Steve Litt
Summer 2022 featured book: Making Mental Models: Advanced Edition
http://www.troubleshooters.com/mmm

[Igor]

In Vim you can disable undo:
setlocal ul=-1

That's it, excellent advice. I now changed script to turn off "undolevels" and after search/replace commands finish the I turn "undolevels" back on.
:setl ul=-1
commands
:setl ul=-123456

This reduced memory consumption from several GB to 10 MB in my case.

Thanks a lot for help.