Question about memory usage

[Michał Kondraciuk]

Hello,

Originally I wanted to report this to bug-gnu-emacs, but it's so basic
that I thought I'd ask here instead.

Basically, when I run the sexp below in emacs -Q, Emacs keeps allocating
a lot of memory. In 10 minutes, it goes from 18 MB to over 200 MB.

(while t
(with-temp-buffer
(setq buffer-undo-list nil)
(insert "a")))


Calling garbage-collect afterwards or even inside the body of the loop
doesn't help (except the loop obviously runs slower, so after 10
minutes, Emacs uses ~100 MB of memory).
So I want to know if this behavior is expected for some reason? Does
Emacs reuse this memory somehow (to make future allocations faster)? I
tested on newest master and 25.3 and the behavior is the same.

[Óscar Fuentes]

Maybe it is related to this (emphasis mine):

-- User Option: gc-cons-threshold
The value of this variable is the number of bytes of storage that
must be allocated for Lisp objects after one garbage collection in
order to trigger another garbage collection. You can use the
result returned by ‘garbage-collect’ to get an information about
size of the particular object type; space allocated to the contents
of buffers does not count. NOTE THAT THE SUBSEQUENT GARBAGE
COLLECTION DOES NOT HAPPEN IMMEDIATELY WHEN THE THRESHOLD IS
EXHAUSTED, BUT ONLY THE NEXT TIME THE LISP INTERPRETER IS CALLED.

As your example runs in one call of the interpreter, the gargage
collector is never kicked.

Curiously, if a call to garbage-collect is put inside the loop, memory
usage also grows, although slowly.

As for returning the memory to the OS, I've seen Emacs doing that for
very large objects. The rest is simply marked as free.

[Eli Zaretskii]

> From: Michał Kondraciuk <k.mi...@zoho.com>
> Date: Mon, 2 Apr 2018 13:57:26 +0200

>
> Basically, when I run the sexp below in emacs -Q, Emacs keeps allocating
> a lot of memory. In 10 minutes, it goes from 18 MB to over 200 MB.
>
> (while t
> (with-temp-buffer
> (setq buffer-undo-list nil)
> (insert "a")))
>
>
> Calling garbage-collect afterwards or even inside the body of the loop
> doesn't help (except the loop obviously runs slower, so after 10
> minutes, Emacs uses ~100 MB of memory).

What do you mean by "afterwards"? The while-loop never ends, so
there's no "afterwards" AFAIU. Am I missing something?

To answer your question: yes, I think this is expected, given that you
set buffer-undo-list to nil (what is the purpose of that, btw?). If
you either delete that line or change it to set buffer-undo-list to t,
you will see a very different picture, as far as the Emacs memory
footprint is concerned.

> So I want to know if this behavior is expected for some reason? Does
> Emacs reuse this memory somehow (to make future allocations faster)?

When Emacs ends up requesting more memory from the OS, it usually
doesn't release that memory when it is no longer needed, but keeps it
in the process's address space and reuses it if/when it needs more
memory. Buffer text allocation and deallocation is treated
differently, in that when a buffer is killed, the memory used for its
text is returned to the OS, which is why deleting the line that sets
buffer-undo-list has the effect it does, I think.

Is there some real-life issue behind this experiment?

[Stefan Monnier]

>> (while t
>> (with-temp-buffer
>> (setq buffer-undo-list nil)
>> (insert "a")))

[...]

> To answer your question: yes, I think this is expected, given that you
> set buffer-undo-list to nil (what is the purpose of that, btw?). If

Hmm... why would setting this var make any significant difference here?


Stefan

[Eli Zaretskii]

> From: Stefan Monnier <mon...@iro.umontreal.ca>
> Date: Tue, 03 Apr 2018 08:28:46 -0400

Because the variable is on/referenced via the stack, I suppose.

[Eli Zaretskii]

> Date: Tue, 03 Apr 2018 15:40:27 +0300
> From: Eli Zaretskii <el...@gnu.org>

Or maybe I misunderstand what you meant by "make significant
difference"? What did you mean?

[Stefan Monnier]

>> >> (while t
>> >> (with-temp-buffer
>> >> (setq buffer-undo-list nil)
>> >> (insert "a")))
>> [...]
>> > To answer your question: yes, I think this is expected, given that you
>> > set buffer-undo-list to nil (what is the purpose of that, btw?). If
>> Hmm... why would setting this var make any significant difference here?
> Because the variable is on/referenced via the stack, I suppose.

I don't follow. At the end of each iteration of the loop, we kill the
temp buffer, so the effect of having set its buffer-undo-list slot or
not should be negligible (or more specifically: this effect should only
exist until the next GC).


Stefan

[Eli Zaretskii]

> From: Stefan Monnier <mon...@iro.umontreal.ca>
> Date: Tue, 03 Apr 2018 09:13:44 -0400

>
> >> Hmm... why would setting this var make any significant difference here?
> > Because the variable is on/referenced via the stack, I suppose.
>
> I don't follow. At the end of each iteration of the loop, we kill the
> temp buffer, so the effect of having set its buffer-undo-list slot or
> not should be negligible (or more specifically: this effect should only
> exist until the next GC).

As I said: maybe I misunderstand what you are asking. So let me say
something that perhaps better matches your question: a temporary
buffer has its undo turned off, but setting buffer-undo-list to nil
turns this on again, so inserting a character into the buffer conses
stuff onto the undo-list.

Does that answer the question "why would setting the variable make any
difference"?

[Stefan Monnier]

> As I said: maybe I misunderstand what you are asking. So let me say
> something that perhaps better matches your question: a temporary
> buffer has its undo turned off, but setting buffer-undo-list to nil
> turns this on again, so inserting a character into the buffer conses
> stuff onto the undo-list.

Yes, but that undo-list is local to the buffer, so once we kill the
buffer (at the end of each iteration) this should be reclaimable.


Stefan

[Stefan Monnier]

>> Yes, but that undo-list is local to the buffer, so once we kill the
>> buffer (at the end of each iteration) this should be reclaimable.

> Why do you think it isn't reclaimed?

I don't know if it's reclaimed or not, but I think it should be
reclaimable. And if the GC reclaims it, next time around the loop the
same (now free) cons-cell should be reused instead of requesting more
memory from the OS.

So according to how I expect the code to behave, this setting might
cause the memory footprint to be (very slightly) larger but it should not
cause it to keep growing.


Stefan

[Stefan Monnier]

>> But shouldn't Emacs reuse the memory from previous loop iteration
>> instead of allocating it?
> That depends on the heap fragmentation and the efficiency of the
> memory allocating functions to deal with fragmentation.

The elements of buffer-undo-list in this case are just cons cells (and
immediate integers and nil), IIUC so I don't see why fragmentation would
get in the way: the previous cons-cell of the buffer-undo-list of the
previous iterations, reclaimed by the GC, should be readily reusable.


Stefan

[Eli Zaretskii]

> From: Stefan Monnier <mon...@iro.umontreal.ca>
> Date: Tue, 03 Apr 2018 17:18:39 -0400

Cons cells are GCed only when enough of them were consed.

[Eli Zaretskii]

> From: Michał Kondraciuk <k.mi...@zoho.com>
> Date: Tue, 3 Apr 2018 21:16:41 +0200
>
> > To disable undo, you should bind buffer-undo-list to t, not to nil.
> > And with-temp-buffer already does that, because temporary buffers have
> > their undo disabled by default.
>
> I know that, I meant it's common to see code like this in some packages,
> especially when the buffer is actually displayed.
>
> > Did you try not setting buffer-undo-list at all? What did you see
> > then?
>
> Emacs behaves as expected, i.e. memory usage is ~20MB all the time.

OK, so what is your question now? Are you asking, like Stefan, why
setting buffer-undo-list to nil in this case makes a difference, or
are you asking a more general question (and if the latter, what are
you asking)?

[Stefan Monnier]

>> >> But shouldn't Emacs reuse the memory from previous loop iteration
>> >> instead of allocating it?
>> > That depends on the heap fragmentation and the efficiency of the
>> > memory allocating functions to deal with fragmentation.
>> The elements of buffer-undo-list in this case are just cons cells (and
>> immediate integers and nil), IIUC so I don't see why fragmentation would
>> get in the way: the previous cons-cell of the buffer-undo-list of the
>> previous iterations, reclaimed by the GC, should be readily reusable.
> Cons cells are GCed only when enough of them were consed.

They get GC'd when the GC is run. Yes, the GC won't be invoked at each
iteration, but that shouldn't affect the long-term trend. At least
I don't see how it can explain the growth that the OP describes: it can
explain a slightly larger footprint, but not a continued growth.


Stefan

[Michał Kondraciuk]

On 04/04/2018 08:55 AM, Eli Zaretskii wrote:
>>> Did you try not setting buffer-undo-list at all? What did you see
>>> then?
>>
>> Emacs behaves as expected, i.e. memory usage is ~20MB all the time.
>
> OK, so what is your question now? Are you asking, like Stefan, why
> setting buffer-undo-list to nil in this case makes a difference, or
> are you asking a more general question (and if the latter, what are
> you asking)?

I found out what the problem was:

(while t
(with-temp-buffer
(setq buffer-undo-list nil)
(insert "a")

(print (length undo-auto--undoably-changed-buffers)
#'external-debugging-output)))

The variable undo-auto--undoably-changed-buffers was storing a lot of
killed buffers. There's a timer that periodically clears this variable,
but it didn't get a chance to run, since Emacs was never idle. Calling
sit-for inside the loop solves this - not that it matters, because this
is not "normal" code. Thanks for the responses.

[Eli Zaretskii]

> From: Michał Kondraciuk <k.mi...@zoho.com>
> Cc: help-gn...@gnu.org, mon...@iro.umontreal.ca
> Date: Thu, 5 Apr 2018 20:06:34 +0200

>
> (while t
> (with-temp-buffer
> (setq buffer-undo-list nil)
> (insert "a")
> (print (length undo-auto--undoably-changed-buffers)
> #'external-debugging-output)))
>
> The variable undo-auto--undoably-changed-buffers was storing a lot of
> killed buffers.

Maybe kill-buffer should remove that buffer's reference from the list,
then.

[Stefan Monnier]

> I found out what the problem was:
>

> (while t
> (with-temp-buffer
> (setq buffer-undo-list nil)
> (insert "a")
> (print (length undo-auto--undoably-changed-buffers)
> #'external-debugging-output)))

Aha!

> The variable undo-auto--undoably-changed-buffers was storing a lot of killed
> buffers.

We could/should probably arrange to scan for dead buffers every time we
add a new buffer to it, or from kill-buffer-hook.


Stefan

[Michał Kondraciuk]

On 04/05/2018 08:54 PM, Stefan Monnier wrote:
> We could/should probably arrange to scan for dead buffers every time we
> add a new buffer to it, or from kill-buffer-hook.

Can you take a look at the attached patch?

[Eli Zaretskii]

> Cc: Eli Zaretskii <el...@gnu.org>, help-gn...@gnu.org
> From: Michał Kondraciuk <k.mi...@zoho.com>
> Date: Sat, 7 Apr 2018 15:15:08 +0200

>
> Can you take a look at the attached patch?
>

> diff --git a/lisp/simple.el b/lisp/simple.el
> index aad8d3b..23f7c4f 100644
> --- a/lisp/simple.el
> +++ b/lisp/simple.el
> @@ -3014,6 +3014,13 @@ undo-auto--undoably-changed-buffers
> `undo-auto--boundaries' and can be affected by changes to their
> default values.")
>
> +(defun undo-auto--remove-buffer-from-changed-list ()
> + "Remove current buffer from list of recently changed ones."
> + (setq undo-auto--undoably-changed-buffers
> + (delq (current-buffer) undo-auto--undoably-changed-buffers)))
> +
> +(add-hook 'kill-buffer-hook #'undo-auto--remove-buffer-from-changed-list)
> +

Thanks, but I'd be more comfortable with just doing this from
kill-buffer itself. IMO, hooks are for users and Lisp applications;
using them for internal bookkeeping purposes should be limited to the
cases where there's no better alternative.

[Stefan Monnier]

> Thanks, but I'd be more comfortable with just doing this from
> kill-buffer itself.

I installed an alternative patch.


Stefan

[Michał Kondraciuk]

Great, this patch also made the loop run much faster. Thank you.