Using current-process-memory with 'cumulative argument

[Alex Harsanyi]

I am trying to use `(current-process-memory 'cumulative)` to determine the total memory used by an application, including the memory that was reclaimed by the garbage collector.  I would expect the results from the call to be constantly increasing, and this is indeed the case at a "global" scale:

However, occasionally, there seems to be a drop in the reported memory use, as shown, if I zoom in the plot.  The drops are about 8Mb each and there are about 20 of them for a program that makes 1 million calls.

At first, I thought this is a bug in the way I collect the data (and there might be a bug), but I can't seem to spot it, so I am wondering if my expectation is correct that `(current-process-memory 'cumulative)` will always report increasing results?

In case this is important, I am using Racket 7.9 CS on a Windows machine.

Thanks,

Alex. 

[Matthew Flatt]

At Tue, 15 Dec 2020 17:00:42 -0800 (PST), Alex Harsanyi wrote:
> I am trying to use `(current-process-memory 'cumulative)` to determine the
> total memory used by an application, including the memory that was
> reclaimed by the garbage collector. I would expect the results from the
> call to be constantly increasing, and this is indeed the case at a "global"

> scale: [...]

>
> However, occasionally, there seems to be a drop in the reported memory use,

I'm able to replicate that with

#lang racket

(define (work n)
(length (let loop ([n n])
(if (zero? n)
'()
(cons (make-vector n) (loop (sub1 n)))))))

(let loop ([u (current-memory-use 'cumulative)])
(work (random 1000))
(let ([u2 (current-memory-use 'cumulative)])
(if (u2 . < . u)
(error "oops")
(loop u2))))

which reports an error fairly quickly for me.


The implementation of

(current-process-memory 'cumulative)

is

(+ (bytes-deallocated) (bytes-allocated))

It's possible for a GC to happen at the start of the call to
`bytes-allocated`, which I think would produced the blip you see (i.e.,
there could be bytes not yet attributed to `bytes-deallocated`, but
they get shifted there before `bytes-allocated` runs).

The solution is make that arithmetic atomic with respect to a GC by
disabling signals around the arithmetic. I've pushed that change, and
my test program doesn't error anymore, at least.


Matthew

[Alex Harsanyi]

Thanks for providing the fix.  

This is most likely the cause, as my own test program is essentially the same as yours, except that I collect the samples separately rather than checking for the memory decrease immediately.

Alex.