[PATCH] initial support for memory usage report

[Qian Yun]

The support for collecting and reporting memory usage is already there,
just missing a few pieces.

After applying this patch, this functionality can be enabled by:
)boot $printStorageIfTrue := true

Not all lisps support this, so I wonder if this option should
be visible under ")set message time".

- Qian

=====

diff --git a/src/interp/g-timer.boot b/src/interp/g-timer.boot
index 501092e2..61febdf4 100644
--- a/src/interp/g-timer.boot
+++ b/src/interp/g-timer.boot
@@ -79,12 +79,7 @@ normalizeStatAndStringify t ==
t := roundStat t
t = 0.0 => '"0"
FORMAT(nil,'"~,2F",t)
- INTEGERP t =>
- K := 1024
- M := K*K
- t > 9*M => CONCAT(STRINGIMAGE((t + 512*K)/M), '"M")
- t > 9*K => CONCAT(STRINGIMAGE((t + 512)/K), '"K")
- STRINGIMAGE t
+ INTEGERP t => FORMAT(nil, '"~:d", t)
STRINGIMAGE t

significantStat t ==
@@ -171,6 +166,7 @@ initializeTimedNames(listofnames,listofclasses) ==
PUT( name, 'ClassSpaceTotal, 0)
$timedNameStack := '(other)
computeElapsedTime()
+ computeElapsedSpace()
PUT('gc, 'TimeTotal, 0.0)
PUT('gc, 'SpaceTotal, 0)
NIL
@@ -178,6 +174,8 @@ initializeTimedNames(listofnames,listofclasses) ==
updateTimedName name ==
count := (GET(name, 'TimeTotal) or 0) + computeElapsedTime()
PUT(name, 'TimeTotal, count)
+ count := (GET(name, 'SpaceTotal) or 0) + computeElapsedSpace()
+ PUT(name, 'SpaceTotal, count)

makeLongTimeString(listofnames,listofclasses) ==
makeLongStatStringByProperty(listofnames, listofclasses, _
@@ -236,9 +234,6 @@ timedEvaluate code ==
"append"/[eval ["LIST",:x] for x in splitIntoBlocksOf200 a]
eval code

-displayHeapStatsIfWanted() ==
- $printStorageIfTrue => sayBrightly OLDHEAPSTATS()
-
--% stubs for the stats summary fns
statRecordInstantiationEvent() == nil
statRecordLoadEvent() == nil
diff --git a/src/interp/macros.lisp b/src/interp/macros.lisp
index d24589f8..1be5f3b2 100644
--- a/src/interp/macros.lisp
+++ b/src/interp/macros.lisp
@@ -675,6 +675,11 @@ This function respects intermediate #\Newline
characters and drops

(defun WHOCALLED(n) nil) ;; no way to look n frames up the stack

+#+:sbcl
+(defun heapelapsed () (sb-ext:get-bytes-consed))
+#+:openmcl
+(defun heapelapsed () (ccl::total-bytes-allocated))
+#-(or :sbcl :openmcl)
(defun heapelapsed () 0)

(defun |goGetTracerHelper| (dn f oname alias options modemap)

[Qian Yun]

More Lisps are supported:

--- a/src/interp/macros.lisp
+++ b/src/interp/macros.lisp
@@ -675,6 +675,17 @@ This function respects intermediate #\Newline

characters and drops

(defun WHOCALLED(n) nil) ;; no way to look n frames up the stack

+#+:sbcl
+(defun heapelapsed () (sb-ext:get-bytes-consed))
+#+:openmcl
+(defun heapelapsed () (ccl::total-bytes-allocated))

+#+:clisp
+(defun heapelapsed ()
+ (multiple-value-bind (used room static gc-count gc-space gc-time)
(sys::%room)
+ (+ used gc-space)))
+#+:ecl
+(defun heapelapsed () (si::gc-stats t))
+#-(or :sbcl :openmcl :clisp :ecl)

(defun heapelapsed () 0)

(defun |goGetTracerHelper| (dn f oname alias options modemap)

- Qian

[Waldek Hebisch]

On Sun, Oct 15, 2023 at 09:25:05PM +0800, Qian Yun wrote:
> More Lisps are supported:

I tried to combine and apply both patches, but after that
I get:

integrate((f*x+e)^3*csc(d*x+c)/(a+b*sin(d*x+c)),x)
<snipped result>
Type: Union(Expression(Integer),...)
Storage: 0 (IN) + 0 (EV) + 0 (OT) + 0 (GC) = 0 bytes

After

)set messages time on

I get usual time display, but still 0-s in storage report. AFAICS

)boot HEAPELAPSED()

reports increasing values. This is with Debian sbcl 1.3.14.

--
Waldek Hebisch

[Ralf Hemmecke]

It seems that Qian wants to revive the Storage line.
If that happens and there is an easy command like

)set message storage on/off

then this will have an effect on jFriCAS.

If you currently say

)lisp (setf |$printStorageIfTrue| T)

then you only get

Storage: 0 (IN) + 0 (EV) + 0 (OT) + 0 (GC) = 0 bytes

That looked like no information at all.

When I programmed it I needed a way to reliably extract the output
counter from the output. So I switch on $printStorageIfTrue every time
something is sent from the webinterface to FriCAS.

The Storage line that comes back in the output is actually ignored, but
it comes with markers and from them the output counter is extracted.
I cannot extract them from the "Time: " or "Type: " line, since a user
might have switched that off.

In other words,

(1) jFriCAS relies on the existence of that Storage line (content
uniportant)
(2) jFriCAS cannot currently show such a storage line.

Both can, of course, be relaxed if there is a separate reliable way to
extract the step number from the output of FriCAS.
Note that this is not that easy in the frame of ")set output FORMATTER
off", i.e. it must be output that is independent of our formatters.

That's just a side remark and nothing to really worry about.

Ralf

[Qian Yun]

I reorganized the patch a little bit, can you try this:

https://github.com/fricas/fricas/pull/139.patch

I have following output for the integral.

Storage: 1,216,256 (IN) + 1,028,097,520 (EV) + 15,961,856 (OT) + 0 (GC)
= 1,045,275,632 bytes

- Qian

[Qian Yun]

On 10/22/23 05:11, Ralf Hemmecke wrote:
> It seems that Qian wants to revive the Storage line.
> If that happens and there is an easy command like
>
> )set message storage on/off
>
> then this will have an effect on jFriCAS.
>
> If you currently say
>
> )lisp (setf |$printStorageIfTrue| T)
>
> then you only get
>
> Storage: 0 (IN) + 0 (EV) + 0 (OT) + 0 (GC) = 0 bytes
>
> That looked like no information at all.

Can you try if this gives meaningful information:
https://github.com/fricas/fricas/pull/139.patch

> When I programmed it I needed a way to reliably extract the output
> counter from the output. So I switch on $printStorageIfTrue every time
> something is sent from the webinterface to FriCAS.

I didn't know about this, I guess we can find a better solution.

- Qian

[Ralf Hemmecke]

Qian,

> Storage: 1,216,256 (IN) + 1,028,097,520 (EV) + 15,961,856 (OT) + 0 (GC)
> = 1,045,275,632 bytes

can you give a hint how to interpret that information?

For "Time: " it seems intuitive to interpret as "time for input", "time
for evaluation", "time for output" and "total time".

But what exactly does that mean for memory?

Ralf

[Qian Yun]

Actually IN means "interpreter" and OT means "other".

See $interpreterTimedNames, it is classified into 4 categories:
$interpreterTimedClasses (interpreter, evaluation, other, reclaim).
For example, "interpreter" includes "analysis, coercion, modemaps,
querycoerce, resolve".

Grep for "startTimingProcess" to see how each component is timed.

For memory statistics, it is about memory allocated in each of
these components (blocks of code).

- Qian

[Ralf Hemmecke]

On 22.10.23 12:12, Qian Yun wrote:
> Actually IN means "interpreter" and OT means "other".

Oh, I misinterpreted that all that time. :-(
Thank you for clarification.

> For memory statistics, it is about memory allocated in each of
> these components (blocks of code).

%%% (1) -> 1+1

(1) 2
Type:
PositiveInteger
Storage: 16,384 (EV) + 215,792 (OT) = 232,176
bytes
%%% (2) -> 1+1

(2) 2
Type:
PositiveInteger
Storage: 16,368 (OT) = 16,368
bytes

OK, memory allocation makes some sense. Thank you.

BTW, naturally, since I ignore the content of the Storage line, jFriCAS
also works with your patch, but, of course, never shows the Storage line.

Ralf

[Waldek Hebisch]

On Sat, Oct 21, 2023 at 11:11:47PM +0200, Ralf Hemmecke wrote:
> It seems that Qian wants to revive the Storage line.
> If that happens and there is an easy command like
>
> )set message storage on/off
>
> then this will have an effect on jFriCAS.
>
> If you currently say
>
> )lisp (setf |$printStorageIfTrue| T)
>
> then you only get
>
> Storage: 0 (IN) + 0 (EV) + 0 (OT) + 0 (GC) = 0 bytes
>
> That looked like no information at all.
>
> When I programmed it I needed a way to reliably extract the output counter
> from the output. So I switch on $printStorageIfTrue every time something is
> sent from the webinterface to FriCAS.
>
> The Storage line that comes back in the output is actually ignored, but it
> comes with markers and from them the output counter is extracted.
> I cannot extract them from the "Time: " or "Type: " line, since a user might
> have switched that off.

So, you really need additional marker different from existing markers.
(Ab)using Storage line and its markers for different purpose is bad
as Storage line should be under user control.

>
> In other words,
>
> (1) jFriCAS relies on the existence of that Storage line (content
> uniportant)
> (2) jFriCAS cannot currently show such a storage line.
>
> Both can, of course, be relaxed if there is a separate reliable way to
> extract the step number from the output of FriCAS.

We should add this.

> Note that this is not that easy in the frame of ")set output FORMATTER off",
> i.e. it must be output that is independent of our formatters.

I am not sure what you mean here. Do you say that the lines
like

)set output FORMATTER off

pose special difficulty? AFAIK command do not count towards
step number. Or you mean that you want step number regardless
of set (possibly empty ???) of active formatters?

--
Waldek Hebisch

[Qian Yun]

Ping? It is updated at https://github.com/fricas/fricas/pull/139

This functionality can be enabled by ")set message storage on".

- Qian

On 10/15/23 18:04, Qian Yun wrote:

[Waldek Hebisch]

On Fri, Nov 03, 2023 at 07:19:01PM +0800, Qian Yun wrote:
> Ping? It is updated at https://github.com/fricas/fricas/pull/139
>
> This functionality can be enabled by ")set message storage on".

Please commit. I answered earlier to message from pull request,
I am not sure if it reached you.

--
Waldek Hebisch

[Qian Yun]

I only saw one email from Oct/24. I guess github is eating email again.

- Qian