Re: [Sbcl-devel] [Sbcl-bugs] `sb-ext:save-lisp-and-die` doesn't call finalizers before dumping image

[Stas Boukarev]

(sb-ext:finalize ... :dont-save t)

On Sun, Oct 15, 2023 at 12:04 PM bohonghuang via Sbcl-bugs <sbcl...@lists.sourceforge.net> wrote:

Dear SBCL Developers,

This issue seems to be present only in recent versions (both 2.3.8 and
2.3.9) of SBCL (at least I didn't encounter it in 2.2.5). SBCL does not
invoke the finalizer of objects during image dumping but rather during
image restoration. Here is the reproducing procedure:

```
$ sbcl
This is SBCL 2.3.9, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
* (progn (sb-ext:finalize (make-array 1) (lambda () (format t "I'm finalized.~%"))) nil)
NIL
* (sb-ext:save-lisp-and-die "test" :executable t)
[undoing binding stack and other enclosing state... done]
Forced compaction moved 1472 pages
[performing final GC...Forced compaction moved 1239 pages
 done]
[saving current Lisp image into test:
writing 14880 bytes from the static space at 0x50100000
writing 83001344 bytes from the dynamic space at 0x1000000000
writing 7404704 bytes from the read-only space at 0xfff8f0000
done]

$ ./test
I'm finalized.
*
```

Since finalizers are commonly used to handle the deallocation of alien
memory, after image restoration, pointers to alien memory will become
invalid, resulting in memory errors within the finalizer:

```
CORRUPTION WARNING in SBCL pid 54447 tid 54448:
Memory fault at 0x55689bc9dc94 (pc=0x7efce5d86c35, fp=0x7efce58a6810, sp=0x7efce58a67f0) tid 54448
The integrity of this image is possibly compromised.
Continuing with fingers crossed.
WARNING:
   Error calling finalizer #<FUNCTION (LAMBDA ()
                                        :IN
                                        GIR::INFO-FFI-FINALIZE) {1000168B9B}>:
  #<SB-SYS:MEMORY-FAULT-ERROR {1002278D83}>
```

Thank you for your dedication in advance. If needed, I am willing to
provide assistance to the best of my abilities.

Best regards,

bohonghuang



_______________________________________________
Sbcl-bugs mailing list
Sbcl...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-bugs

[Stas Boukarev]

No, the default can't be changed.

If you have foreign resources remaining after dumping an image you're going to have a bad time whether it's through finalizers or some other paths.

So it's best to initialize them after starting the saved image.

On Sun, Oct 15, 2023 at 6:00 PM lis...@qq.com <lis...@qq.com> wrote:

Thank you for your response. Please forgive me for not consulting the
manual beforehand. However, I believe many library authors seem to have
overlooked this issue and simply used the `tg:finalize` from
`trivial-garbage` without invoking `sb-ext:finalize` with the
`:dont-save t` argument to manage foreign memory. Do you think
`tg:finalize` should default to calling `sb-ext:finalize` with this
argument? If so, would modifying the default parameter of
`sb-ext:finalize` be more appropriate than modifying the calling form in
`tg:finalize`?

[Douglas Katzman via Sbcl-devel]

I think :dont-save as the default in sb-ext:finalize would make sense because one almost never wants finalizers on foreign resources to be saved.   (And finalizers shouldn't be used to manage non-foreign resources for the most part).  So "do save" is a strange usage of finalizers, and even if there were a reason to save them, they don't work - generation 6 objects can not die.  

Maybe someone who uses other lisps can figure out what they do, and whether SBCL can be more compatible, or already is.

[Stas Boukarev]

It's too late to change the API.

[Stas Boukarev]

Finalizers can run at any time or never at all.

On Sun, Oct 15, 2023 at 6:46 PM bohonghuang <lis...@qq.com> wrote:

Thank you for your explanation, but based on what you've said, it is
highly likely to be a bug. Taking the previous code snippet as an example:

* (progn (sb-ext:finalize (make-array 1) (lambda () (format t "I'm finalized.~%"))) nil)

* (sb-ext:save-lisp-and-die "test" :executable t)

The return value of `(make-array 1)` is obviously inaccessible, so it
should have been reclaimed during the final garbage collection before
the image dump. However, its finalizer is executed when restoring the
image. If we consider `(make-array 1)` as a holder of foreign resources,
then during the image dump, there should be no objects holding foreign
resources in the entire world. However, during image restoration, its
finalizer is still invoked, leading to illegal accesses.

I have some doubts about whether this is the intended behavior,
especially when it behaves inconsistently compared to the old version of
SBCL.