[Sbcl-devel] Announcing parallel-mark-region GC

[Douglas Katzman via Sbcl-devel]

Hi everybody, the new GC by Hayley is available for testing.

Hayley; I did not merge in 0002-simplify-find-object so if you wouldn't mind trying to apply it to the current tip-of-tree and mail it back, that would be fantastic.

Meanwhile I need to add some :skipped-on to some tests, and fix the arena tests which now crash badly and which I'm pretty sure it is my doing.

Doug

[Hayley Patton]

Hi everybody, the new GC by Hayley is available for testing.

Woohoo!

Hayley; I did not merge in 0002-simplify-find-object so if you wouldn't mind trying to apply it to the current tip-of-tree and mail it back, that would be fantastic.

Attached, with fixing/simplifying the maintenance of the remset for incremental compaction too. (Hasn't crashed for three weeks while constantly running a fuzz tester; before it'd last a day at best.)

I'll have to double-check the design document by the way - I know at least "We need to dirty both the start and the slot when the object is large." is no longer true e.g.

[Sebastien Marie]

On Wed, Aug 09, 2023 at 10:31:21PM -0400, Douglas Katzman via Sbcl-devel wrote:
> Hi everybody, the new GC by Hayley

> <https://sourceforge.net/p/sbcl/sbcl/ci/40276b25370f2af1e1870d3b55d62846543a5b47/>
> is available for testing.
>

I am testing it on OpenBSD x86-64 (with adding some missing bits in
Config.x86-64-bsd)

It doesn't compile without gcc-tls.

in mark-region.c:

1169 #ifndef LISP_FEATURE_GCC_TLS
1170 int ok = !pthread_key_create(&recycle_list_key, 0) &&
1171 !pthread_key_create(&output_block_key, 0) &&
1172 !pthread_key_create(&dirty_generation_source_key, 0) &&
1173 !pthread_key_create(&dirty_key, 0) &&
1174 !pthread_key_create(&source_object_key, 0);
1175 gc_assert(ok);
1176 #endif

but there is no corresponding definition for the variables. also, these
variables seems unused.

removing them or compiling with gcc-tls makes the build continue.

it fails later at make-target-2.sh beginning (with CORRUPTION WARNING).

as it isn't a supported configuration, this might be expected. but the
LISP_FEATURE_GCC_TLS chunk needed reporting.

thanks.
--
Sebastien Marie


_______________________________________________
Sbcl-devel mailing list
Sbcl-...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-devel

[Douglas Katzman via Sbcl-devel]

Can you see if adding gcc-tls to your build configuration works for OpenBSD ?   All it's saying is we need __thread annotations to work, versus pthread_[sg]etspecific.

I would think that it's pretty much standard by now.

I have made the new code compile without gcc-tls but it crashed on me; I'm unsure if it was because it was macOS that I developed on, and there's some different issue.

Subsequently I determined that macOS would work without gcc_tls which is why I never pushed the patches.  The changes were very messy and I don't like them at all.

[Sebastien Marie]

On Thu, Aug 10, 2023 at 07:46:58AM -0400, Douglas Katzman wrote:
> Can you see if adding gcc-tls to your build configuration works for OpenBSD
> ? All it's saying is we need __thread annotations to work, versus
> pthread_[sg]etspecific.
> I would think that it's pretty much standard by now.

replying on :gcc-tls part. I will make another mail for mark-region GC.

On OpenBSD, sbcl is compilable with :gcc-tls. It built out-of-box on x86-64, but
needed patching on arm64.

I attached the patch used. I tested it on x86-64 and aarch64. I saw no other asm
usage in sbcl, so it should still build fine on others archs (mostly i386 and
powerpc).

Please note that OpenBSD uses emutls for TLS (Thread Locale Storage). So the
effective implementation is similar to use pthread_[sg]etspecific behind the
scene. But it put the responsability on the OS and not on sbcl itself, so it is
fine.

Thanks.
--
Sebastien Marie

[Douglas Katzman via Sbcl-devel]

Can you trying re-sending that as 'git format-patch -1' rather than 'git show' which is what I think it looks like?

I got errors trying to git automerge it:

error: patch failed: make-config.sh:610
error: make-config.sh: patch does not apply
error: patch failed: src/runtime/arm64-assem.S:141
error: src/runtime/arm64-assem.S: patch does not apply

Great that it works though. Thanks for testing.

btw I remember now why we have very limited testing: the gcc compiler farm has no machine with RWX allowed on file mappings.

[Sebastien Marie]

On Thu, Aug 10, 2023 at 07:46:58AM -0400, Douglas Katzman wrote:

The build fails the same way with or without :gcc-tls.


I am building sbcl with:

- :gcc-tls enable (see patch in previous mail)

- src/runtime/Config.x86-64-bsd change to enable LISP_FEATURE_MARK_REGION_GC

path + /usr/obj/sbcl/sbcl
blob - c2d3b07c8e73c765caebaec8a58ec8022e0d6c36
file + src/runtime/Config.x86-64-bsd
--- src/runtime/Config.x86-64-bsd
+++ src/runtime/Config.x86-64-bsd
@@ -27,6 +27,10 @@ else

ifdef LISP_FEATURE_IMMOBILE_SPACE
GC_SRC = fullcgc.c gencgc.c traceroot.c immobile-space.c elf.c
+else ifdef LISP_FEATURE_MARK_REGION_GC
+ # The parallelized mark-region collector can perform a full GC,
+ # but can't use fullcgc to perform heap->arena pointer detection.
+ GC_SRC = fullcgc.c pmrgc.c traceroot.c mark-region.c incremental-compact.c gc-thread-pool.c
else
GC_SRC = fullcgc.c gencgc.c traceroot.c
endif


- using sbcl HEAD (+few patches for cleaning cc warning) as compiler


The build is done using:

$ sh ./make.sh \
--prefix="${INSTALL_DIR}" \
--xc-host='sbcl --disable-debugger --no-sysinit --no-userinit' \
--with-sb-core-compression \
--with-sb-xref-for-internals \
--with-sb-thread \
\
--without-gencgc \
--with-mark-region-gc

I attached the full build log (gzipped).

it fails in make-target-2.sh:

//entering make-target-2.sh
//doing warm init - compilation phase
This is SBCL 2.3.7.22817.gc.3.9337bf47, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
Initial page table:
Immobile Object Counts
Gen layout fdefn symbol code Boxed Cons Raw Code SmMix Mixed LgRaw LgCode LgMix Waste% Alloc Trig Dirty GCs Mem-age
6 0 0 0 0 0 149 0 449 0 662 0 8 71 0.6 43619424 2000000 0 0 0.0000
Tot 0 0 0 0 0 149 0 449 0 662 0 8 71 0.6 43619424 [5.4% of 805240832 max]
CORRUPTION WARNING in SBCL pid 54636 pthread 0x3625fcfe728:
Memory fault at 0xffffffffe5ef80a5 (pc=0x35f98833ad4, fp=0x361bc4efaa0, sp=0x361bc4efa20) pthread 0x3625fcfe728
The integrity of this image is possibly compromised.
Exiting.
Welcome to LDB, a low-level debugger for the Lisp runtime environment.
ldb> //doing warm init - load and dump phase
CORRUPTION WARNING in SBCL pid 51107 pthread 0x317f2885728:
Memory fault at 0x312320a5 (pc=0x3150641ead4, fp=0x31717b47aa0, sp=0x31717b47a20) pthread 0x317f2885728
The integrity of this image is possibly compromised.
Exiting.
Welcome to LDB, a low-level debugger for the Lisp runtime environment.
ldb> [2] + Stopped (tty input) ./src/runtime/sbcl --noinform --core output/col
[1] - Stopped (tty input) ./src/runtime/sbcl --core output/cold-sbcl.core
0m00.02s real 0m00.00s user 0m00.00s system


--
Sebastien Marie

[Douglas Katzman via Sbcl-devel]

Wondering if an accidental '#+linux' got in there somewhere.  I'm going to try set up a QMU since the GCC farm is useless for this.

Can you just send me a link to whatever network installer is gonna work for me with least pain on QEMU ? Sorry but I'm lazy in regards to finding all the downloadable stuff from OS distros

[Sebastien Marie]

On Thu, Aug 10, 2023 at 09:15:00AM -0400, Douglas Katzman wrote:
> Wondering if an accidental '#+linux' got in there somewhere. I'm going to
> try set up a QMU since the GCC farm is useless for this.
> Can you just send me a link to whatever network installer is gonna work for
> me with least pain on QEMU ? Sorry but I'm lazy in regards to finding all
> the downloadable stuff from OS distros

You could download https://cdn.openbsd.org/pub/OpenBSD/snapshots/amd64/miniroot73.img

It will ask for network for downloading all sets.


Once installed, you could install some packages for ease the initial build:

# pkg_add sbcl zstd git


You could 'upgrade' temporary a partition to wxallowed using:

# mount -u -o wxallowed /home


I am building using the following invocation (mostly to configure zstd search
path without patching):

$ env \
CFLAGS="-I/usr/local/include" \
LDFLAGS="-L/usr/local/lib" \
LINKFLAGS="-L/usr/local/lib" \

sh ./make.sh \
--prefix="${INSTALL_DIR}" \
--xc-host='sbcl --disable-debugger --no-sysinit --no-userinit' \
--with-sb-core-compression \
--with-sb-xref-for-internals \
--with-sb-thread

[Sebastien Marie]

I am not using git, but got (https://gameoftrees.org/).

I remade the patch using git this time.

Sorry.
--
Sebastien Marie

[Douglas Katzman via Sbcl-devel]

Please test whether my wrong sizeof patch fixes anything for you.

Actually I doubt it because os_sem_t should be sem_t for you. But try it anyway?

[Douglas Katzman via Sbcl-devel]

macOS still can't run parallel-exec.sh but it can run-tests.sh

So I would suspect that something around forking is wrong. Parallel-exec is using sb-posix:fork so we should see the (DARWIN-REINIT) take care of things.

Nonetheless it gets SIGILL in a GC thread. I don't suppose you have any idea? Stas probably could spot it in the blink of an eye. (nudge nudge)

[Stas Boukarev]

I don't have an x86-64 macOS anymore, so it'll require more blinking.

_______________________________________________

[Sebastien Marie]

On Thu, Aug 10, 2023 at 10:25:21AM -0400, Douglas Katzman wrote:
> Please test whether my wrong sizeof

> <https://sourceforge.net/p/sbcl/sbcl/ci/5cee9d3c5daf47a599da258702831c2cc238b45f/>

> patch fixes anything for you.
> Actually I doubt it because os_sem_t should be sem_t for you. But try it
> anyway?

I already tested it :)

Same problem, sbcl fails to run at make-target-2.sh stage (but not exactly the
same error message, but the used sbcl compiler isn't exactly the same too).

//entering make-target-2.sh
//doing warm init - compilation phase

This is SBCL 2.3.7.3.master.0.5cee9d3c, an implementation of ANSI Common Lisp.

More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.

fatal error encountered in SBCL pid 85751 pthread 0xac52e287728:
failed to read allocation bitmap from core

[Douglas Katzman via Sbcl-devel]

Add stdlib.h to incremental-compact.c and see if that fixes things for you.  I believe it will because calloc() is getting an assumed return type of 'int' and losing the high 4 bytes of a pointer

[Sebastien Marie]

On Thu, Aug 10, 2023 at 01:42:08PM -0400, Douglas Katzman wrote:
> Add stdlib.h to incremental-compact.c and see if that fixes things for
> you. I believe it will because calloc() is getting an assumed return type
> of 'int' and losing the high 4 bytes of a pointer

I confirm that it makes it to build and run fine.

The master HEAD (97238f8f1afa0a061022d43ae852a8f93f1993b6 at time) is building
fine using --without-gencgc --with-mark-region-gc.

I ran run-tests.sh too, and compared with the same build using gencgc.


I am seeing new failures, and few new success.

Failures:
- hash.pure.lisp / (HASH-TABLE GC-SMASHED-CELL-LIST)
- arena.impure.lisp / INTERN-A-BUNCH
- deadlock.impure.lisp (:gc-deadlock - process killed by me)
- gc.impure.lisp
- hash-table.impure.lisp / ADDRESS-INSENSITIVE-EQL-HASH
- traceroot.impure.lisp / (SEARCH-ROOTS STACK-INDIRECT)
- traceroot.impure.lisp / TRACEROOT-COLLAPSE-LISTS


Full diff of test summary (ran on OpenBSD x86-64).

--- run-tests.sh (after ./make.sh)
+++ run-tests.sh (after ./make.sh --without-gencgc --with-mark-region-gc)
Finished running tests.
Status:
Skipped (broken): compiler.pure.lisp / FULL-WARNING-FOR-UNDEFINED-TYPE-IN-CL
Skipped (broken): compiler.pure.lisp / SINGLE-WARNING-FOR-SINGLE-UNDEFINED-TYPE
Skipped (broken): gethash-concurrency.pure.lisp / (HASH-TABLE UNSYNCHRONIZED)
+ Failure: hash.pure.lisp / (HASH-TABLE GC-SMASHED-CELL-LIST)
Expected failure: hash.pure.lisp / SXHASH-ON-DISPLACED-STRING
+ Unexpected success: aprof.impure.lisp / APROF-SMOKETEST-STRUCT
+ Expected failure: aprof.impure.lisp / APROF-SMOKETEST-STRUCT
+ Failure: arena.impure.lisp / INTERN-A-BUNCH
Expected failure: block-compile.impure.lisp / BLOCK-COMPILE-TOP-LEVEL-CLOSURES.SAME-ENVIRONMENT.LOCAL-CALLS
Expected failure: compiler-2.impure.lisp / TOP-LEVEL-CLOSURE-SEPARATE-COMPONENT
Expected failure: compiler-2.impure.lisp / TOP-LEVEL-CLOSURE-SEPARATE-COMPONENT.2
+ Invalid exit status: deadlock.impure.lisp (:gc-deadlock - process killed by me)
Expected failure: full-eval.impure.lisp / INLINE-FUN-CAPTURES-DECL
+ Invalid exit status: gc.impure.lisp
+ Failure: hash-table.impure.lisp / ADDRESS-INSENSITIVE-EQL-HASH
Expected failure: packages.impure.lisp / USE-PACKAGE-CONFLICT-SET
Expected failure: packages.impure.lisp / IMPORT-SINGLE-CONFLICT
Skipped (broken): run-program.impure.lisp / (RUN-PROGRAM AUTOCLOSE-STREAMS)
- Failure: ../contrib/sb-concurrency/tests/test-mailbox.lisp / MAILBOX.MULTIPLE-PRODUCERS-MULTIPLE-CONS>
Unhandled Error sb-simd.impure.lisp
Skipped (broken): threads.impure.lisp / BACKTRACE
+ Failure: traceroot.impure.lisp / (SEARCH-ROOTS STACK-INDIRECT)
+ Failure: traceroot.impure.lisp / TRACEROOT-COLLAPSE-LISTS
Invalid exit status: run-program.test.sh


Thanks.

[Hayley Patton]

On 11/8/23 18:25, Sebastien Marie wrote:
> I confirm that it makes it to build and run fine.

Congratulations!

> I am seeing new failures, and few new success.
>
> Failures:
> - hash.pure.lisp / (HASH-TABLE GC-SMASHED-CELL-LIST)
> - arena.impure.lisp / INTERN-A-BUNCH
> - deadlock.impure.lisp (:gc-deadlock - process killed by me)
> - gc.impure.lisp
> - hash-table.impure.lisp / ADDRESS-INSENSITIVE-EQL-HASH
> - traceroot.impure.lisp / (SEARCH-ROOTS STACK-INDIRECT)
> - traceroot.impure.lisp / TRACEROOT-COLLAPSE-LISTS

These are known...except for the deadlock one, I can't remember if I saw
that before. But there seems to be something funky about finishing
tracing - I've seen tracing get stuck with parallel-exec.

[Douglas Katzman via Sbcl-devel]

Thanks for the follow-up!

I observed the deadlock one also on the emulator but I figured it was due to non-real hardware.

gdb seemed not to be able to access symbols for me.

Is there any chance you want to help out the RISC-V OpenBSD people make progress on just a baseline build? (https://bugs.launchpad.net/sbcl/+bug/2029382)

[Sebastien Marie]

On Fri, Aug 11, 2023 at 06:50:09AM -0400, Douglas Katzman wrote:
> Thanks for the follow-up!
> I observed the deadlock one also on the emulator but I figured it was due
> to non-real hardware.
> gdb seemed not to be able to access symbols for me.

gdb in base is now too old for doing useful stuff. install the one from packages
(with "pkg_add gdb"), and use the "egdb" binary (it is gdb 9.2).

> Is there any chance you want to help out the RISC-V OpenBSD people make
> progress on just a baseline build? (
> https://bugs.launchpad.net/sbcl/+bug/2029382)

I have little knowing in sbcl internals, and about zero in RISC-V cpus
(used registers, asm…). Robert already asked questions on openbsd lists.