Solaris 11.4, acme, thread library problem

[Noel Hunt]

I have just installed plan9port on the current Solaris release, but there

are problems with what appears to be the threads library. I discovered this

by running a command in an acme tag line, a trivial 'echo foobar' will exhibit

the problem. The +Errors window reports a segmentation violation, a dbx trace

shows the following:

$ dbx /opt/plan9/bin/acme core
Reading acme
core file header read successfully
Reading ld.so.1
Reading libm.so.2
Reading librt.so.1
Reading libpthread.so.1
Reading libsocket.so.1
Reading libnsl.so.1
Reading libthread.so.1
Reading libc.so.1
t@20 (l@20) program terminated by signal SEGV (no mapping at the fault address)
Current function is _threadspawn
   83                   dup2(fd[0], 0);
(dbx) where
current thread: t@20
=>[1] _threadspawn(fd = 0xfd46cdc4, cmd = 0x8110490 "echo", argv = 0x80fa918, dir = 0x8114f80 "/opt/plan9/src/cmd/htmlfmt"), line 83 in "exec.c"
  [2] execproc(v = 0xfd46cb50), line 15 in "exec.c"
  [3] procmain(p = 0x8114380), line 254 in "thread.c"
  [4] startprocfn(v = 0x80fa260), line 99 in "pthread.c"
  [5] _thrp_setup(0xfe4b9240), at 0xfe2a9bd9
  [6] _lwp_start(0x0, 0x0, 0x0, 0xd, 0x10, 0x0), at 0xfe2a9ec0
(dbx) (dbx) up
Current function is execproc
   15           pid = _threadspawn(e->fd, e->cmd, e->argv, e->dir);
(dbx) (dbx) print *e
dbx: cannot access address 0xfd46cb50
(dbx) (dbx) up
Current function is procmain
  254           t->startfn(t->startarg);
(dbx) (dbx) print *t
*t = {
    next       = (nil)
    prev       = (nil)
    allnext    = (nil)
    allprev    = (nil)
    startfn    = 0x80aec20 = &`acme`libthread/exec.c`execproc(void *v)
    startarg   = 0xfd46cb50
    id         = 20U
    osprocid   = 0
    stk        = (nil)
    stksize    = 0
    exiting    = 0
    mainthread = 1
    proc       = 0x8114380
    name       = ""
    state      = ""
    udata      = (nil)
    chanalt    = (nil)
    schedrend  = {
        l      = (nil)
        asleep = 0
        cond   = {
            __pthread_cond_flags = {
                __pthread_cond_flag  = ""
                __pthread_cond_type  = 0
                __pthread_cond_magic = 0
            }
            __pthread_cond_data  = 0
        }
    }
}
(dbx) (dbx) print *t->startarg
*t->startarg = (void)

I can find no documentation or references to _thrp_setup and _lwp_start, no prototypes

even in system include files, so I have no idea where this is coming from.

Noel Hunt

[Russ Cox]

_thrp_setup and _lwp_start are Solaris symbols involved in creating a new thread. It makes sense that they would be at the top of the stack. 

In 2020, I changed libthread to stop doing all the shenanigans we used to do about our own thread stacks, stack switching, and so on. It's all standard pthread use now, so it really should work. The only thing I can think of is that execproc is started with an argument that points to the original pthread's stack. That thread waits for execproc to say it is done before returning, so it should be okay to do that. But maybe Solaris does not let one pthread see another's stack. You could try replacing _runthreadspawn in src/lib/thread/exec.c with this:

int
_runthreadspawn(int *fd, char *cmd, char **argv, char *dir)
{
        int pid;
        Execjob *e;

        e = mallocz(sizeof *e, 1);
        e->fd = fd;
        e->cmd = cmd;
        e->argv = argv;
        e->dir = dir;
        e->c = chancreate(sizeof(void*), 0);
        proccreate(execproc, e, 65536);
        pid = recvul(e->c);
        chanfree(e->c);
        free(e);
        return pid;
}

Best,

Russ

[Noel Hunt]

Thanks for your response, Russ, but the change made no difference,
It did prompt me to dig a bit further though.

What is odd is that it is the 'fd' element of the Execjob struct
that is being affected, the rest, 'cmd' and 'argv' are untouched.

As a reminder of the stack at the time of the SEGV:

(dbx) where
current thread: t@20

=>[1] _threadspawn(e = 0xfd26eedc), line 171 in "exec.c"
  [2] execproc(v = 0x810a358), line 20 in "exec.c"
  [3] procmain(p = 0x8104240), line 254 in "thread.c"
  [4] startprocfn(v = 0x80f9e10), line 99 in "pthread.c"
  [5] _thrp_setup(0xfe4b8240), at 0xfe2a9bd9
  [6] _lwp_start(0x0, 0x0, 0x0, 0xb, 0xf, 0xfd26ee58), at 0xfe2a9ec0
(dbx) print fd[0]
dbx: cannot access address 0xfd46cdc4
(dbx) print cmd, argv[0], argv[1], dir
cmd = 0x81037e0 "echo"
argv[0] = 0x81037e0 "echo"
argv[1] = 0x81037e5 "foo"
dir = (nil)

'print's in '_runthreadspawn' and 'execproc' show that the 'Execjob'
struct is still intact, and in 'threadspawn', before the 'fork', the
int *fd argument is still valid, i.e., you can print the value, but

this becomes bogus after the fork, thus the SEGV. What is special

about that element of Execjob, and why would 'fork' affect it?

[Russ Cox]

Odd. The fd array probably points into some thread's stack (specifically the one in acme calling threadspawnd). Perhaps on Solaris if you fork you lose access to the other thread stacks in the child. You could try making Execjob hold int fd[3] instead of int *fd, copying the 3 fds into it. Then you might survive losing access to the stack.

Best,

Russ

 

[Noel Hunt]

Your fix worked, many thanks.

[Dan Cross]

On Tue, Feb 24, 2026 at 4:31 PM Noel Hunt <noel...@gmail.com> wrote:
> Thanks for your response, Russ, but the change made no difference,
> It did prompt me to dig a bit further though.
>
> What is odd is that it is the 'fd' element of the Execjob struct
> that is being affected, the rest, 'cmd' and 'argv' are untouched.
>
> As a reminder of the stack at the time of the SEGV:
>
> (dbx) where
> current thread: t@20
> =>[1] _threadspawn(e = 0xfd26eedc), line 171 in "exec.c"

This doesn't make a lot of sense to me, as `_threadspawn` is on line
61 of `exec.c`; line 171 is `threadexec` (which does ultimately call
`threadspawn`). It sounds like you've modified that file a bit to
debug, but 110 lines of movement is kind of a lot. Anyway, this
suggests that there's a fair amount of inlining happening.

I can't think of any particularly good reason why `fork` would cause
that memory to go bad, unless it was allocated from a virtual memory
region that was mapped in such a way it was invalidated on fork;
`pthread_atfork` could do something like that, but it doesn't appear
that we use it. There could be some other kind of interaction with a
per-thread storage area that's not inherited across a `fork`, but that
feels extraordinarily dubious: Solaris and illumos are no longer the
same system, but if the latter is any indicator, the former doesn't
have any proc-private memory regions, unlike, say, Plan 9.

Spelunking a little bit, this looks like the `sfd` argument to
`threadspawnd` called from `runproc` in `src/cmd/acme/exec.c`; that's
a stack local on `runproc`'s stack. I'm purely speculating, but it may
be that the appearance with for `fork` is incidental; perhaps
`runproc` has already returned by the time `fork` is invoked. You may
try running with `truss` and seeing what the lwp that calls `runproc`
is doing.

> [2] execproc(v = 0x810a358), line 20 in "exec.c"
> [3] procmain(p = 0x8104240), line 254 in "thread.c"
> [4] startprocfn(v = 0x80f9e10), line 99 in "pthread.c"
> [5] _thrp_setup(0xfe4b8240), at 0xfe2a9bd9
> [6] _lwp_start(0x0, 0x0, 0x0, 0xb, 0xf, 0xfd26ee58), at 0xfe2a9ec0
> (dbx) print fd[0]
> dbx: cannot access address 0xfd46cdc4
> (dbx) print cmd, argv[0], argv[1], dir
> cmd = 0x81037e0 "echo"
> argv[0] = 0x81037e0 "echo"
> argv[1] = 0x81037e5 "foo"
> dir = (nil)

You may have better luck with `mdb` and getting it to print out the
value of `v` (as an `Execjob`) in `procmain`. Something like,
`810a358::print struct Execjob`.

- Dan C.

> 'print's in '_runthreadspawn' and 'execproc' show that the 'Execjob'
> struct is still intact, and in 'threadspawn', before the 'fork', the
> int *fd argument is still valid, i.e., you can print the value, but
> this becomes bogus after the fork, thus the SEGV. What is special
> about that element of Execjob, and why would 'fork' affect it?
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "plan9port-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to plan9port-de...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/plan9port-dev/CAGfO01wONLn3jj6Ch1BAbTBxbeQgN6NXA8VJbJNDD8w-XmMQcw%40mail.gmail.com.

[Russ Cox]

runproc should be blocked in threadspawnd calling _runthreadspawn calling recvul, waiting for the pid of the forked child. 

Until 2020, runproc would have been running on a stack allocated with malloc, and this wouldn't have been a problem. But managing our own stacks was causing more and more fights with host operating systems, so we moved from using our own normally allocated stacks to always running on real pthread stacks. This problem would only have surfaced then.

It seems entirely plausible to me that on Solaris a forked child does not inherit all the other pthread stack mappings. Solaris has always been proud of knowing better than everyone else how things should work and delighted in its weird gratuitous differences compared to other systems. This is exactly the kind of thing they would do. :-)

Best,

Russ

[Dan Cross]

On Wed, Feb 25, 2026 at 9:05 AM Russ Cox <r...@swtch.com> wrote:
> runproc should be blocked in threadspawnd calling _runthreadspawn calling recvul, waiting for the pid of the forked child.

Perhaps it did; I don't see `runproc` in the call stack here, but my
reading of the code last night suggested this is running in a detached
thread, so I wouldn't necessarily expect to see it. But more relevant,
the error happens after the fork; it is possible that `fork` returned
in the parent, unblocked `threadspanwd`, which returned and allowed
`runproc` to return, thus destroying the stack and `sfd`? Then this
thread gets scheduled and runs, and tries to touch stack space that is
now invalid. (I confess I haven't looked too closely today, so this
may be nonsense.)

> Until 2020, runproc would have been running on a stack allocated with malloc, and this wouldn't have been a problem. But managing our own stacks was causing more and more fights with host operating systems, so we moved from using our own normally allocated stacks to always running on real pthread stacks. This problem would only have surfaced then.
>
> It seems entirely plausible to me that on Solaris a forked child does not inherit all the other pthread stack mappings. Solaris has always been proud of knowing better than everyone else how things should work and delighted in its weird gratuitous differences compared to other systems. This is exactly the kind of thing they would do. :-)

Ha! The truth of this hits hard.

- Dan C.

[Russ Cox]

On Wed, Feb 25, 2026 at 5:47 PM Dan Cross <cro...@gmail.com> wrote:

Perhaps it did; I don't see `runproc` in the call stack here, but my
reading of the code last night suggested this is running in a detached
thread, so I wouldn't necessarily expect to see it. But more relevant,
the error happens after the fork; it is possible that `fork` returned
in the parent, unblocked `threadspanwd`, which returned and allowed
`runproc` to return, thus destroying the stack and `sfd`? Then this
thread gets scheduled and runs, and tries to touch stack space that is
now invalid. (I confess I haven't looked too closely today, so this
may be nonsense.)

The child really should not be affected by what happens in the parent after the fork. If the parent unmapping the stack after the fork also unmapped it in the child, that would be more surprising to me than it being unmapped from the child during fork. But you never know.

Best,

Russ

[Dan Cross]

I agree that whatever the parent does should not affect the child
after the fork, but I'm not sure it matters. The thread is detached;
fork will return and unwind the stack through `runproc` in the child
just as it does in the parent. The thread, once started, will execute
independently and ultimately access stack data that may, or may not,
still be valid: it's lifetime is detached from that of the thread that
called `runproc`, which may have even exit'ed by then. Sounds like in
Noel's case, it's coming down on the side of not being valid.

Noel: it may be fruitful to try and `truss` the execution of `acme`
and see if the thread that kicked this whole process off has exited.

At least, that's my guess; perhaps I should go look at the code in a
bit more detail.

- Dan C.

[Russ Cox]

Wow. 

It would never have occurred to me to consider that maybe all the other threads had been duplicated into the child and kept running. No sane kernel would implement that. 

Then again, this is Solaris...

Turns out, yes they did implement that! 

https://docs.oracle.com/cd/E19120-01/open.solaris/816-5137/6mba5vqa4/index.html

We are definitely not using forkall, but I think the discussion there strengthens my case. I'm now 99% sure that "duplicate only the calling thread in the child process" means the other thread's stack is not inherited, which causes the fault.

Best,

Russ

[Dan Cross]

On Wed, Feb 25, 2026 at 8:32 PM Russ Cox <r...@swtch.com> wrote:
> Wow.
>
> It would never have occurred to me to consider that maybe all the other threads had been duplicated into the child and kept running. No sane kernel would implement that.

Ahem.

> Then again, this is Solaris...
>
> Turns out, yes they did implement that!
> https://docs.oracle.com/cd/E19120-01/open.solaris/816-5137/6mba5vqa4/index.html
>
> We are definitely not using forkall, but I think the discussion there strengthens my case. I'm now 99% sure that "duplicate only the calling thread in the child process" means the other thread's stack is not inherited, which causes the fault.

I dunno. It also says that the address space is duplicated.

On both Solaris and illumos, `fork` isn't the actual system call, it's
a library function that delegates to `forkx` with an argument of
flags=0. The actual system call is `sysfork`, which takes an integer
argument describing what type of fork it is (fork1, forkall, vfork)
and whatever flags the user passed.

I'm looking at the illumos code. The `forkx` library function does a
bunch of locking and tries to quiesce other threads before the system
call; there's some Rock-like stuff hiding state in the library itself
to handle all of the `atfork` business and so on. Then it invokes
`__forkx(flags)` with is the stub handler that does the SYSCALL into
the kernel. Once in the kernel, `forksys` switches on the "subcode"
and calls `cfork`, which has this comment early on:

```
/*
* If the calling lwp is doing a fork1() then the
* other lwps in this process are not duplicated and
* don't need to be held where their kernel stacks can be
* cloned. If doing forkall(), the process is held with
* SHOLDFORK, so that the lwps are at a point where their
* stacks can be copied which is on entry or exit from
* the kernel.
*/
```

However, that's referring to the LWP's _kernel_ stack, not user stack,
which should be in the user portion of the address space. Looking a
little further, `cfork` duplicates the address space in the non-vfork
case by calling `as_dup`, but that doesn't look too interesting: it
just walks over the segments in the address space, and as long as
they're not marked with `S_PURGE`, it dup's them into the child. While
it sounds promising, `S_PURGE` is only set on SPARC for "no-fault"
segments, and I don't believe the user-stack for a thread is "no
fault".

There is some more stuff for removing dtrace probes and things in
`cfork` that could conceivably touch the address space, but that
doesn't seem relevant (though there are several dtrace probes in
`cfork` that may be useful for debugging this). The rest is the usual
business about dup'ing resources of various kinds, but I don't see any
reason why the user stack of an LWP calling fork would Not be copied;
that's not to say that I didn't miss something here.

Here's the chain of events on the libthread side:

`acme` calls `threadspawnd`, passing a pointer to a stack-allocated
array of three int's for the three FDs:

* `threadspawnd` invokes `_runthreadspawn`, which packages arguments
up into an `Execjob` on its stack frame and calls, `proccreate`, with
a fn pointer to `execproc` and arg pointer to the (stack-local)
`Execjob` (and stack size of 64k, but that's ignored),
* `proccreate` does an `allocproc` and a `_threadcreate` (which
ignores the stack size argument), caches the `id` from
`_threadcreate`, sets up some "scheduler" state, and then runs
`_procstart`. But note that the thread ID here is just an integer
internal to libthread; `_threadcreate` doesn't actually create a
runnable thread in the `pthead_create` sense; rather, it merely
creates a data structure, and critically, that structure has no
connection to an actual runnable thread yet.
* `_procstart` packages up the thunk and `Execjob` argument in a
two-element array of `void *`s and does `pthread_create`, arranging to
start in `startpthreadfn` with that array as its argument, then
returns to `proccreate`.
`proccreate` then blocks (presumably?) in `recvul`, waiting for the
pid of the new process that was created.

In the thread created by `pthread_create`: on Solaris, that translates
to an LWP, which will invoke `startpthreadfn` with the array above as
that function's argument.
* `startpthreadfn` sets some state on the `_Thread` created by the
`_threadcreate` call in `proccreate` (notably, it sets the thread's
`osprocid` field). It then does `pthread_detach` on itself and invokes
`_threadpthreadmain` which will (finally!!) invoke `execproc`.
* `execproc` calls `_threadspawn`, which will now fork and so on; the
parent returns and does a `sendul` on a channel, sending the pid of
the child process it just created to the thread that is blocked on
`recvul` in `proccreate`.
* The child process is going to set up some state and `execvp`
whatever program `acme` requested; we blow up when we touch the file
descriptor array.

It's notable that the `fork` happens in a totally separate thread than
the one that originally called `threadspawnd` and packaged up the
`Execjob` arguments on its stack; that thread return'ed in
`_procstart`, though it should be blocked in `proccreate` waiting to
receiving a pid on its; I don't see anything that wouldn't copy its
user stack into the child process, so I don't see any way that the
contents would be invalid, unless somehow `recvul` returned
prematurely before the `fork`. I wonder if anything else is scribbling
on the `Execjob` memory.

- Dan C.

[Russ Cox]

Apologies for the excessive snark. 

However, I seriously can't imagine why you would ever fork all the threads. It makes no sense at all. At the point where one thread is calling fork, that thread is in a state where it expects to be duplicated and has a way to signal to the two copies which one they are. Any other thread is just doing its thing independently and then bam! there are two copies of the thread doing the same exact thing, none the wiser, sharing output file descriptors, network connections, and so on. You need some kind of coordination to make this reasonable. If you were coordinating with all the threads and quiesced them all somehow and then did a forkall and woke them back up with different messages in the parent and child, maybe that could work, but I still don't quite see the utility of that approach versus having the child kick off new threads instead. And it's a lot of kernel work to duplicate all the threads, all for this use case that is almost impossible to invoke correctly.

Fork1 has problems, of course, but forkall has so many more problems.

Anyway, enough speculation. I wrote a C program to test the hypothesis, and it seems to indicate that the thread stack is lost across fork. This runs fine on my Mac but crashes on one of the Go solaris builders, which has this uname -a output:

SunOS s11-i386.foss 5.11 11.4.93.215.0 i86pc i386 i86pc kernel-zone

Best,

Russ

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>
#include <sys/wait.h>
#include <sys/types.h>

int *p;

pthread_mutex_t mu = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t ready = PTHREAD_COND_INITIALIZER;
pthread_cond_t never = PTHREAD_COND_INITIALIZER;

void*
thread1(void *v)
{
    int local = 42;
    pthread_mutex_lock(&mu);
    p = &local;
    pthread_cond_signal(&ready);
    pthread_cond_wait(&never, &mu);
    return 0;
}

void*
thread2(void *v)
{
    pthread_mutex_lock(&mu);
    while(p == 0)
        pthread_cond_wait(&ready, &mu);
    pthread_mutex_unlock(&mu);

    printf("parent: *p = %d\n", *p);

    int pid = fork();
    if (pid == 0) {
        printf("child: *p = %d\n", *p);
        exit(0);
    }
    printf("parent: pid = %d\n", pid);
    int status;
    waitpid(pid, &status, 0);
    if(WIFEXITED(status))
        printf("child exit %d\n", WEXITSTATUS(status));
    else if(WIFSIGNALED(status))
        printf("child signal %d\n", WTERMSIG(status));
    else
        printf("unexpected child status %d\n", status);
    return 0;
}

int
main(void)
{
    pthread_t t1, t2;
    pthread_create(&t1, 0, thread1, 0);
    pthread_create(&t2, 0, thread2, 0);
    void *v;
    pthread_join(t2, &v);
    return 0;
}

[Russ Cox]

Noel, I pushed a fix to plan9port. Please see if it works for you. Thanks.

[Dan Cross]

On Thu, Feb 26, 2026 at 9:40 AM Russ Cox <r...@swtch.com> wrote:
> Apologies for the excessive snark.

Ha, no, it's fine: in this case, I was agreeing with your assessment:
the `forkall` behavior _is_ insane.

> However, I seriously can't imagine why you would ever fork all the threads. It makes no sense at all. At the point where one thread is calling fork, that thread is in a state where it expects to be duplicated and has a way to signal to the two copies which one they are. Any other thread is just doing its thing independently and then bam! there are two copies of the thread doing the same exact thing, none the wiser, sharing output file descriptors, network connections, and so on. You need some kind of coordination to make this reasonable. If you were coordinating with all the threads and quiesced them all somehow and then did a forkall and woke them back up with different messages in the parent and child, maybe that could work, but I still don't quite see the utility of that approach versus having the child kick off new threads instead. And it's a lot of kernel work to duplicate all the threads, all for this use case that is almost impossible to invoke correctly.

I'm speculating, but my guess is that this was/is for compatibility
with the earlier `libthread` library that implemented M:N threading. I
don't know that anybody thought it was a good idea, really, but
Solaris has an almost obsessive commitment to backwards compatibility
(in contrast to SunOS4, I think).

> Fork1 has problems, of course, but forkall has so many more problems.
>
> Anyway, enough speculation. I wrote a C program to test the hypothesis, and it seems to indicate that the thread stack is lost across fork. This runs fine on my Mac but crashes on one of the Go solaris builders, which has this uname -a output:
>
> SunOS s11-i386.foss 5.11 11.4.93.215.0 i86pc i386 i86pc kernel-zone
>

> [snip]

Thanks, this is very handy; I was able to reproduce this behavior on
illumos. Here's `truss` output:

: spitfire; truss -f ./rsc
8669: execvex("rsc", 0xFFFFFC7FFFDF9138, 0xFFFFFC7FFFDF9148, 0) argc = 1
8669: sysinfo(SI_MACHINE, "i86pc", 257) = 6
8669: mmap(0x00000000, 56, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FE3030000
8669: mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FE5AC0000
8669: mmap(0x00000000, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FED2A0000
8669: mmap(0x00000000, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FEDAC0000
8669: memcntl(0xFFFFFC7FEF399000, 88824, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
8669: mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FEF320000
8669: memcntl(0x00400000, 4528, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
8669: resolvepath("/usr/lib/amd64/ld.so.1", "/lib/amd64/ld.so.1", 1023) = 18
8669: getcwd("/home/cross", 1019) = 0
8669: resolvepath("/home/cross/rsc", "/home/cross/rsc", 1023) = 15
8669: stat("/home/cross/rsc", 0xFFFFFC7FFFDF8D80) = 0
8669: open("/var/ld/64/ld.config", O_RDONLY) Err#2 ENOENT
8669: stat("/usr/gcc/14/lib/amd64/libc.so.1", 0xFFFFFC7FFFDF8150) Err#2 ENOENT
8669: stat("/lib/64/libc.so.1", 0xFFFFFC7FFFDF8150) = 0
8669: resolvepath("/lib/64/libc.so.1", "/lib/amd64/libc.so.1", 1023) = 20
8669: open("/lib/64/libc.so.1", O_RDONLY) = 3
8669: mmapobj(3, MMOBJ_INTERPRET, 0xFFFFFC7FEF320BE8,
0xFFFFFC7FFFDF80AC, 0x00000000) = 0 8669: close(3) = 0
8669: mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FEDEE0000
8669: memcntl(0xFFFFFC7FEDD00000, 446360, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
8669: mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FEE1D0000
8669: mmap(0x00010000, 24576, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFFFFFC7FEF1F0000
8669: getcontext(0xFFFFFC7FFFDF88A0)
8669: getrlimit(RLIMIT_STACK, 0xFFFFFC7FFFDF8890) = 0
8669: getpid() = 8669 [8668]
8669: lwp_private(0, 0, 0xFFFFFC7FEF1F2A40) = 0x00000000
8669: getrandom("C1AC HDA TBB13 X", 8, 0) = 8
8669: setustack(0xFFFFFC7FEF1F2AE8)
8669: lwp_cond_broadcast(0xFFFFFC7FEDEE01A8) = 0
8669: lwp_cond_broadcast(0xFFFFFC7FEF3201A8) = 0
8669: sysi86(SI86FPSTART, 0xFFFFFC7FFFDF90DC, 0x0000137F, 0x00001F80)
= 0x00000001
8669: sysconfig(_CONFIG_PAGESIZE) = 4096
8669: schedctl() = 0xFFFFFC7FEF31E000
8669: priocntlsys(1, 0xFFFFFC7FFFDF8DE0, 3, 0xFFFFFC7FFFDF8FF0, 0) = 8669
8669: priocntlsys(1, 0xFFFFFC7FFFDF8D50, 1, 0xFFFFFC7FFFDF8F10, 0) = 4
8669: priocntlsys(1, 0xFFFFFC7FFFDF8CF0, 0, 0xFFFFFC7FEDE90BA8, 0) = 4
8669: mmap(0x00000000, 131072, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FEF218000
8669: mmap(0x00000000, 65536, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFFFFFC7FEF2F0000
8669: sigaction(SIGCANCEL, 0xFFFFFC7FFFDF8BC0, 0x00000000) = 0
8669: sysconfig(_CONFIG_STACK_PROT) = 3
8669: mmap(0x00000000, 2088960, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, 4294967295, 0) =
0xFFFFFC7FEE379000
8669: mmap(0x00010000, 65536, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANON|MAP_ALIGN, 4294967295, 0) = 0xFFFFFC7FEF070000
8669: uucopy(0xFFFFFC7FFFDF8B70, 0xFFFFFC7FEE576FE8, 24) = 0
8669: lwp_create(0xFFFFFC7FFFDF8C80, LWP_SUSPENDED, 0xFFFFFC7FFFDF8C7C) = 2
8669/2: lwp_create() (returning as new lwp ...) = 0
8669/1: lwp_continue(2) = 0
8669/2: setustack(0xFFFFFC7FEF0702E8)
8669/1: mmap(0x00000000, 2088960, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, 4294967295, 0) =
0xFFFFFC7FEEAF7000
8669/2: schedctl() = 0xFFFFFC7FEF31E010
8669/1: uucopy(0xFFFFFC7FFFDF8B70, 0xFFFFFC7FEECF4FE8, 24) = 0
8669/1: lwp_create(0xFFFFFC7FFFDF8C80, LWP_SUSPENDED, 0xFFFFFC7FFFDF8C7C) = 3
8669/3: lwp_create() (returning as new lwp ...) = 0
8669/1: lwp_continue(3) = 0
8669/3: setustack(0xFFFFFC7FEF070AE8)
8669/3: schedctl() = 0xFFFFFC7FEF31E020
8669/3: ioctl(1, TCGETA, 0xFFFFFC7FEECF3C50) = 0
8669/3: fstat(1, 0xFFFFFC7FEECF3BD0) = 0
parent: *p = 42
8669/3: write(1, " p a r e n t : * p =".., 16) = 16
8669/3: lwp_suspend(1) = 0
8669/3: lwp_suspend(2) = 0
8669/3: forkx(0) = 8670
8669/3: lwp_continue(1) = 0
8670: forkx() (returning as child ...) = 8669
8669/3: lwp_continue(2) = 0
8670: getpid() = 8670 [8669]
8669/3: lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000, 0x00000000,
0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
parent: pid = 8670
8669/3: write(1, " p a r e n t : p i d ".., 19) = 19
8670: lwp_self() = 3
8670: munmap(0xFFFFFC7FEE379000, 2088960) = 0
8670: lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000, 0x00000000,
0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
8670: Incurred fault #6, FLTBOUNDS %pc = 0x004014B6
8670: siginfo: SIGSEGV SEGV_MAPERR addr=0xFFFFFC7FEE576FAC
8670: Received signal #11, SIGSEGV [default]
8670: siginfo: SIGSEGV SEGV_MAPERR addr=0xFFFFFC7FEE576FAC
8669/3: Received signal #18, SIGCLD, in waitid() [default]
8669/3: siginfo: SIGCLD CLD_DUMPED pid=8670 status=0x000B
8669/3: waitid(P_PID, 8670, 0xFFFFFC7FEECF4E50, WEXITED|WTRAPPED) = 0
child signal 11
8669/3: write(1, " c h i l d s i g n a l".., 16) = 16
8669/3: lwp_sigmask(SIG_SETMASK, 0xFFBFFEFF, 0xFFFFFFF7, 0x000003FF,
0x00000000) = 0xFFBFFEFF [0xFFFFFFFF]
8669/3: open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/SUNW_OST_SGS.mo",
O_RDONLY) Err#2 ENOENT
8669/3: mmap(0x00010000, 65536, PROT_READ|PROT_WRITE|PROT_EXEC,
MAP_PRIVATE|MAP_ANON|MAP_ALIGN, 4294967295, 0) = 0xFFFFFC7FEEFA0000
8669/3: lwp_exit()
8669/1: lwp_wait(3, 0xFFFFFC7FFFDF904C) = 0
8669/1: _exit(0)
: spitfire;

Note that the offending address in the child is 0xfffffc7fee576fac;
that lies within the region that the parent had created via anonymous
memory mapping covering [0xfffffc7fee379000..0xfffffc7fee577000), that
the child explicit `munmap`d when it started running. So it appears
that it _was_ copied into the child, but the child elected to discard
it.

I wonder where that came from. Ah, I see it; there's a function called
in the pthreads library, `postfork1_child`, that runs in the child
after a `fork1`, and clears out LWPs that are no longer runnable. It
contains this comment:

```
/*
* All lwps except ourself are gone. Mark them so.
* First mark all of the lwps that have already been freed.
* Then mark and free all of the active lwps except ourself.
* Since we are single-threaded, no locks are required here.
*/
```

The code stanza following that comment does exactly what it says. The
LWP free routine will free stacks assigned to now-dead LWPs into a
per-process cache, but "trim"s that cache to stay within some bound
(which appears to be tied to the number of active threads) and the
trim function will `munmap` a stack.

So that's almost certainly what's going on: the memory _is_ copied
into the child, but the child's post-fork thread cleanup goo unmaps
it.

- Dan C.

[Noel Hunt]

Yes, that version works fine, thanks.