shm_open / shm_unlink

[Rick Payne]

I've been playing around with my rebar3_osv tool (which turns an erlang
application into an OSv image). I'm trying to move everything to the
latest erlang release (24.1).

OTP-24 comes with asmjit which can give quite a performance boost in
some cases. However, it requires the use of shm_open and shm_unlink.

I don't find this under OSv - whats the status? Is it something that
could be done under OSv or am I going to find this is a blocking issue?
Currently this is where I'm at:

/otp/erts-12.1.1/bin/beam.smp: ignoring missing symbol shm_unlink
/otp/erts-12.1.1/bin/beam.smp: ignoring missing symbol shm_open
dlsym: No error information
Aborted


Cheers,
Rick

[Nadav Har'El]

On Sun, Oct 3, 2021 at 10:08 AM Rick Payne <ri...@rossfell.co.uk> wrote:

I've been playing around with my rebar3_osv tool (which turns an erlang
application into an OSv image). I'm trying to move everything to the
latest erlang release (24.1).

OTP-24 comes with asmjit which can give quite a performance boost in
some cases. However, it requires the use of shm_open and shm_unlink.

I don't find this under OSv - whats the status? Is it something that
could be done under OSv or am I going to find this is a blocking issue?
Currently this is where I'm at:

Shared memory support was never a big priority because it's main use case is multiple processes that want to share memory, and those (multiple processes) were never a thing in OSv. However, you're right that it's a shame to lose compatibility with an existing application just because it wants to use shared memory even if it didn't have to.

OSv does support the System V shared memory API (see libc/shm.cc) but not the Posix one. I think it should be fairly easy to implement the Posix API (shm_open(), shm_unlink()) almost exactly the same as the code in libc/shm.cc. That existing code already has a "shm_file" implementation - a file descriptor that can be mmapped, so I think the implementation should be very easy and straightforward (might be even easier than what's already in the existing libc/shm.cc). Let me know if you need any help doing it.

I opened https://github.com/cloudius-systems/osv/issues/1173

/otp/erts-12.1.1/bin/beam.smp: ignoring missing symbol shm_unlink
/otp/erts-12.1.1/bin/beam.smp: ignoring missing symbol shm_open
dlsym: No error information
Aborted


Cheers,
Rick

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/491e69e8dd9f8e748d96c545cf8159556da1ba47.camel%40rossfell.co.uk.

[Rick Payne]

On Sun, 2021-10-03 at 10:38 +0300, Nadav Har'El wrote:
>
> Shared memory support was never a big priority because it's main use
> case is multiple processes that want to share memory, and those
> (multiple processes) were never a thing in OSv. However, you're right
> that it's a shame to lose compatibility with an existing application
> just because it wants to use shared memory even if it didn't have to.
>
> OSv does support the System V shared memory API (see libc/shm.cc) but
> not the Posix one. I think it should be fairly easy to implement the
> Posix API (shm_open(), shm_unlink()) almost exactly the same as the
> code in libc/shm.cc. That existing code already has a "shm_file"
> implementation - a file descriptor that can be mmapped, so I think
> the implementation should be very easy and straightforward (might be
> even easier than what's already in the existing libc/shm.cc). Let me
> know if you need any help doing it.

Thanks, I'll take a look. My first attempt was to try and add the musl
mman/shm_open.c file but that just resulted in me hitting the
page_fault assert 'assert(ef->rflags & processor::rflags_if);'

I'll take a look at the libc/shm.cc code and see what I can puzzle
out...

Cheers
Rick

[Nadav Har'El]

On Sun, Oct 3, 2021 at 10:49 AM Rick Payne <ri...@rossfell.co.uk> wrote:

On Sun, 2021-10-03 at 10:38 +0300, Nadav Har'El wrote:
>
> Shared memory support was never a big priority because it's main use
> case is multiple processes that want to share memory, and those
> (multiple processes) were never a thing in OSv. However, you're right
> that it's a shame to lose compatibility with an existing application
> just because it wants to use shared memory even if it didn't have to.
>
> OSv does support the System V shared memory API (see libc/shm.cc) but
> not the Posix one. I think it should be fairly easy to implement the
> Posix API (shm_open(), shm_unlink()) almost exactly the same as the
> code in libc/shm.cc. That existing code already has a "shm_file"
> implementation - a file descriptor that can be mmapped, so I think
> the implementation should be very easy and straightforward (might be
> even easier than what's already in the existing libc/shm.cc). Let me
> know if you need any help doing it.

Thanks, I'll take a look. My first attempt was to try and add the musl
mman/shm_open.c

As far as I can tell, that implementation relies on the kernel implementing a "/dev/shm", which on Linux it's usually a mounted tmpfs. This is yet another shared memory API doing yet another variant of the same thing :-)

We can probably make this Musl shm_open.c work by mounting a ramfs (that's our tmpfs-like filesystem) on /dev/shm, and then hopefully mman/shm_open.c should just work. In fact, if your root directory is already a ramfs (maybe it is?), you don't even need to mount anything - just to create an empty directory /dev/shm. But you would need to create the empty directory!

The other alternative is, as I said, ignoring /dev/shm and musl's shm_open() implementation, and do our own similar to the code in libc/shm.cc. 

file but that just resulted in me hitting the
page_fault assert 'assert(ef->rflags & processor::rflags_if);'

I'm curious where -

Maybe we have a bug in our /dev (fs/devfs/*) implementation? It should generate good errors when trying to open /dev/shm/something - not assertion failures and crashes.

[Rick Payne]

On Sun, 2021-10-03 at 11:04 +0300, Nadav Har'El wrote:
> I'm curious where -
> Maybe we have a bug in our /dev (fs/devfs/*) implementation? It
> should generate good errors when trying to open /dev/shm/something -
> not assertion failures and crashes.

Well, this is one of those rabbit holes... The assert was happening in
the abort code. Not sure I understand why yet, but the root cause seems
to be that the erlang runtime is looking for __sigaction. It wants to
play games with the signal handler (for reaons outlined below).

#if !(defined(__GLIBC__) || defined(__DARWIN__) || defined(__NetBSD__)
|| \
defined(__FreeBSD__) || defined(__sun__))
/*
* Unknown libc -- assume musl, which does not allow safe signals
*/
#error "beamasm requires a libc that can guarantee that sigaltstack
works"
#endif /* !(__GLIBC__ || __DARWIN__ || __NetBSD__ || __FreeBSD__
|| \
*
__sun__) \
*/

Now because of the way erts is being built, it actually thinks is on
glibc. So we end up in this function:


static int (*next_sigaction)(int, const struct sigaction *, struct
sigaction *);

static void do_init(void) {
next_sigaction = dlsym(RTLD_NEXT, NEXT_SIGACTION);

if (next_sigaction != 0) {
return;
}

perror("dlsym");
abort();
}

NEXT_SIGACTION is set to '__sigaction' as it would be for glibc. So the
perror("dlsym") is responsible for the message "dlsym: No error
information" we get. Then we abort().

So before I start to put more effort into shm_open / shm_unlink - I
need to understand a bit more about the signal requirements. The
relevnt comment seems to be:

/*
* Erlang code compiled to x86 native code uses RSP as its stack
pointer. This
* improves performance in several ways:
*
* - It permits the use of the x86 call and ret instructions, which
* reduces code volume and improves branch prediction.
* - It avoids stealing a gp register to act as a stack pointer.
*
* Unix signal handlers are by default delivered onto the current
stack, i.e.
* RSP. This is a problem since our native-code stacks are small and
may not
* have room for the Unix signal handler.
*
* There is a way to redirect signal handlers to an "alternate" signal
stack by
* using the SA_ONSTACK flag with the sigaction() library call.
Unfortunately,
* this has to be specified explicitly for each signal, and it is
difficult to
* enforce given the presence of libraries.
*
* Our solution is to override the C library's signal handler setup
procedure
* with our own which enforces the SA_ONSTACK flag.
*/

Rick

[Nadav Har'El]

On Sun, Oct 3, 2021 at 3:27 PM Rick Payne <ri...@rossfell.co.uk> wrote:

On Sun, 2021-10-03 at 11:04 +0300, Nadav Har'El wrote:
> I'm curious where -
> Maybe we have a bug in our /dev (fs/devfs/*) implementation? It
> should generate good errors when trying to open /dev/shm/something -
> not assertion failures and crashes.

Well, this is one of those rabbit holes...  The assert was happening in
the abort code. Not sure I understand why yet, but the root cause seems
to be that the erlang runtime is looking for __sigaction.

So the good (?) news is that the shm_* code didn't matter at all. Maybe it isn't even getting used... The problem is something completely different:

There is one minor side-bug here - that dlsym() apparently fails to set errno which is why we get the silly "No error information" message there.

The main problem here is we are missing an alias __sigaction for sigaction, which this code seems to be looking for. You can triviall add such an alias in libc/aliases.ld and see if it solves the problem.

OSv's signal handler *do* support sigaltstack() and SA_ONSTACK so that should work.

I'm not sure their "overriding sigaction()" hack will work as expected in OSv - whether or not the "libraries" (?) will see the modified sigaction() or OSv's one also depends on the load order - I guess you can check that.

Rick

[Rick Payne]

Hi,

On Mon, 2021-10-04 at 09:38 +0300, Nadav Har'El wrote:
>
> So the good (?) news is that the shm_* code didn't matter at all.
> Maybe it isn't even getting used... The problem is something
> completely different:

Ah, I think the shm_open is still relevant. For now I'm including the
musl file and will see where that ends up. We're using zfs and I have
/dev/shm created.

> There is one minor side-bug here - that dlsym() apparently fails to
> set errno which is why we get the silly "No error information"
> message there.
>
> The main problem here is we are missing an alias __sigaction for
> sigaction, which this code seems to be looking for. You can triviall
> add such an alias in libc/aliases.ld and see if it solves the
> problem.

Yup, thats what I tried today. Making progress - I now seem to get to
the point that some erlang code is being run - so this is encouraging.

> OSv's signal handler *do* support sigaltstack() and SA_ONSTACK so
> that should work.

I saw that.

> I'm not sure their "overriding sigaction()" hack will work as
> expected in OSv - whether or not the "libraries" (?) will see the
> modified sigaction() or OSv's one also depends on the load order - I
> guess you can check that.

Yes, its going to be somewhat interesting. I'll let you know how I get
on...

Thanks for all the pointers...

Cheers
Rick

[Nadav Har'El]

You're welcome. If you have patches that might be useful to others as well, please post them.

Cheers
Rick

[Rick Payne]

On Mon, 2021-10-04 at 10:37 +0300, Nadav Har'El wrote:
> You're welcome. If you have patches that might be useful to others as
> well, please post them.

Will do. Not there yet - but once I can verify things, I'll send
patches..

Cheers
Rick

[Gregory Burd]

Rick,

I've been following your work with rebar3 and considering adapting it into distillery (https://github.com/bitwalker/distillery).  I'd love to be able to generate an image (AMI, or whatever format) that is the combination of OSv, Elixir, BEAM, etc. in a single easy step.  Thanks for digging into the missing pieces in OTP-24, if you want help or at least a reviewer I'm happy to do what I can.

-greg

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/eea05cd62db7700330ac2a86e6794e7f9d6e0ec9.camel%40rossfell.co.uk.

[Rick Payne]

Hi Greg,

Yes - it would be great to get support for elixir. I did try and engage with the elixir community before to see how we could use the same tooling - but I didn’t get much back. I’ll contact you off list…

Cheers

Rick

On 13 Oct 2021, at 06:53, Gregory Burd <gr...@burd.me> wrote: