gVisor and something introduced between 3.3.1 and 3.7.1

[Cefan Daniel Rubin]

Good morning all, happy 2025 cosmonauts!

One of the popular ways to isolate a container at present is to use the Go-based gVisor runtime. Using the runtime (when installed) with Docker is as simple as the following:

docker run --runtime=runsc --rm -it ubuntu /bin/bash

Unfortunately something about more recent cosmocc compiled binaries results in a "Segmentation fault" almost immediately (seemingly before the first syscall succeeds in the case below). You can reproduce by contrasting the execution of these binaries by the gVisor runtime:

https://cosmo.zip/pub/cosmos/v/3.3.1/bin/date

https://cosmo.zip/pub/cosmos/v/3.7.1/bin/date

Does anyone have a sense of what I can do to keep using the latest cosmocc but avoid this problem? Thank you!

- Cefan

[Justine Tunney]

Can you get a core dump? First, try building a small program with cosmocc that reproduces the issue. This way you'll have the `hello` and the `hello.com.dbg` binaries for debug symbols. Next, once you have your core dump, you can `p/x $rip` and `et` which gives you a backtrace:

define et
  set $x = (void **)$rbp
  while $x
    x/2a $x
    set $x = (void **)$x[0]
  end
end

I normally keep that in my ~/.gdbinit config. Once you have rip, e.g. 0x412443, you can say:

cosmoaddr2line hello.com.dbg 0x412443 ...

To get a backtrace.

--
You received this message because you are subscribed to the Google Groups "cosmopolitan cosmonauts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cosmopolitan-l...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/cosmopolitan-libc/9f512c77-0e4f-4eec-8f36-3f3238aa4246n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[cd rubin]

Unfortunately gVisor does not allow core dumps as far as I can tell. 

Within gdb hello.dbg, after issuing run I see:

Program received signal SIGSEGV, Segmentation fault.
0x0000000000405637 in __init_fds (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at libc/intrin/fds.c:117
117  fds->p[0].flags = O_RDONLY;

I can get: 

$1 = (void (*)()) 0x405637 <__init_fds+1415>

from p $rip

That et command returns:

0x7ec98cf78620: 0x7ec98cf78640 0x4048f6 <_init_fds+16>
0x7ec98cf78640: 0x7ec98cf78650 0x4049f9 <cosmo+35>
0x7ec98cf78650: 0x0 0x4045f9 <_start+130>

but tool/cosmocc/bin/cosmoaddr2line o/examples/hello.dbg 0x405637 (run on the host machine)

responds with error: addr2line failed

I'm copying a hello.dbg binary built on my host machine into this gVisor container as I cannot build the binary within the container because the current cosmo tools segfault in there. 

Not sure this is much help! Thanks as ever for everyone's efforts in this project!

[Justine Tunney]

What's probably happening is gVisor is preventing the mmap(0x6fe000040000, MAP_FIXED) that happens earlier in _init_fds() from succeeding. Try running strace on the process or pass the --strace flag to the cosmo binary. We don't like making assumptions about an address being available. FD tracking is the one and only place where we do it (we used to do it a lot more). We want to get rid of that requirement soon. It's going to take a little bit of refactoring to get there.

[cd rubin]

strace ./hello.dbg responds with:

execve("./hello.dbg", ["./hello.dbg"], 0x7eea6af73680 /* 9 vars */) = 0
getpid()                                = 3869
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
arch_prctl(ARCH_SET_GS, 0x42f680)       = -1 EINVAL (Invalid argument)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x6fe000004} ---
+++ killed by SIGSEGV +++
Segmentation fault

while ./hello.dbg --strace doesn't show anything at all before the segfault.

Thank you for the super quick replies! I'll try to switch to an earlier version of cosmo in the meantime. Cheers!

[Justine Tunney]

Whaaat. Why won't it let us change %gs? We need that for TLS.

[cd rubin]

For anyone else passing this way and wanting to execute APE binaries within a gVisor environment, cosmocc 3.2.4 seems to work on this side. <3

[Justine Tunney]

It was possibly around then that cosmo switched from using %fs to %gs on Linux. It might be possible that there's a simple patch that can be made to libc/runtime/enable_tls.c that'll unblock you.