Debugging multi-threaded segfault in Ruby V8 host

[Sam Saffron]

Hi There,

I am a maintainer of https://github.com/rubyjs/mini_racer , V8 binding for Ruby.

Recently after upgrading to 12.4.254.21 we started seeing segfaults per:

https://github.com/rubyjs/mini_racer/issues/300

After... a rather giant adventure I manager to create a repro of the issue by that was somewhat debugguable.

When the segfault happens GDB says it is deep inside some interpreter tampoline

#131 0x00007e79de5b565e in Builtins_InterpreterEntryTrampoline () from /home/sam/Source/mini_racer/lib/mini_racer_extension.so                  try=0x7ffc85d0b210, nfds=nfds@entry=2,
#132 0x000037edcd572331 in ?? ()
#133 0x00000ede17b61561 in ?? ()
#134 0x00000ede17b55259 in ?? ()
#135 0x00000ede17b61561 in ?? ()
#136 0x000037edcd572331 in ?? ()
#137 0x0000135dd0aa93e1 in ?? ()
#138 0x000010765a0c0069 in ?? ()
#139 0x0000004a00000000 in ?? ()
#140 0x000016bc18c16201 in ?? ()
#141 0x0000000000000002 in ?? ()
#142 0x00000ede17b61931 in ?? ()
#143 0x00000ede17b61901 in ?? ()
#144 0x00007e79cfcfe778 in ?? ()
#145 0x00007e79de5b565e in Builtins_InterpreterEntryTrampoline () from /home/sam/Source/mini_racer/lib/mini_racer_extension.so
#146 0x00000ede17b61969 in ?? ()
#147 0x00000ede17b55259 in ?? ()
#148 0x00000ede17b5f329 in ?? ()
#149 0x00000ede17b55259 in ?? ()
#150 0x00000ede17b61969 in ?? ()
#151 0x00000ede17b61931 in ?? ()
#152 0x000010765a0c0069 in ?? ()
#153 0x000010765a0c0069 in ?? ()
#154 0x0000014400000000 in ?? ()
#155 0x000016bc18c14fa1 in ?? ()
#156 0x0000000000000002 in ?? ()
#157 0x000018d80e8ccd69 in ?? ()
#158 0x000020b3bd980109 in ?? ()
#159 0x00007e79cfcfe7a8 in ?? ()
#160 0x00007e79de5b325c in Builtins_JSEntryTrampoline () from /home/sam/Source/mini_racer/lib/mini_racer_extension.so                           _racer_extension.so
#161 0x000020b3bd980059 in ?? ()
#162 0x000010765a0c0c79 in ?? ()
#163 0x000018d80e8ccd69 in ?? ()
#164 0x000000000000002c in ?? ()
#165 0x00007e79cfcfe820 in ?? ()
#166 0x00007e79de5b2f9b in Builtins_JSEntry ()

To reproduce I have 2 Ruby threads kick off a bunch of JavaScript on a single contex.

All access to ->Run() is guarded with Locker, full source is at:

https://github.com/rubyjs/mini_racer/blob/standalone/ext/mini_racer_extension/mini_racer_extension.cc

And crash happens at:

https://github.com/rubyjs/mini_racer/blob/standalone/ext/mini_racer_extension/mini_racer_extension.cc#L546

This is guarded here:

https://github.com/rubyjs/mini_racer/blob/standalone/ext/mini_racer_extension/mini_racer_extension.cc#L1148

My questions:

1. Are we building v8 correctly? to eliminate variables I am using this very particular build to https://github.com/rubyjs/mini_racer/blob/standalone/docker/Dockerfile, are there any flags we should not be using or should be using? (this is used to build the monolith)

2. Should I simply avoid mulithreading and instead queue all work to a single thread per Isolate/Context?

3. Are there any tips and tricks to debugging this, from what I can tell clang builds of both my extension and libv8_monolith.a are the only way I can get symbols going

4. For some reason if I build with `is_debug` as soon as I kick off an isolate I get warned about stack smashing, is this normal? Do "is_debug" builds work for people?

5. Are there any experts on this list that would be interested in some consulting work to help resolve this issue?

Any other ideas?

As it stands we are now stuck on a 1-2 year old version of v8 due to this crash and I worry that about having such an old version of v8 out there in wide adoption.

Thanks heaps

Sam

[Ben Noordhuis]

This is just from eyeballing mini_racer_extension.cc for 2 minutes but
you store v8::Locals in EvalParams, then exit the HandleScope (and
IsolateScope, and Locker) those values were created in. That's
essentially a use-after-free because they become unrooted and may be
moved or collected by the GC.

I run a consultancy business around V8 and Node.js, happy to help out.

[Jakob Kummerow]

Just to quickly comment on these two points:

 

> 3. Are there any tips and tricks to debugging this, from what I can tell clang builds of both my extension and libv8_monolith.a are the only way I can get symbols going

Use an "is_debug" build :-)

 

> 4. For some reason if I build with `is_debug` as soon as I kick off an isolate I get warned about stack smashing, is this normal? Do "is_debug" builds work for people?

That's not normal, "is_debug" builds should absolutely work, I use them all the time. They're also much nicer to debug than Release builds. Sometimes they even give you explicit warnings telling you exactly what is going wrong.

[Ben Noordhuis]

On Fri, Aug 9, 2024 at 9:02 AM Sam Saffron <sam.s...@gmail.com> wrote:

> 4. For some reason if I build with `is_debug` as soon as I kick off an isolate I get warned about stack smashing, is this normal? Do "is_debug" builds work for people?

For posterity, this turned out to be a mismatch between the V8 build
and the consumer of V8's API headers. Both need to be compiled with
V8_ENABLE_CHECKS defined; it affects object layouts.

(Nitty gritty: v8::HandleScope has an extra field in debug mode that
its constructor initializes. The consumer didn't allocate enough stack
space.)

[Jakob Kummerow]

Thanks for reporting back!

If you're so inclined, feel free to submit a patch that detects this problem and explicitly warns about it. Grep for "build configuration mismatch" (you'll find hits in api.cc) to find existing examples where we do this.

--

[Ben Noordhuis]

That's a good idea, I'll do that.

To close the loop on Sam's original report, I tracked it down to
unfortunate interaction between threads and Linux memory protection
keys. Full bug report here:
https://issues.chromium.org/issues/360909072

> --
> --
> v8-users mailing list
> v8-u...@googlegroups.com
> http://groups.google.com/group/v8-users
> ---
> You received this message because you are subscribed to the Google Groups "v8-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to v8-users+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/v8-users/CAKSzg3TVTUgXsdtm6n6PU0_puUJ%3D5QKkVq1OTV4ZvSF_cKSEzA%40mail.gmail.com.

[Ben Noordhuis]

Somewhat belated: https://chromium-review.googlesource.com/c/v8/v8/+/5878258

I've picked you as a reviewer, Jakob. Hope you don't mind.

On Fri, Aug 16, 2024 at 11:37 AM Jakob Kummerow <jkum...@chromium.org> wrote:
>