Issue 11615 in v8: EXC_BAD_ACCESS in Builtins_CallVarargs

[josha… via monorail]

Status: Untriaged
Owner: ----
Type: Bug

New issue 11615 by josha...@gmail.com: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615

Version: 8.8.278.14
OS: macOS
Architecture: x64

. cd v8bug.

What steps will reproduce the problem?
1. clone https://github.com/josharian/v8bug
2. cd v8bug
3. go test

What is the expected output?

PASS


What do you see instead?

A panic from inside the Go runtime. Running the same thing under GDB yields this:

* thread #14, stop reason = EXC_BAD_ACCESS (code=2, address=0x700008caf000)
* frame #0: 0x0000000004b05321 bug.report.test`Builtins_CallVarargs + 97
frame #1: 0x0000000004b0ba2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #2: 0x00001298000a903e
frame #3: 0x0000000004b0ba2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #4: 0x0000000004b0ba2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #5: 0x0000000004b0ba2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #6: 0x0000000004b0ba2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #7: 0x0000000004b0ba2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #8: 0x0000000004b0969b bug.report.test`Builtins_JSEntryTrampoline + 91
frame #9: 0x0000000004b09478 bug.report.test`Builtins_JSEntry + 120
frame #10: 0x000000000424d9cf bug.report.test`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 2863
frame #11: 0x000000000424ce86 bug.report.test`v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 230
frame #12: 0x0000000004131978 bug.report.test`v8::Script::Run(v8::Local<v8::Context>) + 456
frame #13: 0x000000000410381d bug.report.test`RunScript + 317
frame #14: 0x00000000041008ba bug.report.test`_cgo_872099a846a0_Cfunc_RunScript + 42
frame #15: 0x000000000406ea10 bug.report.test`runtime.asmcgocall + 112
frame #16: 0x00000000040fd245 bug.report.test`rogchap.com/v8go._Cfunc_RunScript + 101
frame #17: 0x00000000040fdbe7 bug.report.test`rogchap.com/v8go.(*Context).RunScript.func3 + 103
frame #18: 0x00000000040fda2a bug.report.test`rogchap.com/v8go.(*Context).RunScript + 266
frame #19: 0x00000000040ffd25 bug.report.test`bug%2ereport.(*Doc).exec + 229
frame #20: 0x00000000041002f5 bug.report.test`bug%2ereport.Do + 1045
frame #21: 0x0000000004100439 bug.report.test`bug%2ereport.TestCrash + 25
frame #22: 0x00000000040c1e50 bug.report.test`testing.tRunner + 272



Please use labels and text to provide additional information.

My apologies for not writing a straight C reproducer. If you uncomment the printlns in x.go around line 63, it'll dump all executed javascript to stdout. (Any before any of that runs, automerge.js, included in that repo, also runs.)

Thanks.

--
You received this message because:
1. The project was configured to send all issue notifications to this address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[v… via monorail]

Updates:
Cc: jgr...@chromium.org ish...@chromium.org
Components: Runtime

Comment #1 on issue 11615 by va...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c1

(No comment was entered for this change.)

[jgru… via monorail]

Comment #2 on issue 11615 by jgr...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c2

Could you check whether this still repros on a current V8 version? We've fixed a few issues related to argc in recent months.

[josha… via monorail]

Comment #3 on issue 11615 by josha...@gmail.com: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c3

I can reproduce with 8.9.255.20, which is the most recent that is conveniently available. Is that recent enough to be helpful?

The backtrace with that version is:

```
$ lldb ./bug.report.test
(lldb) target create "./bug.report.test"
Current executable set to '/Users/josh/x/v8bug/bug.report.test' (x86_64).
(lldb) r
Process 75253 launched: '/Users/josh/x/v8bug/bug.report.test' (x86_64)
Process 75253 stopped
* thread #14, stop reason = EXC_BAD_ACCESS (code=2, address=0x70000890b000)
frame #0: 0x0000000004b4b521 bug.report.test`Builtins_CallVarargs + 97
bug.report.test`Builtins_CallVarargs:
-> 0x4b4b521 <+97>: movq %r11, (%r8,%r9,8)
0x4b4b525 <+101>: incl %r9d
0x4b4b528 <+104>: jmp 0x4b4b504 ; <+68>
0x4b4b52a <+106>: addq %r9, %rax
Target 0: (bug.report.test) stopped.
(lldb) bt
* thread #14, stop reason = EXC_BAD_ACCESS (code=2, address=0x70000890b000)
* frame #0: 0x0000000004b4b521 bug.report.test`Builtins_CallVarargs + 97
frame #1: 0x0000000004b51c2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #2: 0x000009500009c3be
frame #3: 0x0000000004b51c2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #4: 0x0000000004b51c2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #5: 0x0000000004b51c2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #6: 0x0000000004b51c2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #7: 0x0000000004b51c2f bug.report.test`Builtins_InterpreterEntryTrampoline + 207
frame #8: 0x0000000004b4f89b bug.report.test`Builtins_JSEntryTrampoline + 91
frame #9: 0x0000000004b4f678 bug.report.test`Builtins_JSEntry + 120
frame #10: 0x000000000426af9a bug.report.test`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 2906
frame #11: 0x000000000426a426 bug.report.test`v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 230
frame #12: 0x0000000004146898 bug.report.test`v8::Script::Run(v8::Local<v8::Context>) + 456
```

[jgru… via monorail]

Comment #4 on issue 11615 by jgr...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c4

> I can reproduce with 8.9.255.20, which is the most recent that is conveniently available. Is that recent enough to be helpful?

Unfortunately not, development is at 92 by now and like I said, there have been multiple related fixes in this area recently. I suspect this is fixed on current V8.

With a simple d8 repro we could easily check, but with the Go-based repro it's more involved. So please verify this still occurs on current V8 git master. Alternatively, if you provide a d8 repro we can check on our side.

[josha… via monorail]

Comment #5 on issue 11615 by josha...@gmail.com: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c5

OK, I've reproduced using v8 version 9.2.15 (53d8ce6eaaa65772b7074bd390b1fcd29208f387).

I did it by following the instructions at https://github.com/rogchap/v8go#upgrading-the-v8-binaries and using a local Go module replace directive to use the newly built binaries.

Backtrace:

```
$ lldb ./bug.report.test
(lldb) target create "./bug.report.test"
Current executable set to '/Users/josh/x/v8bug/bug.report.test' (x86_64).
(lldb) r

Process 99046 launched: '/Users/josh/x/v8bug/bug.report.test' (x86_64)
using v8 version 9.2.15-v8go
Process 99046 stopped
* thread #11, stop reason = EXC_BAD_ACCESS (code=2, address=0x70001046b000)
frame #0: 0x0000000004b88781 bug.report.test`Builtins_CallVarargs + 97
bug.report.test`Builtins_CallVarargs:
-> 0x4b88781 <+97>: movq %r11, (%r8,%r9,8)
0x4b88785 <+101>: incl %r9d
0x4b88788 <+104>: jmp 0x4b88764 ; <+68>
0x4b8878a <+106>: addq %r9, %rax

Target 0: (bug.report.test) stopped.
(lldb) bt

* thread #11, stop reason = EXC_BAD_ACCESS (code=2, address=0x70001046b000)
* frame #0: 0x0000000004b88781 bug.report.test`Builtins_CallVarargs + 97
frame #1: 0x0000000004b8fba1 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #2: 0x00000f13000aa962
frame #3: 0x0000000004b8fba1 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #4: 0x0000000004b8fba1 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #5: 0x0000000004b8fba1 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #6: 0x0000000004b8fba1 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #7: 0x0000000004b8fba1 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #8: 0x0000000004b8dc5b bug.report.test`Builtins_JSEntryTrampoline + 91
frame #9: 0x0000000004b8d9e3 bug.report.test`Builtins_JSEntry + 131
frame #10: 0x000000000426a209 bug.report.test`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 2937
frame #11: 0x0000000004269679 bug.report.test`v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 233
frame #12: 0x0000000004144ae5 bug.report.test`v8::Script::Run(v8::Local<v8::Context>) + 789
```

[josha… via monorail]

Comment #6 on issue 11615 by josha...@gmail.com: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c6

I've also reproduced at current master (fd29e246f65a7cee130e72cd10f618f3b82af232).

```
$ lldb ./bug.report.test
(lldb) target create "./bug.report.test"
Current executable set to '/Users/josh/x/v8bug/bug.report.test' (x86_64).
(lldb) r

Process 3977 launched: '/Users/josh/x/v8bug/bug.report.test' (x86_64)
using v8 version 9.2.0-v8go (candidate)
Process 3977 stopped
* thread #14, stop reason = EXC_BAD_ACCESS (code=2, address=0x700003339000)
frame #0: 0x0000000004b88761 bug.report.test`Builtins_CallVarargs + 97
bug.report.test`Builtins_CallVarargs:
-> 0x4b88761 <+97>: movq %r11, (%r8,%r9,8)
0x4b88765 <+101>: incl %r9d
0x4b88768 <+104>: jmp 0x4b88744 ; <+68>
0x4b8876a <+106>: addq %r9, %rax

Target 0: (bug.report.test) stopped.
(lldb) bt

* thread #14, stop reason = EXC_BAD_ACCESS (code=2, address=0x700003339000)
* frame #0: 0x0000000004b88761 bug.report.test`Builtins_CallVarargs + 97
frame #1: 0x0000000004b8fb81 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #2: 0x00000860000b1f22
frame #3: 0x0000000004b8fb81 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #4: 0x0000000004b8fb81 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #5: 0x0000000004b8fb81 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #6: 0x0000000004b8fb81 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #7: 0x0000000004b8fb81 bug.report.test`Builtins_InterpreterEntryTrampoline + 225
frame #8: 0x0000000004b8dc3b bug.report.test`Builtins_JSEntryTrampoline + 91
frame #9: 0x0000000004b8d9c3 bug.report.test`Builtins_JSEntry + 131
frame #10: 0x000000000426a1e9 bug.report.test`v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 2937
frame #11: 0x0000000004269659 bug.report.test`v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 233
frame #12: 0x0000000004144ac5 bug.report.test`v8::Script::Run(v8::Local<v8::Context>) + 789

[jgru… via monorail]

Updates:
Labels: Priority-1
Owner: victo...@chromium.org
Status: Assigned

Comment #7 on issue 11615 by jgr...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c7

Thanks for checking! Victor, could you ptal? *Might* be related to recent argc changes.

[jgru… via monorail]

Comment #8 on issue 11615 by jgr...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c8

The crash context is

0x2e9e0004211a 5a 4d8b9d90000000 REX.W movq r11,[r13+0x90] (root (undefined_value))
0x2e9e00042121 61 4f891cc8 REX.W movq [r8+r9*8],r11
0x2e9e00042125 65 41ffc1 incl r9
0x2e9e00042128 68 ebda jmp 0x2e9e00042104 (CallVarargs)

which translates to

__ LoadRoot(value, RootIndex::kUndefinedValue);
__ bind(&push);
__ movq(Operand(dest, current, times_system_pointer_size, 0), value);

https://source.chromium.org/chromium/chromium/src/+/master:v8/src/builtins/x64/builtins-x64.cc;l=2140;drc=9438fb3fff97c803d1ead34c0e4f223db168526f

`Operand(dest, current, times_system_pointer_size, 0)` may be invalid. Or maybe OSX doesn't like accessing far from the previous rsp value (that's why the windows-specific AllocateStackSpace impl exists).

[victo… via monorail]

Comment #9 on issue 11615 by victo...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c9

Thanks @josharian.
I tried running your test case (attached) in d8, but it worked fine.

Could you please:
1. Give us the values in the registers and the stack (last ~20 words).
2. Show us how to recreate this 'bug.report.test" binary.

Thanks

Attachments:
test.js 814 KB

[josha… via monorail]

Comment #10 on issue 11615 by josha...@gmail.com: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c10

'go test -c' will spit out the bug.report.test binary. The rest of the instructions necessary should be above. (I'm happy to write out a new, complete list if necessary.)

Note that I see failures on a high percentage of runs, but not always.

Here's an lldb session with backtrace, registers, and stack: https://gist.github.com/josharian/0a3bdf5d7e0ec49bfeb1fbf0c473022b. (I always mess up whether the stack grows up or down so I dumped memory in both directions.) Gathered using v8 at fd29e246f65a7cee130e72cd10f618f3b82af232.

[Git Watcher via monorail]

Comment #11 on issue 11615 by Git Watcher: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c11

The following revision refers to this bug:
https://chromium.googlesource.com/v8/v8/+/885b1ac91ffa02adb8c9293a35d64eb97d48433e

commit 885b1ac91ffa02adb8c9293a35d64eb97d48433e
Author: Victor Gomes <victo...@chromium.org>
Date: Wed Apr 14 08:37:23 2021

[x64] Fix allocating large stack space on macOS

Similarly to Windows, on macOS we should touch the memory in a page
when allocating stack space that crosses page boundaries.

Change-Id: I8968805c4abe255123a41d0f63f89d4af509b6c8
Bug: v8:11615
Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2825588
Commit-Queue: Victor Gomes <victo...@chromium.org>
Reviewed-by: Jakob Gruber <jgr...@chromium.org>
Cr-Commit-Position: refs/heads/master@{#73948}

[modify] https://crrev.com/885b1ac91ffa02adb8c9293a35d64eb97d48433e/src/builtins/arm/builtins-arm.cc
[modify] https://crrev.com/885b1ac91ffa02adb8c9293a35d64eb97d48433e/src/builtins/arm64/builtins-arm64.cc
[modify] https://crrev.com/885b1ac91ffa02adb8c9293a35d64eb97d48433e/src/builtins/ia32/builtins-ia32.cc
[modify] https://crrev.com/885b1ac91ffa02adb8c9293a35d64eb97d48433e/src/builtins/x64/builtins-x64.cc
[modify] https://crrev.com/885b1ac91ffa02adb8c9293a35d64eb97d48433e/src/codegen/turbo-assembler.h
[modify] https://crrev.com/885b1ac91ffa02adb8c9293a35d64eb97d48433e/src/codegen/x64/macro-assembler-x64.cc
[modify] https://crrev.com/885b1ac91ffa02adb8c9293a35d64eb97d48433e/src/codegen/x64/macro-assembler-x64.h

[victo… via monorail]

Updates:
Status: Fixed

Comment #12 on issue 11615 by victo...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c12

Hi @josharian,

Please check if the fix worked for you (when it lands) and feel free to open it again otherwise.
Thanks,

[josha… via monorail]

Comment #13 on issue 11615 by josha...@gmail.com: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c13

Unfortunately, I still see exactly the same crash at 885b1ac91f.

Perhaps of note, I see other test failures too (in a different test suite), although those are presumably unrelated bugs due to running at tip.

Did you manage to reproduce locally, or was this an optimistic fix? If the latter, anything else I can do to help?

[victo… via monorail]

Updates:
Status: Started

Comment #14 on issue 11615 by victo...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c14

So, looking at the register values, they all look good. The crash seems to happen when writing to a memory between rsp and rbp with a very large number of arguments in the call (80k). So Jakob and I though about the page size issue that happens on Windows.

I did manage to reproduce the crash before the patch using Go. After the fix, I run a couple of times and it seemed to work.

I cannot get the crash on d8. I was wondering if the problem is actually in the embedder.

[josha… via monorail]

Comment #15 on issue 11615 by josha...@gmail.com: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c15

> After the fix, I run a couple of times and it seemed to work.

I'll double-check all my steps early next week and report back. Hopefully I've just done something wrong.

> I cannot get the crash on d8. I was wondering if the problem is actually in the embedder.

Maybe, although I suspect that the embedding just happens to create the right environment, maybe stack alignment or dirty pages or whatnot. Playing around a bit, the reproducer itself is somewhat sensitive to mild changes in unrelated code, which again suggests an alignment-of-the-stars situation.

[sheriffbot via monorail]

Comment #16 on issue 11615 by sheriffbot: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c16

Dear owner, please check the status of the issue and update it according to our guideline: http://go/v8-issue-guidelines

Thanks for your time! To disable nags, add the Disable-Nags label.

[victo… via monorail]

Updates:
Labels: Priority-2

Comment #17 on issue 11615 by victo...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c17

(No comment was entered for this change.)

[victo… via monorail]

Updates:
Status: Fixed

Comment #18 on issue 11615 by victo...@chromium.org: EXC_BAD_ACCESS in Builtins_CallVarargs
https://bugs.chromium.org/p/v8/issues/detail?id=11615#c18

Since josharian didn't report it back, I'll consider this fixed. Please, re-open the issue otherwise.