Debugging mksnapshot bus error

[Ryan Manuel]

Hi all, I'm looking into a mksnapshot crash which I logged here: https://issues.chromium.org/issues/42203948

I haven't gotten much traction on the issue itself, but I've been able to identify the commit that introduced the bug: https://chromium.googlesource.com/v8/v8.git/+/91637c25fcdb199526a7060ceca858aeef16bd3d

I'm open to working on the issue if I can, but I could use some advice debugging the issue. What advice would you give to try and track down where the issue is?

I'm new to v8 development and would appreciate any help.

[Ben Noordhuis]

Are you sure you're linking to the right issue/commit? That issue was
fixed over a year ago.

[Ryan Manuel]

Oops sorry about that. I meant this issue: https://issues.chromium.org/issues/345280736

[Ryan Manuel]

And the commit where I started seeing the problem was here: https://github.com/v8/v8/commit/8cd1e89fae9005237bba014fad72bddc0434394b

[Ben Noordhuis]

On Fri, Aug 2, 2024 at 4:17 PM 'Ryan Manuel' via v8-users
<v8-u...@googlegroups.com> wrote:
>
> Oops sorry about that. I meant this issue: https://issues.chromium.org/issues/345280736

Does the problem manifest with a debug build? What error message, if
any, do you get?

Debug builds have lots of additional checks enabled and often catch
bugs that show up as crashes in release builds.

[Ryan Manuel]

Ah thanks for the advice. I put the output below of running the command below. Any thoughts on what to do next?

#
# Fatal error in ../../src/common/ptr-compr-inl.h, line 81
# Debug check failed: (tagged & kPtrComprCageBaseMask) == base() || ((static_cast<i::Tagged_t>(tagged) & ::i::kSmiTagMask) == ::i::kSmiTag).
#
#
#
#FailureMessage Object: 0x16fc594b8

==== C stack trace ===============================

0 libv8_libbase.dylib 0x00000001152776ec v8::base::debug::StackTrace::StackTrace() + 32
1 libv8_libbase.dylib 0x0000000115277728 v8::base::debug::StackTrace::StackTrace() + 28
2 libv8_libplatform.dylib 0x0000000115495fe4 v8::platform::(anonymous namespace)::PrintStackTrace() + 60
3 libv8_libbase.dylib 0x000000011524327c V8_Fatal(char const*, int, char const*, ...) + 352
4 libv8_libbase.dylib 0x0000000115242c2c v8::base::SetFatalFunction(void (*)(char const*, int, char const*)) + 0
5 libv8_libbase.dylib 0x0000000115243388 V8_Dcheck(char const*, int, char const*) + 108
6 mksnapshot 0x00000001001bd720 v8::internal::V8HeapCompressionSchemeImpl<v8::internal::MainCage>::CompressObject(unsigned long) + 124
7 mksnapshot 0x00000001005688c4 v8::internal::TaggedField<v8::internal::MaybeWeak<v8::internal::Object>, 0, v8::internal::V8HeapCompressionSchemeImpl<v8::internal::MainCage>>::full_to_tagged(unsigned long) + 24
8 mksnapshot 0x00000001005686a8 v8::internal::TaggedField<v8::internal::MaybeWeak<v8::internal::Object>, 0, v8::internal::V8HeapCompressionSchemeImpl<v8::internal::MainCage>>::Relaxed_Store(v8::internal::Tagged<v8::internal::HeapObject>, int, v8::internal::Tagged<v8::internal::MaybeWeak<v8::internal::Object>>) + 216
9 mksnapshot 0x00000001005684c8 v8::internal::TorqueGeneratedFeedbackVector<v8::internal::FeedbackVector, v8::internal::HeapObject>::set_raw_feedback_slots(int, v8::internal::Tagged<v8::internal::Union<v8::internal::HeapObject, v8::internal::MaybeWeak<v8::internal::Object>, v8::internal::Smi>>, v8::internal::WriteBarrierMode) + 648
10 mksnapshot 0x0000000100567184 v8::internal::FeedbackVector::Set(v8::internal::FeedbackSlot, v8::internal::Tagged<v8::internal::MaybeWeak<v8::internal::Object>>, v8::internal::WriteBarrierMode) + 180
11 mksnapshot 0x00000001012f702c v8::internal::NexusConfig::SetFeedbackPair(v8::internal::Tagged<v8::internal::FeedbackVector>, v8::internal::FeedbackSlot, v8::internal::Tagged<v8::internal::MaybeWeak<v8::internal::Object>>, v8::internal::WriteBarrierMode, v8::internal::Tagged<v8::internal::MaybeWeak<v8::internal::Object>>, v8::internal::WriteBarrierMode) const + 484
12 mksnapshot 0x00000001012fa8ac void v8::internal::FeedbackNexus::SetFeedback<v8::internal::MaybeWeak<v8::internal::Map>, v8::internal::MaybeWeak<v8::internal::Object>>(v8::internal::Tagged<v8::internal::MaybeWeak<v8::internal::Map>>, v8::internal::WriteBarrierMode, v8::internal::Tagged<v8::internal::MaybeWeak<v8::internal::Object>>, v8::internal::WriteBarrierMode) + 160
13 mksnapshot 0x00000001012fb470 v8::internal::FeedbackNexus::ConfigureMonomorphic(v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::MaybeObjectHandle const&) + 324
14 mksnapshot 0x0000000100f28e1c v8::internal::IC::ConfigureVectorState(v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::MaybeObjectHandle const&) + 160
15 mksnapshot 0x0000000100f28d50 v8::internal::IC::ConfigureVectorState(v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Object>) + 112
16 mksnapshot 0x0000000100f38f04 v8::internal::KeyedStoreIC::UpdateStoreElement(v8::internal::Handle<v8::internal::Map>, v8::internal::KeyedAccessStoreMode, v8::internal::Handle<v8::internal::Map>) + 416
17 mksnapshot 0x0000000100f3c49c v8::internal::StoreInArrayLiteralIC::Store(v8::internal::Handle<v8::internal::JSArray>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) + 1176
18 mksnapshot 0x0000000100f44c24 v8::internal::__RT_impl_Runtime_StoreInArrayLiteralIC_Miss(v8::internal::Arguments<(v8::internal::ArgumentsType)0>, v8::internal::Isolate*) + 1012
19 mksnapshot 0x0000000100f445dc v8::internal::Runtime_StoreInArrayLiteralIC_Miss(int, unsigned long*, v8::internal::Isolate*) + 288
20 ??? 0x0000391473cfc750 0x0 + 62760005125968
21 ??? 0x0000000170e98ec0 0x0 + 6189321920
22 ??? 0x0000391473a7031c 0x0 + 62760002454300
23 ??? 0x0000391473a7031c 0x0 + 62760002454300
24 ??? 0x0000391473a7031c 0x0 + 62760002454300
25 ??? 0x0000391473a7031c 0x0 + 62760002454300
26 ??? 0x0000391473a7031c 0x0 + 62760002454300
27 ??? 0x0000391473a6a098 0x0 + 62760002429080
28 ??? 0x0000391473a69cd4 0x0 + 62760002428116
29 mksnapshot 0x00000001008ff004 v8::internal::GeneratedCode<unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, long, unsigned long**>::Call(unsigned long, unsigned long, unsigned long, unsigned long, long, unsigned long**) + 76
30 mksnapshot 0x00000001008fc2e4 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) + 3760
31 mksnapshot 0x00000001008fc964 v8::internal::Execution::CallScript(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) + 372
32 mksnapshot 0x0000000100210d4c v8::Script::Run(v8::Local<v8::Context>, v8::Local<v8::Data>) + 1100
33 mksnapshot 0x00000001002108c4 v8::Script::Run(v8::Local<v8::Context>) + 80
34 mksnapshot 0x0000000101aafe58 v8::internal::(anonymous namespace)::RunExtraCode(v8::Isolate*, v8::Local<v8::Context>, char const*, char const*) + 640
35 mksnapshot 0x0000000101aafa8c v8::internal::CreateSnapshotDataBlobInternal(v8::SnapshotCreator::FunctionCodeHandling, char const*, v8::SnapshotCreator&, v8::base::Flags<v8::internal::Snapshot::SerializerFlag, int, int>) + 348
36 mksnapshot 0x000000010019e33c (anonymous namespace)::CreateSnapshotDataBlob(v8::SnapshotCreator&, char const*) + 128
37 mksnapshot 0x000000010019da00 main + 1340
38 dyld 0x00000001899320e0 start + 2360

[Ben Noordhuis]

Maybe check if a v8_enable_pointer_compression=false build works okay?
It only happens with large snapshots, right? Comrpessed pointers don't
work for heap sizes > 4 GB.

On Tue, Aug 6, 2024 at 6:45 AM 'Ryan Manuel' via v8-users

[Ryan Manuel]

That did resolve the issue, but I'm not 100% positive it's a heap size issue. I was able to whittle my snapshot file down to a much smaller subset (55.1 MB down to 83 KB for the input file) and it still causes the same problem. The updated file is in this gist. Is there a way I could validate that it is the heap size problem vs. something else? I’m going to keep trying to see if I can figure out a smaller reproduction.

[Ryan Manuel]

I was able to simplify the code even more and still recreate the error. I've pasted it below.

The weird thing is if I change this line:

applyDefs.definitions[modulePath].apply(mod.exports, [mod, applyDefs])

to

applyDefs.definitions[modulePath].call(mod.exports, mod, applyDefs)

Everything works properly.

let defs = {}

defs['nat-mods.json'] = function (mod, applyDefinition) {

mod.exports = {}

}

defs['plugins.json'] = function (mod, applyDefinition) {

mod.exports = {}

}

defs['nat-mods.js'] = function (mod, applyDefinition) {

mod.exports = applyDefinition('nat-mods.json')

}

defs['plugins.js'] = function (mod, applyDefinition) {

mod.exports = applyDefinition('plugins.json')

}

defs['index.js'] = function (mod, applyDefinition) {

mod.exports = function gensync () {}

}

defs['async'] = function (mod, applyDefinition) {

function gensync () {

return applyDefinition('index.js')

}

gensync()()

}

defs['utils'] = function (mod, applyDefinition) {

}

defs['caching.js'] = function (mod, applyDefinition) {

applyDefinition('async')

applyDefinition('utils')

}

defs['import.cjs'] = function (mod, applyDefinition) {

}

defs['firstPath'] = function (mod, applyDefinition) {

applyDefinition('plugins.json')

applyDefinition('nat-mods.js')

applyDefinition('plugins.js')

applyDefinition('caching.js')

applyDefinition('import.cjs')

}

applyDefs.definitions = defs

applyDefs.exports = {}

function applyDefs (modulePath) {

const mod = {

exports: {},

children: [],

loaded: true,

parent: {},

paths: [],

require: applyDefs,

filename: modulePath,

id: modulePath,

path: modulePath,

}

applyDefs.exports[modulePath] = mod

applyDefs.definitions[modulePath].apply(mod.exports, [mod, applyDefs])

return mod.exports

}

applyDefs('firstPath')

[Ryan Manuel]

Any further thoughts on this? It definitely seems like a bug in V8 to me now. Switching between a call to `fn.call` vs. a call to `fn.apply` should work. I'd be willing to try my hand at resolving the issue, but I'm not really sure where to start.