Strategies to debug memory alignment issues

[Mike Lischke]

Hi all,

I have a memory alignment problem in a library, which I want to use via wasm. To get more information I enabled USAN which reported the error, but it makes no sense to me, because the error is within the STL, not my code. The class I use is std::unordered_set<> with a custom hasher and a custom comparer. The pointer given to the `insert` method (the error happens already on first insertion) is aligned, I triple checked that. Here's the full stack trace:

This is the pointer that should be inserted in the set:

proposed: 0x3bebe8

/opt/homebrew/Cellar/emscripten/3.1.43/libexec/cache/sysroot/include/c++/v1/__memory/unique_ptr.h:469:12: runtime error: reference binding to misaligned address 0x3307474f for type 'std::__hash_node_base<std::__hash_node<antlr4::dfa::DFAState *, void *> *> *', which requires 4 byte alignment

0x3307474f: note: pointer points here

RuntimeError: memory access out of bounds

    at __ubsan::Diag::~Diag() (wasm://wasm/0a18e74a:wasm-function[23303]:0xa1fae2)

    at handleTypeMismatchImpl(__ubsan::TypeMismatchData*, unsigned long, __ubsan::ReportOptions) (wasm://wasm/0a18e74a:wasm-function[23315]:0xa20b40)

    at __ubsan_handle_type_mismatch_v1 (wasm://wasm/0a18e74a:wasm-function[23314]:0xa20695)

    at std::__2::unique_ptr<std::__2::__hash_node_base<std::__2::__hash_node<antlr4::dfa::DFAState*, void*>*>* [], std::__2::__bucket_list_deallocator<std::__2::allocator<std::__2::__hash_node_base<std::__2::__hash_node<antlr4::dfa::DFAState*, void*>*>*>>>::operator[][abi:v160004](unsigned long) const (wasm://wasm/0a18e74a:wasm-function[16520]:0x6dde0a)

    at std::__2::pair<std::__2::__hash_iterator<std::__2::__hash_node<antlr4::dfa::DFAState*, void*>*>, bool> std::__2::__hash_table<antlr4::dfa::DFAState*, antlr4::dfa::DFA::DFAStateHasher, antlr4::dfa::DFA::DFAStateComparer, std::__2::allocator<antlr4::dfa::DFAState*>>::__emplace_unique_key_args<antlr4::dfa::DFAState*, antlr4::dfa::DFAState* const&>(antlr4::dfa::DFAState* const&, antlr4::dfa::DFAState* const&) (wasm://wasm/0a18e74a:wasm-function[16515]:0x6dbbfb)

    at std::__2::__hash_table<antlr4::dfa::DFAState*, antlr4::dfa::DFA::DFAStateHasher, antlr4::dfa::DFA::DFAStateComparer, std::__2::allocator<antlr4::dfa::DFAState*>>::__insert_unique[abi:v160004](antlr4::dfa::DFAState* const&) (wasm://wasm/0a18e74a:wasm-function[16387]:0x6cb146)

    at std::__2::unordered_set<antlr4::dfa::DFAState*, antlr4::dfa::DFA::DFAStateHasher, antlr4::dfa::DFA::DFAStateComparer, std::__2::allocator<antlr4::dfa::DFAState*>>::insert[abi:v160004](antlr4::dfa::DFAState* const&) (wasm://wasm/0a18e74a:wasm-function[16382]:0x6ca8f2)

    at antlr4::atn::LexerATNSimulator::addDFAState(antlr4::atn::ATNConfigSet*, bool) (wasm://wasm/0a18e74a:wasm-function[16379]:0x6c9deb)

    at antlr4::atn::LexerATNSimulator::matchATN(antlr4::CharStream*) (wasm://wasm/0a18e74a:wasm-function[16306]:0x6a4858)

    at antlr4::atn::LexerATNSimulator::match(antlr4::CharStream*, unsigned long) (wasm://wasm/0a18e74a:wasm-function[16291]:0x6a095c)

To me it looks like the set tries to insert a random memory address. I added console logging in the entire path, including the hasher, to see what's going on (feeling like debugging in the 90s :-( ) and everything looks good. The hash is created and looks valid, but after that the crash happens. So what else could I do to debug further?

For completeness, this is how the set member is defined:

std::unordered_set<DFAState *, DFAStateHasher, DFAStateComparer> states; // States are owned by this class.

No unique pointers involved here, so this must be something internal to the set.

Mike

-- 
www.soft-gems.net

[Sam Clegg]

How did the issue present itself before you enabled UBSAN?   (Are you sure its the same issue?)

Is there some way you could make/share a minimal repro case?

cheers,

sam

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/ECB3B320-77B3-4B6E-A51C-8199FC547AAB%40googlemail.com.

[Tomas Rokicki]

To me this looks like a heap smash.  Try enabling the address sanitizer.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va2-eEvdH3LBcenC7bLnVWmvsHmmWo2gqnweYzdeARZG%3DZA%40mail.gmail.com.

--

-  https://cube20.org/  -  https://golly.sf.net/  - https://alpha.twizzle.net/

/ -

[Mike Lischke]

How did the issue present itself before you enabled UBSAN?   (Are you sure its the same issue?)

It's pretty much the same:

RuntimeError: memory access out of bounds

    at std::__2::pair<std::__2::__hash_iterator<std::__2::__hash_node<antlr4::dfa::DFAState*, void*>*>, bool> std::__2::__hash_table<antlr4::dfa::DFAState*, antlr4::dfa::DFA::DFAStateHasher, antlr4::dfa::DFA::DFAStateComparer, std::__2::allocator<antlr4::dfa::DFAState*>>::__emplace_unique_key_args<antlr4::dfa::DFAState*, antlr4::dfa::DFAState* const&>(antlr4::dfa::DFAState* const&, antlr4::dfa::DFAState* const&) (wasm://wasm/06ddd9e2:wasm-function[16499]:0x222e01)

    at std::__2::__hash_table<antlr4::dfa::DFAState*, antlr4::dfa::DFA::DFAStateHasher, antlr4::dfa::DFA::DFAStateComparer, std::__2::allocator<antlr4::dfa::DFAState*>>::__insert_unique[abi:v160004](antlr4::dfa::DFAState* const&) (wasm://wasm/06ddd9e2:wasm-function[16371]:0x21f157)

    at std::__2::unordered_set<antlr4::dfa::DFAState*, antlr4::dfa::DFA::DFAStateHasher, antlr4::dfa::DFA::DFAStateComparer, std::__2::allocator<antlr4::dfa::DFAState*>>::insert[abi:v160004](antlr4::dfa::DFAState* const&) (wasm://wasm/06ddd9e2:wasm-function[16366]:0x21ef8f)

    at antlr4::atn::LexerATNSimulator::addDFAState(antlr4::atn::ATNConfigSet*, bool) (wasm://wasm/06ddd9e2:wasm-function[16363]:0x21ec90)

    at antlr4::atn::LexerATNSimulator::matchATN(antlr4::CharStream*) (wasm://wasm/06ddd9e2:wasm-function[16290]:0x218e8a)

    at antlr4::atn::LexerATNSimulator::match(antlr4::CharStream*, unsigned long) (wasm://wasm/06ddd9e2:wasm-function[16275]:0x2183e1)

    at antlr4::Lexer::nextToken() (wasm://wasm/06ddd9e2:wasm-function[12053]:0x17b1bf)

    at embind_init_main()::$_6::operator()(LexerHelper&) const (wasm://wasm/06ddd9e2:wasm-function[3619]:0xa43d3)

    at embind_init_main()::$_6::__invoke(LexerHelper&) (wasm://wasm/06ddd9e2:wasm-function[3618]:0xa4362)

    at emscripten::internal::FunctionInvoker<std::__2::unique_ptr<antlr4::Token, std::__2::default_delete<antlr4::Token>> (*)(LexerHelper&), std::__2::unique_ptr<antlr4::Token, std::__2::default_delete<antlr4::Token>>, LexerHelper&>::invoke(std::__2::unique_ptr<antlr4::Token, std::__2::default_delete<antlr4::Token>> (**)(LexerHelper&), LexerHelper*) (wasm://wasm/06ddd9e2:wasm-function[3611]:0xa4111)

Is there some way you could make/share a minimal repro case?

I can try, but what I'd really need is a way to debug the C++ code. Otherwise it becomes a pretty frustrating trial and error experience, which costs a lot of time.

cheers,

sam

Thanks for trying to help Sam!

Mike
-- 
www.soft-gems.net

[Mike Lischke]

To me this looks like a heap smash. Try enabling the address sanitizer.

I tried, but that lead to this issue: https://github.com/emscripten-core/emscripten/issues/19346.

On Wed, Aug 9, 2023 at 7:13 AM 'Mike Lischke' via emscripten-discuss <emscripte...@googlegroups.com> wrote:

Hi all,

I have a memory alignment problem in a library, which I want to use via wasm. To get more information I enabled USAN which reported the error, but it makes no sense to me, because the error is within the STL, not my code. The class I use is std::unordered_set<> with a custom hasher and a custom comparer. The pointer given to the `insert` method (the error happens already on first insertion) is aligned, I triple checked that. Here's the full stack trace:

This is the pointer that should be inserted in the set:

proposed: 0x3bebe8

/opt/homebrew/Cellar/emscripten/3.1.43/libexec/cache/sysroot/include/c++/v1/__memory/unique_ptr.h:469:12: runtime error: reference binding to misaligned address 0x3307474f for type 'std::__hash_node_base<std::__hash_node<antlr4::dfa::DFAState *, void *> *> *', which requires 4 byte alignment

0x3307474f: note: pointer points here

RuntimeError: memory access out of bounds

    at __ubsan::Diag::~Diag() (wasm://wasm/0a18e74a:wasm-function[23303]:0xa1fae2)

    at handleTypeMismatchImpl(__ubsan::TypeMismatchData*, unsigned long, __ubsan::ReportOptions) (wasm://wasm/0a18e74a:wasm-function[23315]:0xa20b40)

    at __ubsan_handle_type_mismatch_v1 (wasm://wasm/0a18e74a:wasm-function[23314]:0xa20695)

    at std::__2::unique_ptr<std::__2::__hash_node_base<std::__2::__hash_node<antlr4::dfa::DFAState*, void*>*>* [], std::__2::__bucket_list_deallocator<std::__2::allocator<std::__2::__hash_node_base<std::__2::__hash_node<antlr4::dfa::DFAState*, void*>*>*>>>::operator[][abi:v160004](unsigned long) const (wasm://wasm/0a18e74a:wasm-function[16520]:0x6dde0a)

    at std::__2::pair<std::__2::__hash_iterator<std::__2::__hash_node<antlr4::dfa::DFAState*, void*>*>, bool> std::__2::__hash_table<antlr4::dfa::DFAState*, antlr4::dfa::DFA::DFAStateHasher, antlr4::dfa::DFA::DFAStateComparer, std::__2::allocator<antlr4::dfa::DFAState*>>::__emplace_unique_key_args<antlr4::dfa::DFAState*, antlr4::dfa::DFAState* const&>(antlr4::dfa::DFAState* const&, antlr4::dfa::DFAState* const&) (wasm://wasm/0a18e74a:wasm-function[16515]:0x6dbbfb)

    at std::__2::__hash_table<antlr4::dfa::DFAState*, antlr4::dfa::DFA::DFAStateHasher, antlr4::dfa::DFA::DFAStateComparer, std::__2::allocator<antlr4::dfa::DFAState*>>::__insert_unique[abi:v160004](antlr4::dfa::DFAState* const&) (wasm://wasm/0a18e74a:wasm-function[16387]:0x6cb146)

    at std::__2::unordered_set<antlr4::dfa::DFAState*, antlr4::dfa::DFA::DFAStateHasher, antlr4::dfa::DFA::DFAStateComparer, std::__2::allocator<antlr4::dfa::DFAState*>>::insert[abi:v160004](antlr4::dfa::DFAState* const&) (wasm://wasm/0a18e74a:wasm-function[16382]:0x6ca8f2)

    at antlr4::atn::LexerATNSimulator::addDFAState(antlr4::atn::ATNConfigSet*, bool) (wasm://wasm/0a18e74a:wasm-function[16379]:0x6c9deb)

    at antlr4::atn::LexerATNSimulator::matchATN(antlr4::CharStream*) (wasm://wasm/0a18e74a:wasm-function[16306]:0x6a4858)

    at antlr4::atn::LexerATNSimulator::match(antlr4::CharStream*, unsigned long) (wasm://wasm/0a18e74a:wasm-function[16291]:0x6a095c)

Thanks,

Mike

-- 
www.soft-gems.net

[Mike Lischke]

To me this looks like a heap smash.  Try enabling the address sanitizer.

I finally was able to enable ASAN. With the help of Sam Clegg I found why I got that "too many locals" error and once I fixed that I got a much better error report (use-after-free), with which I can continue to isolate the problem.

In case someone is interested: don't make your EMSCRIPTEN_BINDINGS call too large. I had bound 53 classes in a single call, some with a large list of members. Once I split the single call up into multiple ones, things started to work much better.

Thanks,

Mike
-- 
www.soft-gems.net

[Sam Clegg]

If you need real debugging, have you tried the chrome DWARF debugger extension: https://developer.chrome.com/blog/wasm-debugging-2020/?   As far as I know that is the only way to get a real debugging experience with webassembly on the web today.   

 

cheers,

sam

Thanks for trying to help Sam!

Mike
-- 
www.soft-gems.net

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/5C24A416-9A4C-48BA-9871-4E10B8508A4E%40googlemail.com.

[Mike Lischke]

Is there some way you could make/share a minimal repro case?

I can try, but what I'd really need is a way to debug the C++ code. Otherwise it becomes a pretty frustrating trial and error experience, which costs a lot of time.

If you need real debugging, have you tried the chrome DWARF debugger extension: https://developer.chrome.com/blog/wasm-debugging-2020/?   As far as I know that is the only way to get a real debugging experience with webassembly on the web today.   

 

This is probably one of my next steps, though this is really a Node.js, not a web app, which means I have to strip out the Node.js stuff for debugging. Weird that there's no wasm debugger for VS Code yet. There are a number of wasm extensions, but they all more or less provide wasm syntax highlighting, nothing else. Maybe at some later time...

Mike
-- 
www.soft-gems.net

[Floh]

LLTP, but VSCode now has WASM DWARF debugging support, and I just confirmed that it works for Emscripten's node.js output.

First install this VSCode extension:

https://marketplace.visualstudio.com/items?itemName=ms-vscode.wasm-dwarf-debugging

...then...

emcc hello.c -o hello.js -g

...open the directory in VSCode, open hello.js, set a breakpoint at a point where the WASM has been loaded (e.g. in the JS function "callMain()").

Press F5 to run hello.js in the debugger, it should stop at that breakpoint in callMain().

Next open hello.c, set a breakpoint in the C code. Continue debugging. It should stop now at the breakpoint in the C code.

Voila :)

...with a bit more tinkering this even works with WASM running in Chrome :)