Running in thread sanitizer gives Segmentation fault for protobuf c++ v27.3

[Sundar Swamy]

Platform: Mac OS,. 10.15 & Mac OS 11.x ( arm64)

Compiler: AppleClang 12.0.0.12000032

Mac OSX SDK: 11

I link libprotouf.a to my shared libary (dylib). The dylib is compiled using -fsanitize=thread -O1 but protobuf is compiled as normal ( without any sanitizer flags )

When i run in thread sanitizer, i get SEGV and here is a sample callstack

[2024-08-13T17:26:21.490Z] ThreadSanitizer:DEADLYSIGNAL [2024-08-13T17:26:21.490Z] ==1855==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0001021e9840 bp 0x7e8000186920 sp 0x7e8000186900 T2629743) [2024-08-13T17:26:21.490Z] ==1855==The signal is caused by a READ memory access. [2024-08-13T17:26:21.490Z] ==1855==Hint: address points to the zero page.

...

...

 #0 google::protobuf::internal::UntypedMapBase::TableEntryIsNonEmptyList(unsigned int) const map.h:630 (mySample.9.1.2.17.dylib:x86_64+0xe7840)

[2024-08-13T17:26:23.250Z]     #1 google::protobuf::internal::KeyMapBase<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::FindHelper(absl::lts_20240116::string_view, absl::lts_20240116::container_internal::btree_iterator<absl::lts_20240116::container_internal::btree_node<absl::lts_20240116::container_internal::map_params<google::protobuf::internal::VariantKey, google::protobuf::internal::NodeBase*, std::__1::less<google::protobuf::internal::VariantKey>, google::protobuf::internal::MapAllocator<std::__1::pair<google::protobuf::internal::VariantKey const, google::protobuf::internal::NodeBase*> >, 256, false> >, std::__1::pair<google::protobuf::internal::VariantKey const, google::protobuf::internal::NodeBase*>&, std::__1::pair<google::protobuf::internal::VariantKey const, google::protobuf::internal::NodeBase*>*>*) const map.h:953 (mySample.9.1.2.17.dylib:x86_64+0xe7594)

[2024-08-13T17:26:23.251Z]     #2 std::__1::pair<google::protobuf::Map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::iterator, bool> google::protobuf::Map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::TryEmplaceInternal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) map.h:1619 (mySample.9.1.2.17.dylib:x86_64+0xea059)

[2024-08-13T17:26:23.251Z]     #3 std::__1::pair<google::protobuf::Map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::iterator, bool> google::protobuf::Map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::ArenaAwareTryEmplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::integral_constant<bool, false>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) map.h:1692 (mySample.9.1.2.17.dylib:x86_64+0xe9f25)

[2024-08-13T17:26:23.251Z]     #4 std::__1::pair<google::protobuf::Map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::iterator, bool> google::protobuf::Map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::try_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) map.h:1466 (mySample.9.1.2.17.dylib:x86_64+0xe9e35)

[2024-08-13T17:26:23.252Z]     #5 std::__1::pair<google::protobuf::Map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::iterator, bool> google::protobuf::Map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::insert<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, 0>(std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&&) map.h:1476 (mySample.9.1.2.17.dylib:x86_64+0xb5643)

….

….

…

…

2024-08-13T17:26:23.253Z]     #18 MySDK::CMySDKSocket::TestMonitorThread() MySDKSocket.cpp:191 (mySample.9.1.2.17.dylib:x86_64+0x1a0b5f)

[2024-08-13T17:26:23.253Z]     #19 decltype(*(std::__1::forward<MySDK::CMySDKSocket*>(fp0)).*fp()) std::__1::__invoke<void (MySDK::CMySDKSocket::*)(), MySDK::CMySDKSocket*, void>(void (MySDK::CMySDKSocket::*&&)(), MySDK::CMySDKSocket*&&) type_traits:3486 (mySample.9.1.2.17.dylib:x86_64+0x1ac338)

[2024-08-13T17:26:23.253Z]     #20 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (MySDK::CMySDKSocket::*)(), MySDK::CMySDKSocket*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (MySDK::CMySDKSocket::*)(), MySDK::CMySDKSocket*>&, std::__1::__tuple_indices<2ul>) thread:273 (mySample.9.1.2.17.dylib:x86_64+0x1ac122)

[2024-08-13T17:26:23.254Z]     #21 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (MySDK::CMySDKSocket::*)(), MySDK::CMySDKSocket*> >(void*) thread:284 (mySample.9.1.2.17.dylib:x86_64+0x1ab37b)

[2024-08-13T17:26:23.254Z]     #22 __tsan_thread_start_func <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2d89b)

[2024-08-13T17:26:23.254Z] waitForEchoResponse: 19

    #23 _pthread_start <null>:2 (libsystem_pthread.dylib:x86_64+0x6108)

[2024-08-13T17:26:23.254Z]     #24 thread_start <null>:2 (libsystem_pthread.dylib:x86_64+0x1b8a)

[2024-08-13T17:26:23.254Z] 

[2024-08-13T17:26:23.254Z] ==1855==Register values:

[2024-08-13T17:26:23.254Z] rax = 0x000003ffffffffff  rbx = 0x0000000000000000  rcx = 0x0000000003390000  rdx = 0x0000600003390000  

[2024-08-13T17:26:23.254Z] rdi = 0x0000000105730000  rsi = 0x00000001021e9840  rbp = 0x00007e8000186920  rsp = 0x00007e8000186900  

[2024-08-13T17:26:23.254Z]  r8 = 0x0000000000000000   r9 = 0x0000000000000000  r10 = 0x0000000000000000  r11 = 0x0000000000000000  

[2024-08-13T17:26:23.254Z] r12 = 0x00007b3400006830  r13 = 0x00007e8000186ab8  r14 = 0x0000000000000000  r15 = 0x0000000000000000  

[2024-08-13T17:26:23.254Z] ThreadSanitizer can not provide additional info.

[2024-08-13T17:26:23.254Z] SUMMARY: ThreadSanitizer: SEGV map.h:630 in google::protobuf::internal::UntypedMapBase::TableEntryIsNonEmptyList(unsigned int) const

[2024-08-13T17:26:23.254Z] ==1855==ABORTING

What could be causing this issue? we never had this issue with our previous protobuf version compiled the same way 3.21.12 using the same clang compiler version as universal binary. Any advise?

[Sundar Swamy]

Any idea where i could look for some clues as what's causing this crash? 

The code is trying to insert a pair in to ::google::protobuf::Map< ::std::string, ::std::string >

[Samuel Benzaquen]

On Tue, Aug 13, 2024 at 3:57 PM Sundar Swamy <sundar...@gmail.com> wrote:

Platform: Mac OS,. 10.15 & Mac OS 11.x ( arm64)

Compiler: AppleClang 12.0.0.12000032

Mac OSX SDK: 11

I link libprotouf.a to my shared libary (dylib). The dylib is compiled using -fsanitize=thread -O1 but protobuf is compiled as normal ( without any sanitizer flags )

This is the problem.

Turning on sanitizers only in part of your binary causes ODR violations.

https://en.cppreference.com/w/cpp/language/definition

Many parts of protobuf have special logic for sanitizer support, including some that change the ABI of the library.

Even if it didn't change the ABI, mixing sanitized and non-sanitized libraries will lead to false positives/negatives. Sanitizers that use shadow memory (ASan, MSan, TSan) need all the access to memory to properly update the shadow memory.

For example, if you have part of your binary without TSan instrumentation it might miss a happens-before relationship and give false positives.

 

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/bcc31496-50bd-419d-bdd7-cac894796cfbn%40googlegroups.com.

[Sundar Swamy]

Thanks @samuel for the clarification. just curious and a naive question that up until v25.3 we didn't have to compile protobuf with sanitizer.. we started to see the TSAN failure with that mentioned stack only from v25.4 release.??

[Samuel Benzaquen]

On Wed, Oct 30, 2024 at 5:30 PM Sundar Swamy <sundar...@gmail.com> wrote:

Thanks @samuel for the clarification. just curious and a naive question that up until v25.3 we didn't have to compile protobuf with sanitizer.. we started to see the TSAN failure with that mentioned stack only from v25.4 release.??

Not sure if there was a particular change between those two releases.

But the ODR violation is undefined behavior, so "it's working" does not necessarily mean that it is correct.

Undefined behavior might happen to work at some point and then fail under different conditions.

 

To view this discussion visit https://groups.google.com/d/msgid/protobuf/79e9fa14-872e-4efe-adbb-4264ee1b429bn%40googlegroups.com.