does gRPC play nice with ASAN?

[mandrews]

I'm new to ASAN and ran into a situation running C++ program after compiling in -fsanitize=address to investigate a segfault that happens after running an hour or so.

compiling a x64 C++17 with gcc 8.4.0 under ubuntu 18.04.1

The program doesn't actually try to allocate 1.8 petabytes, as it starts up and runs properly on my desktop computer for a while without ASAN compiled in. With ASAN, i get the stack below immediately on startup:

Any known issues with gRPC?  I realize this is not much context at all, but would appreciate suggestions on how to proceed at this point.

I might add that my first try was to run the program under valgrind to see if i could catch the segfault, but valgrind exits with an unknown instruction while executing the code prior to entering main()

==44393==WARNING: AddressSanitizer failed to allocate 0x676af86372000 bytes
==44393==AddressSanitizer's allocator is terminating the process instead of returning 0
==44393==If you don't like this behavior set allocator_may_return_null=1
==44393==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_allocator.cc:216 "((0)) != (0)" (0x0, 0x0)
    #0 0x7f48dbca3522  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xf9522)
    #1 0x7f48dbcc00e9 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x1160e9)
    #2 0x7f48dbca9456  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xff456)
    #3 0x7f48dbca9493  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xff493)
    #4 0x7f48dbbd2595  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x28595)
    #5 0x7f48dbbd0fa6  (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x26fa6)
    #6 0x7f48dbc97f5a in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xedf5a)
    #7 0x7f48db4c5dfd in gpr_malloc src/core/lib/gpr/alloc.cc:32

    #8 0x559e8e36cf1b in grpc::internal::FillMetadataArray(std::multimap<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, unsigned long*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /var/dev/github/sdf/modules/analytics-flow-ingestion/flow_ingester/../../../../cpp-third-party/grpc/include/grpcpp/impl/codegen/call_op_set.h:61
    #9 0x559e8e374aa8 in grpc::internal::CallOpSendInitialMetadata::AddOp(grpc_op*, unsigned long*) /var/dev/github/sdf/modules/analytics-flow-ingestion/flow_ingester/../../../../cpp-third-party/grpc/include/grpcpp/impl/codegen/call_op_set.h:243
   
 #10 0x559e8e4c259b in grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpRecvInitialMetadata, grpc::internal::CallOpRecvMessage<liveaction::analytics::node::grpc::application_id_mapper::v1::ApplicationIdMapping>, grpc::internal::CallOpClientSendClose, grpc::internal::CallOpClientRecvStatus>::ContinueFillOpsAfterInterception() /var/dev/github/sdf/modules/analytics-flow-ingestion/flow_ingester/../../../../cpp-third-party/grpc/include/grpcpp/impl/codegen/call_op_set.h:931
   
 #11 0x559e8e4d562b in grpc::internal::Call::PerformOps(grpc::internal::CallOpSetInterface*) /var/dev/github/sdf/modules/analytics-flow-ingestion/flow_ingester/../../../../cpp-third-party/grpc/include/grpcpp/impl/codegen/call.h:69
   
 #12 0x559e8e4d562b in grpc::internal::BlockingUnaryCallImpl<liveaction::analytics::node::grpc::application_id_mapper::v1::GetOrCreateApplicationIdMappingRequest, liveaction::analytics::node::grpc::application_id_mapper::v1::ApplicationIdMapping>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc_impl::ClientContext*, liveaction::analytics::node::grpc::application_id_mapper::v1::GetOrCreateApplicationIdMappingRequest const&, liveaction::analytics::node::grpc::application_id_mapper::v1::ApplicationIdMapping*) /var/dev/github/sdf/modules/analytics-flow-ingestion/flow_ingester/../../../../cpp-third-party/grpc/include/grpcpp/impl/codegen/client_unary_call.h:71
   
 #13 0x559e8e49ef7c in grpc::Status grpc::internal::BlockingUnaryCall<liveaction::analytics::node::grpc::application_id_mapper::v1::GetOrCreateApplicationIdMappingRequest, liveaction::analytics::node::grpc::application_id_mapper::v1::ApplicationIdMapping>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc_impl::ClientContext*, liveaction::analytics::node::grpc::application_id_mapper::v1::GetOrCreateApplicationIdMappingRequest const&, liveaction::analytics::node::grpc::application_id_mapper::v1::ApplicationIdMapping*) /var/dev/github/sdf/modules/analytics-flow-ingestion/flow_ingester/../../../../cpp-third-party/grpc/include/grpcpp/impl/codegen/client_unary_call.h:43
   
 #14 0x559e8e49ef7c in liveaction::analytics::node::grpc::application_id_mapper::v1::ApplicationIdMapper::Stub::GetOrCreateApplicationIdMapping(grpc_impl::ClientContext*, liveaction::analytics::node::grpc::application_id_mapper::v1::GetOrCreateApplicationIdMappingRequest const&, liveaction::analytics::node::grpc::application_id_mapper::v1::ApplicationIdMapping*) /var/dev/github/sdf/modules/analytics-flow-ingestion/flow_ingester/../grpc/linux/liveaction/analytics/node/grpc/application_id_mapper_v1.grpc.pb.cc:137


[gautam tadavarthi]

Facing the same issue with grpc 1.25.0, any solution for this ?

[Dan Liew]

The output from the original poster is **not** a seg-fault. It's ASan
catching the fact that a huge allocation is being requested and by
default ASan halts when it detects this. You can disable this behavior
by setting "ASAN_OPTIONS=allocator_may_return_null=1" in your
environment. If you set this the allocation will return a nullptr.

It's up to gRPC to handle the return of the nullptr. It may be the
case that gRPC doesn't handle it gracefully, in which case my
suggestion won't really fix your problem, it will just have moved it
somewhere else.

Thanks,
Dan.

> --
> You received this message because you are subscribed to the Google Groups "address-sanitizer" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to address-saniti...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/address-sanitizer/41b69de0-6933-411b-9095-f4fbcbc89aben%40googlegroups.com.