grpc crash during streaming call

[Pragadeesh nagaraj]

I am running grpc on a low resource system. grpc server is streaming messages to client, after sometime grpc is crashing.

Please help me understand the behavior.

bt for the same:

Program terminated with signal SIGABRT, Aborted.

#0  0x00007fcf5ad9800b in raise () from /lib/x86_64-linux-gnu/libc.so.6

[Current thread is 1 (Thread 0x7fcefa7fc700 (LWP 583))]

#0  0x00007fcf5ad9800b in raise () from /lib/x86_64-linux-gnu/libc.so.6

#1  0x00007fcf5ad77859 in abort () from /lib/x86_64-linux-gnu/libc.so.6

#2  0x00007fcf5ade226e in ?? () from /lib/x86_64-linux-gnu/libc.so.6

#3  0x00007fcf5adea2fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6

#4  0x00007fcf5adea96b in ?? () from /lib/x86_64-linux-gnu/libc.so.6

#5  0x00007fcf5adeaaaf in ?? () from /lib/x86_64-linux-gnu/libc.so.6

#6  0x00007fcf5adecc83 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

#7  0x00007fcf5adef299 in malloc () from /lib/x86_64-linux-gnu/libc.so.6

#8  0x00007fcf5a438043 in gpr_malloc () from /usr/local/lib/libgpr.so.10

#9  0x00007fcf5a990069 in ru_slice_create(grpc_resource_user*, unsigned long) () from /usr/local/lib/libgrpc.so.10

#10 0x00007fcf5a990840 in ru_alloc_slices(grpc_resource_user_slice_allocator*) () from /usr/local/lib/libgrpc.so.10

#11 0x00007fcf5a991edb in grpc_resource_user_alloc_slices(grpc_resource_user_slice_allocator*, unsigned long, unsigned long, grpc_slice_buffer*) () from /usr/local/lib/libgrpc.so.10

#12 0x00007fcf5a998d3a in tcp_continue_read((anonymous namespace)::grpc_tcp*) () from /usr/local/lib/libgrpc.so.10

#13 0x00007fcf5a998eb9 in tcp_handle_read(void*, grpc_error*) () from /usr/local/lib/libgrpc.so.10

#14 0x00007fcf5a7e7924 in grpc_core::Closure::Run(grpc_core::DebugLocation const&, grpc_closure*, grpc_error*) () from /usr/local/lib/libgrpc.so.10

#15 0x00007fcf5a99903c in tcp_read(grpc_endpoint*, grpc_slice_buffer*, grpc_closure*, bool) () from /usr/local/lib/libgrpc.so.10

#16 0x00007fcf5a974231 in grpc_endpoint_read(grpc_endpoint*, grpc_slice_buffer*, grpc_closure*, bool) () from /usr/local/lib/libgrpc.so.10

#17 0x00007fcf5a92e240 in continue_read_action_locked(grpc_chttp2_transport*) () from /usr/local/lib/libgrpc.so.10

#18 0x00007fcf5a92e157 in read_action_locked(void*, grpc_error*) () from /usr/local/lib/libgrpc.so.10

#19 0x00007fcf5a973b48 in grpc_combiner_continue_exec_ctx() () from /usr/local/lib/libgrpc.so.10

#20 0x00007fcf5a9865de in grpc_core::ExecCtx::Flush() () from /usr/local/lib/libgrpc.so.10

#21 0x00007fcf5a97d847 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) () from /usr/local/lib/libgrpc.so.10

#22 0x00007fcf5a98592a in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) () from /usr/local/lib/libgrpc.so.10

#23 0x00007fcf5a98d9f0 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) () from /usr/local/lib/libgrpc.so.10

#24 0x00007fcf5a9f40bc in cq_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) () from /usr/local/lib/libgrpc.so.10

#25 0x00007fcf5a9f4383 in grpc_completion_queue_pluck () from /usr/local/lib/libgrpc.so.10

#26 0x00007fcf629ad14d in grpc::CoreCodegen::grpc_completion_queue_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) () from /usr/local/lib/libgrpc++.so.1

#27 0x0000556bdacbcef4 in grpc_impl::CompletionQueue::Pluck (this=0x7fcef0000ba8, tag=0x7fcefa7fbb30) at /usr/local/include/grpcpp/impl/codegen/completion_queue_impl.h:321

#28 0x00007fcf5c4f4e27 in grpc_impl::ClientReader<grpcservices::commandservices::ParameterPacketData>::Read (this=0x7fcef0000b90, msg=0x556bdb10de10) at /usr/local/include/grpcpp/impl/codegen/sync_stream_impl.h:215

[AJ Heller]

What version of gRPC are you using? Can you try with the latest release? The ru_alloc_slices code was rewritten a while ago, and it would be valuable to see if this bug still shows up or not.

[Pragadeesh nagaraj]

Hi,

I was using the below versions

grpc: v1.30.0

protobuf: 3.12.2

I think these seems to be latest versions.

[AJ Heller]

I see. I'm guessing that's your system package, maybe on Ubuntu or Debian?

Please try a build with the latest official release from https://github.com/grpc/grpc/releases. We are on v1.48 now, the v1.30 release is 2 years old and outside of our maintenance window. Numerous bug fixes and implementation changes have happened since then.