deadlock in iomgr

[ro...@ubimo.com]

Hi,

I wrote a short C++ client (see the code below) and it gets stuck at  grpc_shutdown() line with the following printings:

D0406 16:27:46.618330686   25515 iomgr.c:107] Waiting for 1 iomgr objects to be destroyed

D0406 16:27:56.618329425   25515 iomgr.c:130] Failed to free 1 iomgr objects before shutdown deadline: memory leaks are likely

E0406 16:27:56.818038715   25515 sync_posix.c:48] assertion failed: pthread_mutex_lock(mu) == 0

What can be the cause of it?

int main() 

   grpc_init();

   std::shared_ptr<ChannelInterface> channel =    grpc::CreateChannel(address, grpc::InsecureCredentials(), ChannelArguments());

  std::unique_ptr<be::TestServer::Stub> stub = be::TestServer::NewStub(channel);

  ClientContext context;

  be::ServerRequest req, resp;

  grpc::CompletionQueue cq;

  std::unique_ptr<::grpc::ClientAsyncResponseReader<::be::ServerRequest>> resp_reader =    stub->AsyncCall(&context, req, &cq, tag(1));

  void* got_tag;

  bool ok = false;

  cq.Shutdown();

  while (cq.Next(&got_tag, &ok)) {

    LOG(INFO) << "Got tag " << got_tag << ", ok " << ok;

  }

  stub.reset();

  grpc_shutdown();

  return 0;

}

[Todd Poynor]

Filed an issue internally on this.  

If the Server object (std::unique_ptr<grpc::Server> server_) is in static/global, rather than as a member of a class for which an instance is constructed and destroyed (as does end2end_test.cc), then the std::unique_ptr to the Server object is still unreclaimed at exit.  If grpc_shutdown() is called, it calls grpc_iomgr_shutdown(), which calls:

    gpr_mu_destroy(&g_mu);

At server exit the iomgr code is still called to lock that mutex, causing an assertion failure on the lock error...

fD0228 10:54:08.329319078    3814 iomgr.c:107] Waiting for 1 iomgr objects to be destroyed

D0228 10:54:18.329435002    3814 iomgr.c:121] Failed to free 1 iomgr objects before shutdown deadline: memory leaks are likely

I0228 10:54:19.329318697    3814 census_init.c:47] Shutdown census library.

I0228 10:54:19.329365407    3814 census_rpc_stats.c:238] Shutdown census stats store.

I0228 10:54:19.329433439    3814 census_tracing.c:170] Shutdown census trace store.

Segmentation fault (core dumped)

#0  0x00007f2828284cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

#1  0x00007f28282880d8 in __GI_abort () at abort.c:89

#2  0x0000000000615274 in gpr_mu_lock (mu=0x8c3bc0 <g_mu>)

    at /myproj/grpc/src/core/support/sync_posix.c:48

#3  0x000000000062a6ba in grpc_iomgr_add_delayed_callback (cb=0x637fae <do_nothing>, cb_arg=0x0, success=1)

    at /myproj/grpc/src/core/iomgr/iomgr.c:161

#4  0x000000000062a745 in grpc_iomgr_add_callback (cb=0x637fae <do_nothing>, cb_arg=0x0)

    at /myproj/grpc/src/core/iomgr/iomgr.c:174

#5  0x0000000000637f07 in unref_by (fd=0x109d620, n=2)

    at /myproj/grpc/src/core/iomgr/fd_posix.c:116

#6  0x0000000000638132 in grpc_fd_unref (fd=0x109d620)

    at /myproj/grpc/src/core/iomgr/fd_posix.c:170

#7  0x000000000062b0db in unary_poll_pollset_destroy (pollset=0x10a1370)

    at /myproj/grpc/src/core/iomgr/pollset_posix.c:299

#8  0x000000000062ac65 in grpc_pollset_destroy (pollset=0x10a1370)

    at /myproj/grpc/src/core/iomgr/pollset_posix.c:167

#9  0x000000000061afd8 in grpc_completion_queue_destroy (cc=0x10a1360)

    at /myproj/grpc/src/core/surface/completion_queue.c:394

#10 0x0000000000606ca4 in grpc::CompletionQueue::~CompletionQueue (this=0x10a12c0)

    at /myproj/grpc/src/cpp/common/completion_queue.cc:48

#11 0x0000000000607c41 in grpc::Server::~Server (this=0x10a12b0)

    at /myproj/grpc/src/cpp/server/server.cc:209

#12 0x0000000000607ae9 in grpc::Server::~Server (this=0x10a12b0)

    at /myproj/grpc/src/cpp/server/server.cc:197

#13 0x00000000004f83f2 in std::default_delete<grpc::Server>::operator() (this=0x8c0dd8 <server_>, __ptr=0x10a12b0)

    at /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:67

#14 0x00000000004f80a6 in std::unique_ptr<grpc::Server, std::default_delete<grpc::Server> >::~unique_ptr (

    this=0x8c0dd8 <server_>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:184

#15 0x00007f282828a259 in __run_exit_handlers (status=0, listp=0x7f282860d6c8 <__exit_funcs>, 

    run_list_atexit=run_list_atexit@entry=true) at exit.c:82

The assertion failure is at this code:

#2  0x0000000000615274 in gpr_mu_lock (mu=0x8c3bc0 <g_mu>)

    at /myproj/grpc/src/core/support/sync_posix.c:48

48        void gpr_mu_lock(gpr_mu *mu) { GPR_ASSERT(pthread_mutex_lock(mu) == 0); }

(gdb) p mu

$1 = (gpr_mu *) 0x8c3bc0 <g_mu>

(gdb) p *mu

$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = -1, __spins = 0, __elision = 0, __list = {

      __prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 16 times>, "\377\377\377\377", '\000' <repeats 19 times>, 

  __align = 0}

I believe if the unique_ptr to the Server object were cleaned up prior to calling grpc_shutdown() this would be OK.

Todd

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/eca527c9-68c4-414a-a29c-4b294edff6e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yang Gao]

I think you need a channel.reset() next to the stub.reset() to release the channel, calling grpc_shutdown() afterwards should be fine.

The use of grpc_init() and grpc_shutdown() will soon not be required and that will solve the confusion.

On Mon, Apr 6, 2015 at 6:32 AM, <ro...@ubimo.com> wrote:

--

[Roman Gershman]

FYI,

calling channel.reset() does not help. it's still stuck on grpc_shutdown();

--

Roman Gershman  Architect, Ubimo Ltd.

|   ro...@ubimo.com      |    www.ubimo.com