[C++]Server Completion queue processed from unexpected thread

[Arpit Baldeva]

Hi,

Version - 1.4.2 - Windows - C++.

I noticed some oddities with the server log and want to ensure what I am seeing is intended and not a bug. 

I have multiple completion queues in my server and a thread is dedicated to block on each completion queue separately. I am seeing debug output from a thread processing a completion tag which was NOT queued for it. The cc info printed is correct however.

Debug output:

2017/10/02-10:18:42.850 INFO   Thread 1 grpc_core                       grpc_server_request_registered_call(server=0000025D33F28860, rmp=0000025D342A2F20, call=00000267141ABE78, deadline=0000027F140C24F8, initial_metadata=0000027F140C2530, optional_payload=00000267141ABE80, cq_bound_to_call=0000025D34125530, cq_for_notification=0000025D34125530, tag=00000267141ABE40)

2017/10/02-10:19:05.163 INFO   Thread 2 grpc_core                       cq_end_op_for_next(exec_ctx=000000EF0B5FF3B8, cc=0000025D34125530, tag=00000267141ABE40, error="No Error", done=00007FF69F8FD790, done_arg=0000025D342C3C10, storage=0000025D342C3C48)

2017/10/02-10:19:05.164 INFO   Thread 3 grpc_core                       RETURN_EVENT[0000025D34125530]: OP_COMPLETE: tag:00000267141ABE40 OK

In the output above, the thread 2 and thread 3 are blocking on different completion queues. The tag was added for the completion queue that is blocked by thread 3. However, the thread 2 is also spewing the debug output for the tag. Is that intended?

Thanks.

[Vijay Pai]

cq_end_op_for_next is an internal part of gRPC core; it is the part that actually enqueues ready events onto the CQ. In this case, it's just some other thread that was available to pick up some work and did some processing that caused the system to realize that that particular event was ready. Thread 2 injected the event notification onto the CQ so that Thread 3 could pick it up. The thread that makes the notification available is not always the same as the one that processes the notification (which is the one that called completion_queue_next).