C fork() server->embedded Python->gRPC-python deadlock on method call

[ascani...@gmail.com]

I have a Linux forking C server with an embedded Python 2.7.5 interpreter loading a script that uses gRPC.

This is CentOS with kernel 3.10.0-693.11.6.el7.x86_64.

The server creates the channels and grpc stubs post-fork().

However if I attempt any method call at all, the wokder deadlocks, always in the same place, even before hitting the wire.

I have read various issues with grpc and Python multiprocessing/fork(); however this is the case where it is the C server that forks. I'm wondering whether these are essentially the same issue.

grpcio is 1.9.1

(gdb) bt
#0  0x00007f1474a29a0b in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007f1474a29a9f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007f1474a29b3b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007f14297d75f5 in PyThread_acquire_lock () from /lib64/libpython2.7.so.1.0
#4  0x00007f14297a3156 in PyEval_RestoreThread () from /lib64/libpython2.7.so.1.0
#5  0x00007f14297c6358 in PyGILState_Ensure () from /lib64/libpython2.7.so.1.0
#6  0x00007f1421eae385 in __pyx_f_4grpc_7_cython_6cygrpc__slice_from_bytes (__pyx_v_value=0x7f14b5cf6f30)
    at src/python/grpcio/grpc/_cython/cygrpc.c:20167
#7  0x00007f1421ed6522 in __pyx_pf_4grpc_7_cython_6cygrpc_7Channel_2create_call (
    __pyx_v_deadline=0x7f1429a5bf20 <_Py_NoneStruct>, __pyx_v_host=0x7f1429a5bf20 <_Py_NoneStruct>,
    __pyx_v_method=<optimized out>, __pyx_v_queue=0x3015120, __pyx_v_flags=0,
    __pyx_v_parent=0x7f1429a5bf20 <_Py_NoneStruct>, __pyx_v_self=0x3018e10)
    at src/python/grpcio/grpc/_cython/cygrpc.c:6899
#8  __pyx_pw_4grpc_7_cython_6cygrpc_7Channel_3create_call (__pyx_v_self=0x3018e10,
    __pyx_args=<optimized out>, __pyx_kwds=<optimized out>) at src/python/grpcio/grpc/_cython/cygrpc.c:6834
#9  0x00007f14297aabb0 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#10 0x00007f14297aa57d in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#11 0x00007f14297acefd in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#12 0x00007f1429736858 in function_call () from /lib64/libpython2.7.so.1.0
#13 0x00007f14297119a3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#14 0x00007f1429720995 in instancemethod_call () from /lib64/libpython2.7.so.1.0

[ascani...@gmail.com]

(gdb) py-bt
#4 Waiting for a lock (e.g. GIL)
#11 Frame 0x30654e0, for file /usr/local/lib/python2.7/site-packages/grpc/_channel.py, line 475,
 in _blocking (self=<_UnaryUnaryMultiCallable(_managed_call=<function at remote 0x3080320>, _request_serializer=<met
hod_descriptor at remote 0x2f40ab8>, _channel=<grpc._cython.cygrpc.Channel at remote 0x2ffde10>, _response_deseriali
zer=<built-in method FromString of GeneratedProtocolMessageType object at remote 0x2eec1c0>, _method='/
Service/Retrieve') at remote 0x307cbd0>, request=<RetrieveRequest at remote 0x3080230>, ti
meout=None, metadata=None, credentials=None, state=<_RPCState(code=None, due=set([0, 1, 2, 4, 5, 6]), callbacks=[],
trailing_metadata=None, cancelled=False, initial_metadata=None, response=None, condition=<_Condition(_Condition__loc
k=<_RLock(_Verbose__verbose=False, _RLock__owner=None, _RLock__block=<thread.lock at remote 0x2f68930>, _RLock__coun
t=0) at remote 0x307cc90>, acquire=<instancemethod at remote 0x2f64780>, _is_owned=<instancemethod a...(truncated)
    self._method, None, deadline)

[Srini Polavarapu]

Looks similar to https://github.com/grpc/grpc/issues/14258 . Have you tried setting GRPC_ENABLE_FORK_SUPPORT=FALSE as described in https://github.com/grpc/grpc/issues/14056 ?