Getting no status received when running grpc behind apache using http2

[hardi...@gmail.com]

In my system the grpc server is running a secure server behind port 80. The apache server takes request at port 443 and proxies it to port 80. I could see that the server is getting the request and processing it. But after its done, the client errors out with the following:

Traceback (most recent call last):

  File "grpc_secure_client.py", line 40, in <module>

    response = stub. Process(request)

  File "/home/harora/.pyenv/versions/py-3.5.0-grpc/lib/python3.5/site-packages/grpc/_channel.py", line 550, in __call__

    return _end_unary_response_blocking(state, call, False, None)

  File "/home/harora/.pyenv/versions/py-3.5.0-grpc/lib/python3.5/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking

    raise _Rendezvous(state, None, None, deadline)

grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:

        status = StatusCode.UNKNOWN

        details = "No status received"

        debug_error_string = "{"created":"@1548225523.353383280","description":"No status received","file":"src/core/lib/surface/call.cc","file_line":1058,"grpc_status":2}"

>

The apache logs shows the following line:

 -- 23/Jan/2019:15:59:08 +0000] "POST /service/Process HTTP/2.0" 200 7 "-" "grpc-python/1.18.0 grpc-c/7.0.0 (manylinux; chttp2; goose)" "-" 18539

Any help to debug this would be really appreciated.

For what its worth, i am using python-grpcio compiled with openssl, Apache has h2 enabled and is listening at port 443.

[rbel...@google.com]

Can you run your client with the following environment variables set so we can get a better idea of what's going on?

GRPC_VERBOSITY=debug

GRPC_TRACE=all,-timer,-timer_check

[hardi...@gmail.com]

Server log:  https://drive.google.com/open?id=12g5ixgumXZcy9zoDutujRclEu_NcaU-P

Client log : https://drive.google.com/open?id=12g5ixgumXZcy9zoDutujRclEu_NcaU-P

[hardi...@gmail.com]

Has someone seen this kind of error before?

[hardi...@gmail.com]

Has someone seen this kind of error before?

On Wednesday, 23 January 2019 21:11:21 UTC-5, hardi...@gmail.com wrote:

[rbel...@google.com]

I'm not seeing anything out of the ordinary on the server side, but it looks like you've linked to the server logs twice. Can you please include a link to the client logs as well?

[hardi...@gmail.com]

Sorry about that, updated.

Server log:  https://drive.google.com/open?id=12g5ixgumXZcy9zoDutujRclEu_NcaU-P

Client log : https://drive.google.com/file/d/1sQ1Dw0Om1qliKLNRBuN7xK7Dww3C5rNh

[rbel...@google.com]

Sorry for the delay. Looking at both logs, it seems pretty clear that Apache is dropping some data:

Server:

I0124 01:55:13.735421963    1072 chttp2_transport.cc:1355]   HTTP:1:HDR:SVR: :status: 200
I0124 01:55:13.735426447    1072 chttp2_transport.cc:1355]   HTTP:1:HDR:SVR: content-type: application/grpc
I0124 01:55:13.735430871    1072 chttp2_transport.cc:1355]   HTTP:1:HDR:SVR: grpc-accept-encoding: identity,deflate,gzip
I0124 01:55:13.735435318    1072 chttp2_transport.cc:1355]   HTTP:1:HDR:SVR: accept-encoding: identity,gzip
I0124 01:55:13.735439684    1072 chttp2_transport.cc:1355]   HTTP:1:TRL:SVR: grpc-status: 0
I0124 01:55:13.735443949    1072 chttp2_transport.cc:1355]   HTTP:1:TRL:SVR: grpc-message:

Client:

D0123 20:55:13.778892713   20815 call.cc:1054]               Received trailing metadata with no error and no status
D0123 20:55:13.778896699   20815 call.cc:719]                set_final_status CLI
D0123 20:55:13.778904381   20815 call.cc:720]                {"created":"@1548294913.778895884","description":"No status received","file":"src/core/lib/surface/call.cc","file_line":1058,"grpc_status":2}

On the server side, you can see the gRPC server clearly setting both headers and trailers. But on the client side, we receive empty trailers. They should have contained "grpc-status" and "grpc-metadata". This looks like a bug within Apache to me. I would report it to them.

[robert engels]

I thin the chance that Apache has a bug of this sort at this time is pretty remote… I would probably look elsewhere.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/acec4620-5731-49e7-ae8f-2835ae024411%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[robert engels]

To clarify, wouldn’t that mean that Apache HTTP2 is VERY broken - seems unlikely.

[yas...@google.com]

I haven't seen evidence of data being "dropped". The gRPC server is not sending a status as shown by this log

I0124 01:55:13.735187808    1072 call.cc:1508]               ops[0]: SEND_STATUS_FROM_SERVER status=0 details=(nil)

The client saying that it did not receive a status is actually what the server sent. Why the server is sending nil status is what the next step to debug should be. To be clear, I am not too familiar with the gRPC python internals but I am speaking from what I see in the gRPC core layer (over which gRPC python operates). So in this case, I would first verify what the application server is actually sending and how that's getting converted to a nil status.