esp v1 - GRPC -timeout issue with bidir/full duplex stream

[Ludovic Havet]

 Hello,

I have an issue with GRPC in bidir stream (stream both in request and response for an rpc call). When the client initiates the call, I get a 408 Request timeout. I isolated the fact that it comes from the ESP. 

I tried changing the timeout values by passing a custom nginx config but it changes nothing:

http {
  include /etc/nginx/mime.types;
  server_tokens off;
  client_max_body_size 32m;
  client_body_buffer_size 128k;

    proxy_send_timeout 86400s;
    proxy_read_timeout 86400s;
    keepalive_timeout 86400s;

 

 Here is what I get on the client side:

[Wed, 19 Feb 2020 12:48:33 GMT][rcv] error { Error: 9 FAILED_PRECONDITION: REQUEST_TIMEOUT
    at Object.exports.createStatusError (/home/lhavet/dev/orion/common/orion-fake-agent/node_modules/grpc/src/common.js:91:15)
    at ClientDuplexStream._emitStatusIfDone (/home/lhavet/dev/orion/common/orion-fake-agent/node_modules/grpc/src/client.js:233:26)
    at ClientDuplexStream._receiveStatus (/home/lhavet/dev/orion/common/orion-fake-agent/node_modules/grpc/src/client.js:211:8)
    at Object.onReceiveStatus (/home/lhavet/dev/orion/common/orion-fake-agent/node_modules/grpc/src/client_interceptors.js:1311:15)
    at InterceptingListener._callNext (/home/lhavet/dev/orion/common/orion-fake-agent/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/home/lhavet/dev/orion/common/orion-fake-agent/node_modules/grpc/src/client_interceptors.js:618:8)
    at /home/lhavet/dev/orion/common/orion-fake-agent/node_modules/grpc/src/client_interceptors.js:1127:18
  code: 9,
  metadata: Metadata { _internal_repr: {}, flags: 0 },
  details: 'REQUEST_TIMEOUT' }

It runs in GKE. It tried doing a port forward directly to the container serving the grpc, and in this case there is no timeout. Hence it has to come from the ESP.

Could it be a bug in the ESP?

Could you please help me resolve this issue? 

thank!

[Ludovic Havet]

Ok with client_body_timeout 86400s

the issue does not happen.

This could be a default for grpc esp conf? 

Otherwise this means that if the client does not send anything in the stream for one minute (default), then connection is broken.

[qiwz...@google.com]

I created this PR so you can set it in esp flag, not using custom nginx config.

[Ludovic Havet]

Great thanks!

I'll check when it's release.

[qiwz...@google.com]

Just released ESP 1.46.0 should have this flag. Thanks -Wayne