[grpc c++] Stream buffer writting policy

[David Audrain]

Hi,

I am doing some testing between a client and a server to understand how GRPC C++ buffers messages being written to a RPC stream when the network connection is switched off. 

GRPC_ARG_HTTP2_WRITE_BUFFER_SIZE is set to 64 bytes when creating the channel, and each message written to the stream is 52 bytes large.

Steps are the following:

• Start server and client
  
• Client creates a channel with no keep alive and GRPC_ARG_HTTP2_WRITE_BUFFER_SIZE is set to 64 bytes
  
• Client writes 52 bytes large message every 5 seconds, server logs each message received
  
• Network connection is turned off at time T
  
• Client keeps writing 52 bytes large message every 5 seconds, WriterInterface::Write keeps returning true.
  
• Network connection is turned on at time T + 1 minute
  
• Clients continues  writing 52 bytes large message every 5 seconds, Server logs the message written around time T and messages written after T + 1 minute
  

As expected only one message has been buffered by GRPC library.

However it is the first message being written after the network connection has been turned off and subsequent writing operations did not escalate any error saying that written messages were lost.

Is there any API that would notify about such situation?

Thank you,

David

[Noah Eisen]

Can you elaborate on what you mean by "Network connection is turned off at time T"? If the underlying TCP connection is broken, then the client should definitely notice something is wrong in that dark minute...

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/483d654c-0654-4fc1-8612-f929820e9f85%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Audrain]

"Network connection is turned off at time T" would be like turning Wi-Fi off on a Mac while client is writing messages on a client streaming service.

[Noah Eisen]

I am not familiar with how a Mac handles network change like that.

Can you re run this case with more verbose tracing turned on?

export GRPC_TRACE=call_error,http,api

export GRPC_VERBOSITY=debug

If the client isn't seeing any strangeness when it gets to sending the bytes, then there is nothing to do here. gRPC does not have a built in mechanism for messages in a long lives stream to be ack-ed. I would suggest adding something to your service as a sanity check that all messages came through. Seems like even a dumb incremental counter might work.

On Mon, May 14, 2018 at 7:47 PM David Audrain <david....@gmail.com> wrote:

"Network connection is turned off at time T" would be like turning Wi-Fi off on a Mac while client is writing messages on a client streaming service.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/1a42aec9-80a0-4b54-9ad0-ae8e98619e7f%40googlegroups.com.

[David Audrain]

Please find a debug.log generated by GRPC C++ 1.7.1

Wifi connection has been turned off between I0515 16:06:35 and 16:07:40.

GRPC_ARG_HTTP2_WRITE_BUFFER_SIZE is set to 64

[Noah Eisen]

These log statements are quite old... what version of gRPC are you using?

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/0db25e69-8c08-4447-b343-b400c345f8d0%40googlegroups.com.

[David Audrain]

v1.7.1

I am in the process of using v1.11.1

[Noah Eisen]

Ok, let me know if things change with 1.11.

But as I said, if gRPC client does not notice anything wrong with the TCP connection, it has no way to report failures.

The server would be responsible for noticing the missing data and closing the stream accordingly

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/4bec5fdf-e981-4219-8830-345c8e818dc0%40googlegroups.com.

[David Audrain]

Attached is the log generated with grpc v1.11.1 .

My test application looks to behave the same way.

Everything looks fine from a grpc API perspective after the wifi connection is being turned off around 9h34m35s.

Wifi connection is turned on around 9h37m00s and test application processes messages from the backend around 9h37m29s.

[Noah Eisen]

In that case, this is something that you will have to handle in application logic. gRPC bidi streams to not have ack mechanisms per-message

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/82e14867-e4bb-440b-8c19-3bee86a864ff%40googlegroups.com.