CloudFlare and bidirectional streaming

[Peter Olah]

Hey,

We're facing issues when we're trying to communicate via a gRPC endpoint where both the input and return values are streams (bidirectional streaming). In the attached minimal example, we're having a service which hashes the received data upon requests and returns with calculated hash. By increasing the data size in the client the communication becomes more unreliable. The sweet spot seems to be somewhere between 32k and 64k.

However we don't understand the reason for this behaviour. We didn't get any real error, the communication just 'stops'. Eventually it times out and the channel gets torn down. We're experiencing this behaviour only when we're communicating through CF: in case of a direct connection to one of our IP it doesn't happen.

Please help us understand what's happening here.

The minimal example can be found here: https://github.com/nothingam/grpc-hash-service

Thanks,
Peter

[Yuri Golobokov]

Have you tried capturing debug logs on the client and the server? I think comparing the logs of CF path vs the logs of direct connection might help.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAJDqnpLWr%3D1MLQvjiJcG85NkhV%2B82rpRSof7wZqYU5jnoCxjig%40mail.gmail.com.

[Yuri Golobokov]

Did you try just the http, i.e., `GRPC_TRACE=http` ?

On Mon, Jun 10, 2024 at 3:43 AM Peter Olah <p.not...@gmail.com> wrote:

Hey!

Thanks for the suggestion. I tried to compare those logs, but I'm not sure what to look for. The trace log contains an overwhelming amount of log lines, I can hardly mark the boundaries between each message. Do you have any suggestion what component could be excluded from the trace? Are there any troubleshooting guides?

Thanks,
Peter

[Peter Olah]

Hey!

I did, also I went through the logs from the secure_endpoint component.

I've added a counter to each message buffer, so I could follow each message in the trace logs. It seems just before the client stops, it sends a (first part of) request to the server side (which is not in the server side logs) and receives a couple responses from previous requests. Although the last message never reached the server. 

Logs are here: https://mega.nz/file/FxJExLiZ#YlTwpIx_FadyRtcT5DDmz4pCefDxeBsixkADGcUJJpo

Thanks,
Peter

[Peter Olah]

Hey!

Thanks for the suggestion. I tried to compare those logs, but I'm not sure what to look for. The trace log contains an overwhelming amount of log lines, I can hardly mark the boundaries between each message. Do you have any suggestion what component could be excluded from the trace? Are there any troubleshooting guides?

Thanks,
Peter

Yuri Golobokov <golo...@google.com> ezt írta (időpont: 2024. jún. 7., P, 20:56):