A few questions as beginner to gRPC

[Mary Zhang]

Hi gRPC team,

I just started to explore gRPC as a possible solution for all our client side device to device(AR and Robotic to a medical imaging device), and backend server to client side device communications including server push.

We have used WebSocket briefly with server push prototype working and device communication too. However, we've not tried any subprotocol for WebSocket yet and found gRPC may be convenient since we use Protobuf for most device communication already.

Our major concern is the gRPC server scale issue which needs managing possible several thousands of long live concurrent connections for server push to customer devices. 

Our backend servers are mainly Flask + Python now and so I am trying Python examples now. 

Here are some questions:

1. Is gRPC in Python good enough to support several thousands of long live concurrent connections? For sure, application load balancing plus a cluster of servers will do. My question is more about whether Python is a good choice for this case OR Java/GO can do much better using default gRPC implementation?

2. If Python is good enough for above case, which approach or example should I follow from grpc/examples/python on grpc github repo?

3. I've tried the helloworld and hellostreamingworld examples with some load tests too. For the hellpstreamworld which uses asyncio and stream API, I got it tested with 1000 concurrent clients and server replied with 100 messages per client yesterday on my Laptop. However, when I tried to add stats about all expected messages received and the total time today, I got error about gRPC call timeout and unavailable even after I reduced to 10 clients and 10 messages and roll back most changes. Any clue about the cause?  

 

Thank you very much!

Mary

[Sourabh Singh]

1. Is gRPC in Python good enough to support several thousands of long live concurrent connections?

-  Python's Global Interpreter Lock (GIL) can limit the performance of a single gRPC server process.
- with horizontal scaling you might still achieve the scale.
- Language choice will depend on teams expertise. project goal, many other factors. although per server Go / Java will give better performance.

2. Load Testing
- Testing with laptop won't isolate and identify problem areas.
- Client might struggle to initiate and manage 1000 concurrent connections, even if the server can handle them.
- Resource Exhaustion / open file limits / network buffer limitations
- Using a cloud-based VM for load testing offers a more controlled and scalable environment, helping identify bottlenecks.

[Prince Roshan]

Saurabh, could you clarify where the GIL became a bottleneck in your use case? I'd also love to know more about the server setup. Did the gRPC team conduct any benchmarks beyond the documented ones, like memory profiling or CPU usage analysis, for such scenarios?

From what I see, CPU-intensive operations (if any) could be optimized using tools like Gunicorn or a reverse proxy in front of the gRPC server. For his setup, I doubt image processing or heavy computations are offloaded to the gRPC server, so Python should handle it, even on a laptop, provided memory footprint and blocking I/O are managed effectively with asyncio.

Key concerns might be:

1. CPU-intensive tasks (manageable with optimizations).
2. High memory usage for large connections.
3. Blocking I/O issues (if present).

Regarding the timeout issues in his hellostreamingworld test, try using an interceptor to log channel state. Channels can handle multiple RPCs concurrently, so consider adding unique RPC IDs for better debugging. Configure timeouts for long-running processes and ensure proper resource limits on the server. These steps could help pinpoint the issue.