grpc-python response streaming request always end together

[jame...@gmail.com]

Hi:

I'm writing a rpc server/client very similar to the one on tutorial with server-side pool connection. There are multiple clients and they requsted at a defferent time.

However, if the server receive an additional request while already streaming to one client, these two requests always end simultaneously, and the total transmiting time is longer than doing it seperately. So you can imagine if I have 100 clients, the requests are never finished.

I'm not sure if it is a config problem or other, has anyone run into similar issues?

[Nathaniel Manista]

It is unexpected that a server's servicing different, separate, unrelated RPCs will interfere with each other in this way. The first thing that comes to mind as a possibility is that you may have limited your server's thread pool to too few threads. What is the maximum number of threads you've set for your server-side thread pool?

Even more information would be very helpful. With what version of gRPC Python are you working, and on what platform and operating system? Are you able to share with us the code you've written that demonstrates the surprising behavior?

Thanks much,

-Nathaniel

[James Ge]

Hi Nathaniel:

I set pool threads to 10 on server side, and only run 5 clients process. In addition, I tried to increase the server threads to 100, but got the same result. 

here is my specs:

server:

os: windows 7

grpc: 0.3.prost19

grpcio:1.0

client:

os: windows 7 / centos

grpc: 0.3.prost19

grpcio:1.0

code is attached, thank you very much for your help!

[Nathaniel Manista]

On Thu, Sep 15, 2016 at 8:48 PM, James Ge <jame...@gmail.com> wrote:

Hi Nathaniel:

I set pool threads to 10 on server side, and only run 5 clients process. In addition, I tried to increase the server threads to 100, but got the same result. 

here is my specs:

server:

os: windows 7

grpc: 0.3.prost19

I'm not sure what this is as a thing separate and distinct from grpcio.

grpcio:1.0

client:

os: windows 7 / centos

grpc: 0.3.prost19

Still not sure what this is.

grpcio:1.0

code is attached, thank you very much for your help!

That's a lot of code - would you be able at all to minimize it? What's the most minimal code you can write that demonstrates the undesirable behavior? Do you have a defect in your servicer in which you set a _tslock attribute but access a _ts_lock attribute?

-Nathaniel

[James Ge]

Hi Nat

regarding grpc and grpcio, I installed it via using pip, i.e. 

pip install grpc

pip install grpcio

I believe it would make sure me using the latest version?

 

I simplify the code a little bit. Inside client.py, when I was using only one process to run multi_test, i.e. multi_test(1), the streaming takes about 9 seconds on my PC. However, when using 2 processes, i.e. multi_test(2), it takes 21 seconds. I don't really understand why it takes more time than running multi_test(1) twice

Thanks a lot for the reply

[Nathaniel Manista]

On Tue, Sep 20, 2016 at 2:32 AM, James Ge <jame...@gmail.com> wrote:

Hi Nat

Hello James! My name is Nathaniel.

regarding grpc and grpcio, I installed it via using pip, i.e. 

pip install grpc

Unfortunately, the "grpc" package on PyPI is something completely unrelated to gRPC. Your having installed it probably doesn't hurt (I don't know with certainty) but it is certainly not relevant. In terms of investigating your problem it's only going to get in the way.

pip install grpcio

I believe it would make sure me using the latest version?

"pip freeze" will tell you what versions of what packages are installed; please share its output in your next report.

 

I simplify the code a little bit. Inside client.py, when I was using only one process to run multi_test, i.e. multi_test(1), the streaming takes about 9 seconds on my PC. However, when using 2 processes, i.e. multi_test(2), it takes 21 seconds. I don't really understand why it takes more time than running multi_test(1) twice

The first thing that leaps out at me is the use of multiprocessing - we have designed and tested gRPC Python in a multithreaded context, but until now multiprocessing hasn't come up and I have no knowledge whatsoever of any reasons why performance should or shouldn't be affected by the use of multiprocessing. Do you continue to observe the problematic behavior if you change the use of multiple processes to use of multiple threads?

My next reaction is that your observations need a baseline: how fast does your code run when you replace the use of gRPC Python with simple function calls but keep everything else (including the use of multiple processes or multiple threads) exactly the same?

-Nathaniel