Why increasing max_workers from 1 to any number reduce concurrency in Python gRPC

[alireza hoseini]

I have used the below sample code to run the server:

self.server = grpc.server(futures.ThreadPoolExecutor(max_workers=1)), maximum_concurrent_rpcs=3000)

Some unnecessary codes are removed for better readability. I have used ghz to load test my gRPC server:

ghz --insecure --total=1000 --proto my_proto.proto --call my_proto.XServer.GetUserById -d '{"user_id": "5d32d36bec68e90844d88ae7"}' 0.0.0.0:9000

Summary:

  Count: 1000

  Total: 1.05 s

  Slowest: 66.93 ms

  Fastest: 5.59 ms

  Average: 51.43 ms

  Requests/sec: 948.35

Response time histogram:

  5.586 [1] |

  11.721 [3] |

  17.856 [4] |

  23.990 [4] |

  30.125 [4] |

  36.260 [5] |

  42.395 [6] |

  48.530 [11] |∎

  54.665 [870] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎

  60.799 [84] |∎∎∎∎

  66.934 [8] |

Latency distribution:

  10% in 49.43 ms 

  25% in 50.32 ms 

  50% in 51.82 ms 

  75% in 53.61 ms 

  90% in 54.61 ms 

  95% in 55.38 ms 

  99% in 59.97 ms 

Status code distribution:

  [OK]   1000 responses   

But when I increase max_workers to 2 I get RPS of around 700 and if I increase max_workers I reach to 600 RPS and lower. Why this behavior happens?

My understanding is that max_workers increase workers that need to get the job done I have one worker then it will be served far more slowly. Why this happens? Shouldn't max_workers increase RPS? 

[Lidi Zheng]

In short, because of GIL (https://wiki.python.org/moin/GlobalInterpreterLock).

The `max_workers` flag is meant for IO intensive tasks.

If you are reading from a database or calling an API in your handler, you should observe performance increase as you increase `max_workers`.

Unfortunately, for CPU-bound tasks. The more thread workers you spawned in gRPC will result in more GIL contentions, and slow the whole process down.

Also, due to some legacy issue, the synchronization overhead between threads is quite significant.

If you DO CARE about Python server performance, please help me by posting issue in GitHub! https://github.com/grpc/grpc/issues

So the priority of improving Python server performance can raise.