suffering async c++ server poor throughput, what is wrong?

[Arthur Wang]

Hi all:

    I've done a benchmark for the asynchronous version of the c++ server, but the result is terribly poor compared to either the official one or some other results like this done by other users, I'm wondering what is wrong with my test, could someone do me a favor?

    There are the details for my test.:

    Enrivonment :  

• win 10pro 64bit :  intel i7 4.0GHZ * 4 physical core * 2 logic core   RAM:16GB
• mac high serria 10.13.6:  intel i7 2.2GHZ * 4 physical core * 2 logic core  RAM:16G
  

       Note: I don't have any linux machines, so there are no results for it.

   Grpc version: 

• win :  77ec6c94ad58bedfdf8f28682914e9893e318be0, around 2018.1
• mac:  e3f37b7f4320f0e016a806796e2adaed03bf23f7, around 2019.2

   Compiler:

• win: vs2015 Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x86
• mac: Apple LLVM version 10.0.0 (clang-1000.10.44.4)

   Parameters:

• number of CQs an server instance has.
• number of threads totally exist, evenly distributed on each CQ.
• number of pool size(which is the CallData instances pre-allocated for each CQ.

    Client & Server Behavior:  

• Client: Sending arond 30-50k requests to the async server in parallel, and waiting for all of them to be responded.
• Server: Based on the greeter_async_server.cc of the helloworld example - just do some basically ignorable logic and response immediately after that.
  

       The code are in the attachments.

    

    The result (requests the server can deal per second in average) is :

	OS	pool	1 thread	2 thread	4 thread	8 thread	16 thread
1 CQ	win	100	3647	4016	3366	3012	3427
		200	3963	5202	3308	3411	4507
		400	3787	4494	3476	3203	3123
	mac	100	38880	35803	22629	22128	22321
		200	37778	35285	24801	21805	22172
		400	39200	36231	23446	22311	22553
2 CQ	win	100	3086	4144	4575	3662	3959
		200	3334	4354	3588	3507	3536
		400	3069	4299	3565	3830	3852
	mac	100	39154	32362	31948	25227	23441
		200	37792	33046	31908	25169	24142
		400	40584	33909	32446	25284	24943
4 CQ	win	100	3204	4199	4644	3988	3742
		200	3125	4097	3954	3997	3638
		400	3207	4196	3731	4008	3526
	mac	100	39793	33101	28851	32310	25214
		200	38804	31545	32605	32268	24869
		400	38819	32776	31786	32133	25207

     

Recap of the result:

• number of CQs basically has nothing to do with the throughput, which is unbelievable, in my understanding, it is an critical factor after learned how the polling-engine works.
• number of pool size  has nothing to do with the throughput, I'm not sure whether this is normal or not.
• number of threads which are evenly distributed among the CQs  has nothing to do with the throughput on windows, but decreasing it on mac. This is somewhere wired.

      Besides, I tested the synchronous version(greeter_server.cc) of c++ server, giving me the result of 18807 on mac and 3041 on win, which is also very poor.

I've read the example of grpc\test\cpp\qps\server_async.cc which said to be an good example of how to write a high throughput async c++ server. I found its approach of enhancing the throughput is more of less the same with mine:

• scale #CQ.
• scale #thread.

I haven't test the qps example on my machine, I haven't find a easier way to build it and hard to believe to it can achieve a better result with the same environment of mine.

I know there are something must be wrong, but where are they?

- Thanks a lot.

- Arthur.

[Arthur Wang]

And,  of course, the cpu are not fully running on both win(~30+%, 100% fully load) and mac(200+%,800% fully load ).

[Arthur Wang]

And, both the client and the server on the same machine, communicating with tcp, even though I don't think this could be a reason for poor throughput.

[Arthur Wang]

And, I've compiled the released versions for all the dependencies and the application itself. Results are basically the same. 

[Lalit Kumar]

Just to add I'm trying server streaming using c++ bindings. Client opens just one long lived RPC and Server starts streaming data to client. Both running on same Debian machine having 4 cores. Server is implemented using async api. I'm getting max 23-25K messages/second. Message payload is just 4 bytes. Tried multiple threads to service completionQ  but its not changing anything.

-Lalit 

[Lei Wang]

Are you implementing a server for streaming reply? I have provided you an implementation how to implement 'CallStreamDataReply' otherwise you won't be able to use asynchronous mechanism by grpc c++ server. Also, try to use c++ threads pool instead of grpc threads pool for cq.

I have written an end to end test for pubsub services to test how asynchronous server works. There is no serious problems in my tests.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/0bb9c0de-e91e-42ce-b99f-9ace2b44d5b7%40googlegroups.com.

[Lei Wang]

And one I thing I want to figure out is that CQ is just for event loop processing. More cqs does not promise you more speed. You could run cq in following modes:

Sever

- 1 cq in a main loop thread : many worker threads pool to process tagged events each of which releases itself once reading and writing finish. 

- 1 cq in many worker threads because cqs are thread safe : many worker threads pool to process tagged events each of which releases itself once reading and writing finish. 

- many cq in different loop threads : many worker threads to process tagged events events each of which releases itself once reading and writing finish. 

What ever the model you implement, only the number of worker threads affect how much you can process when they arrive.

[Arthur Wang]

After did these optimizations on windows (sorted by the their contributions to the optimized result):

1. build the released version from the server itself and all its dependent libraries.
2. using different threads and assigning each of the threads with a dedicated asynchronous client, each client with a distinguished channel.
3. separating the client and the server from being located on one single machine.

And, I've compiled the released versions for all the dependencies and the application itself. Results are basically the same  .

Note:  this previously posted conclusion is wrong.

I got a better result now:

OS

pool

1 thread

2 thread

4 thread

8 thread

16 thread

1 CQ	win	100	42589	45289	43610	42194	37023
		200	45998	48053	43840	41823	40048
		400	43610	43497	39984	41084	41928
	mac	100	42283	39416	25693	24497	24113
		200	42408	39385	25354	24348	24691
		400	42517	39261	25614	24491	24295
2 CQ	win	100	42319	42918	49480	46533	41928
		200	46189	47236	47824	48030	45310
		400	47664	47778	42480	47058	36941
	mac	100	42265	36456	36523	26695	24838
		200	39463	36127	35842	24570	24467
		400	42444	39292	36995	26702	24642
4 CQ	win	100	40436	43610	41631	45850	40799
		200	46882	44903	50813	41034	42354
		400	45766	49504	46189	48590	45330
	mac	100	41684	39619	34578	36805	26143
		200	42753	36153	36140	36710	24844
		400	42158	36710	36443	36927	26329

Some conclusions based on the above:

1. the release version is a critical factor for benchmark on windows.
2. scaling #CQ / #thread / #pool_size still doesn't help in improving the throughput.
3. on mac, more threads even meaning worse result.

There is still a big gap between my test and the official benchmark for C++, and my cpu usage still being low(50~60% on windows and 100~260% on mac), I don't know what's next.

And I thought different channel objects on the client side may stands for different TCP connections, but my test proved it to be wrong: even I've created 4~8 channel objects on the client side, there was still only one connection between the client and the server, and I presumed that the #connections also being an important factor for throughput, although one single connection can be multiplexed.

The code files are attached.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAF84%3DtevjXUAcuZ-e4dHQO1S-86BxMH%3DNT0Vcy7DPd%3D50%2BZ_Dg%40mail.gmail.com.

[hcas...@google.com]

Hi Arthur,

I see in your client code that you have the "delete" for the call object commented out. This could certainly cause issues with throughput.

In terms of why there is only one connection between the client and server, keep in mind that gRPC attempts subchannel sharing when the endpoints are the same. So if you have several channel objects on the client side and they are going to the same endpoint, they might share the same subchannel, and that's why you're seeing only one connection. If you would like, you can force each channel to have its own connection by passing in different channel arguments for each channel. I believe this is what our official benchmark code does.

To unsubscribe from this group and stop receiving emails from it, send an email to grp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/0bb9c0de-e91e-42ce-b99f-9ace2b44d5b7%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grp...@googlegroups.com.