error on accepting massive number of connection with grpc c++ server

[iya...@gmail.com]

hi, all. 

I try to use gRPC C++ server as front end, that means it may accept huge number of connection in short time. 

so now I test its ability to process incoming connection. problem is, if connection number increases, server drop connection (seems no accept failure)

detail description of environment: 

host: OSX El Capitan (10.11.6), 16GB Mem, Intel Core i7 2.6GHz (8 core)

server: Sync mode gRPC C++ running in ubuntu 16.10 based container (just copy server binary) with docker 1.12.0

client: gRPC nodeJS v0.12.7 running in host machine, all client run on same nodejs process, with node-fibers

each client has one instance of service object, which is created by grpc.load(proto_file).package_name.service_name(); and send echo request to server 1000 times, with ssl enabled. 

if number of connection < 100, it seems no problem. but above 200 clients start to show error like following.

```

E0905 13:26:29.887599000 123145305219072 handshake.c:128] Security handshake failed: {"created":"@1473049589.887582000","description":"Handshake read failed","file":"../src/core/lib/security/transport/handshake.c","file_line":237,"referenced_errors":[{"created":"@1473049589.887580000","description":"FD shutdown","file":"../src/core/lib/iomgr/ev_poll_posix.c","file_line":427}]}

```

at the same time server reports like this:

```

E0905 04:26:29.998263585     491 handshake.c:128]            Security handshake failed: {"created":"@1473049589.998250914","description":"Handshake read failed","file":"src/core/lib/security/transport/handshake.c","file_line":237,"referenced_errors":[{"created":"@1473049589.988649722","description":"EOF","file":"src/core/lib/iomgr/tcp_posix.c","file_line":235}]}

```

increasing number of client, around 500, almost half of clients terminate with error like above. 

I try to change net.core.somaxconn  to 1024 or 2048, because it seems to change s_max_accept_queue_size also, but no help. 

I also try to run server on host environment, but result is same. 

it may be a problem of my host's system setting, but currently I have no idea. does anyone have some hint for this problem? 

regards, 

Takehiro Iyatomi (iya...@gmail.com)

[Nicolas Noble]

Can you describe which version of the code you're using ? This seems a bit odd, given we have stress testing in place that shows we can handle load, but I couldn't make sense of these line numbers in 1.0, so I am assuming you're on an older version.

[takehiro iyatomi]

sorry for late reply, and making confusion, as you said, version was not v1.0.0. based on master, commit 19ea0cffd7ba3df686ed05671f9ed35b8e9fa830.

about original question of me, I could connect all of my client correctly with retry when grpc server returns UNAVAILABLE(=14). so now I think gRPC runtime made acceptance limit in somewhere. that's correct? 

2016年9月8日木曜日 2時19分27秒 UTC+9 Nicolas Noble: