grpc server has hundreds of connections in CLOSE_WAIT state.

55 views
Skip to first unread message

solomon lifshits

unread,
Aug 28, 2019, 4:32:38 PM8/28/19
to grpc.io
Hello,

My setup has asynchronous grpc servers that implement 2 simple rpc calls and asynchronous clients making this calls.
Clients create a channel and occasionally make rpc calls, in case of grpc failure  they destroy and recreate the channel.

With the time servers stop accepting new connections (Connect failed returned), seemingly running out of available descriptors,  and netstat shows most of the connections in CLOSE_WAIT state,

here is the quote from netstat output for one of the servers (9484 is the grpc port, currently having over 1300 (and counting) CLOSE_WAIT connections and  only 88 ESTABLISHED):

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.217:52670   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.225:61316   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.225:61770   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.225:61784   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.108:39800     CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.108:40357     CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.110:38224     CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.111:61617     CLOSE_WAIT

tcp6       0      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.119:53365     ESTABLISHED

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.120:43786     CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.127:50726     CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.129:56527     CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.133:55294     CLOSE_WAIT

tcp6       0      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.138:59345     ESTABLISHED

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.138:59450     CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.141:46481     CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.223:49494   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.227:49438   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.230:59050   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.232:50686   CLOSE_WAIT

tcp6       0      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.232:51112   ESTABLISHED

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.232:51152   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.236:36119   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.169:44782   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.174:59332   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.174:59795   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.177:60827   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.177:61279   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.177:61303   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.178:43530   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.180:64340   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.184:59960   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.188:43219   CLOSE_WAIT

tcp6       0      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.189:63695   ESTABLISHED

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.191:50126   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.196:63114   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.187:46165   CLOSE_WAIT

tcp6       0      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.187:46591   ESTABLISHED

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.187:46614   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.199:35151   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.210:63536   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.219:54092   CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.13:40982    CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.13:41451    CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.15:60025    CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.16:49409    CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.19:49737    CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.23:37534    CLOSE_WAIT

tcp6     190      0 YYY.YYY.YYY.YYY:9484 XXX.XXX.XXX.25:55782    CLOSE_WAIT




Any idea for such behavior?

Thank you in advance!


Reply all
Reply to author
Forward
0 new messages