Performance of customized-grpc-go as gateway

[Zeymo Wang]

 I customed grpc-go as gatway proxy which remove IDL and implements self-TLS1.3 with `cgo` for authenticor to proxy binary-grpc-call forwad to inner binary-RPC.In production I find some unusual phenomenon when profile

KVM 4core + 8G + Intel Xeon E3-12xx v2 (Ivy Bridge) 2099Mhz

connections when profile : 7500

1.  it seem much more than I think per conn mem ( 57k/connetion)

478: 15663104 [13261: 434536448] @ 0x6eadfa 0x6e39d0 0x6ed51f 0x770e03 0x770cde 0x45ce11
#	0x6eadf9	mwcs/http/h2c/transport.newFramer+0x309 /src/src/mwcs/http/h2c/transport/http2_util.go:196

2. gzip malloc much more heap and occupy much more cpu in a periodic time

5: 3317760 [1546645: 1026279383040] @ 0x602b51 0x60e903 0x6ef4dc 0x773389 0x846da6 0x7d3de3 0x7c8f26 0x77969c 0x77adc0 0x7788e2 0x771c31 0x7717b6 0x77667c 0x45ce11
#	0x602b50	compress/flate.NewWriter+0x30    /usr/local/go/src/compress/flate/deflate.go:667

dstat

 19   1  79   0   0   0|   0    76k|2086k 1021k|   0     0 |9740  9289
  4   1  95   0   0   1|   0    80k|1974k  981k|   0     0 |  11k   12k
  6   2  92   0   0   0|   0   136k|2220k 1070k|   0     0 |  11k   12k
  3   1  95   0   0   0|   0   104k|2303k 1082k|   0     0 |  11k   11k
  3   1  95   0   0   0|   0    88k|2121k 1015k|   0     0 |  11k   11k
  3   1  96   0   0   0|   0    72k|1723k  855k|   0     0 |  10k   10k
  3   2  95   0   0   1|   0    84k|2291k  956k|   0     0 |  10k   11k
  3   1  96   0   0   0|   0    80k|2107k  983k|   0     0 |  11k   11k
  3   1  95   0   0   0|   0    52k|2273k 1088k|   0     0 |  11k   12k
  3   1  95   0   0   1|   0    80k|2451k 1065k|   0     0 |  11k   12k
  4   1  95   0   0   0|   0   308k|2059k 1016k|   0     0 |  11k   12k
  4   2  94   0   0   0|   0    44k|2256k 1020k|   0     0 |  11k   12k
  5   3  92   0   0   1|   0    88k|1947k  892k|   0     0 |  11k   12k
  4   1  95   0   0   0|   0   128k|2278k 1102k|   0     0 |  12k   13k
  4   1  95   0   0   0|   0    48k|2290k 1002k|   0     0 |  11k   12k
  3   2  95   0   0   1|   0    76k|2674k 1106k|   0     0 |  11k   12k
  3   2  96   0   0   1|   0    80k|1758k  884k|   0     0 |  10k   11k
  3   1  96   0   0   1|   0   112k|1641k  868k|   0     0 |  11k   11k
 20   1  79   0   0   0|   0    48k|1699k  939k|   0     0 |9811  9175
  3   1  95   0   0   0|   0    84k|2098k 1003k|   0     0 |  11k   12k
  3   1  95   0   0   0|   0    88k|1966k  990k|   0     0 |  11k   11k

when i use sync.Pool of *gzip.Writer, the periodic time become more frequently

dstat

 20   1  78   0   0   1|   0    80k|1950k  774k|   0     0 |9627  9141
  6   2  91   0   0   1|   0    88k|1688k  955k|   0     0 |  11k   12k
  5   2  93   0   0   1|   0    96k|2167k 1027k|   0     0 |  11k   12k
  6   2  92   0   0   1|   0    52k|2088k  969k|   0     0 |  11k   12k
  6   2  92   0   0   1|   0    88k|1813k  970k|   0     0 |  11k   12k
  5   2  93   0   0   1|   0   124k|2386k 1030k|   0     0 |  11k   12k
  5   2  93   0   0   1|   0    48k|2217k  900k|   0     0 |  10k   11k
 11   3  85   0   0   1|   0    88k|2461k 1042k|   0     0 |  11k   12k
  5   2  92   0   0   1|   0    92k|1924k  956k|   0     0 |  11k   12k
 23   2  75   0   0   1|   0    84k|1886k  839k|   0     0 |9515  8898
  4   2  93   0   0   1|   0    72k|1687k  830k|   0     0 |  10k   11k
  4   2  94   0   0   1|   0    76k|1235k  638k|   0     0 |9287  9295
  5   2  93   0   0   1|   0    76k|2205k  966k|   0     0 |  11k   11k
  5   2  93   0   0   1|   0   124k|2194k  968k|   0     0 |  10k   11k
  6   2  92   0   0   1|   0    52k|2383k 1009k|   0     0 |  11k   12k
  5   1  94   0   0   1|   0   112k|1654k  795k|   0     0 |9915    11k
  4   2  95   0   0   0|   0    64k|1431k  636k|   0     0 |8865  8596
  4   2  94   0   0   1|   0    84k|1683k  847k|   0     0 |  10k   11k
  3   1  95   0   0   1|   0    72k|1212k  654k|   0     0 |9362  9317
  5   2  93   0   0   1|   0    32k|2011k  929k|   0     0 |  11k   12k
 22   2  75   0   0   1|   0   150k|2234k  999k|   0     0 |  10k   10k

3.runtime.scanobject may be the solution of cpu periodic time high

perf top

5.16%          mwcs                  rumtime.scanobject
4.36%          mwcs                  compress/flat.(*compress).deflate
3.27%          mwcs                  runtime.memclrNoHeapPointers
3.22%          mwcs                  runime.heapBitsForObject


Question:

1.connction take upon those more memory is nomarl work?
2.why use sync.Pool of *gzip.Writer cause cpu periodic time high frequently?
3.generic-grpc-go have no GC problem in high concurrency ? maybe since proto is more gzip-friendly then json(inner RPC serializer)?
4. is any work for gc performance in gprc-go?

[Zeymo Wang]

maybe cpu periodic time high frequently relate to sync.Pool 's rumtime.futex when accqurie?

[mmu...@google.com]

Answered via github: https://github.com/grpc/grpc-go/issues/1228