net/http client: MaxConcurrentStreams exceeded

[Eddie Shvartsman]

Greetings! 

We are seeing a problem at high RPS where repeated http2 calls using the same Transport (reusing as recommended in go docs) are exceeding the server's  MAX_CONCURRENT_STREAMS setting, resulting in a protocol error. The server appears to be setting MAX_CONCURRENT_STREAMS to 100 as indicated in this http2 debug output...

2018/09/13 11:18:04 http2: Framer 0xc42037b180: wrote DATA flags=END_STREAM stream=1 len=0 data=""

2018/09/13 11:18:04 http2: Framer 0xc42037b180: read SETTINGS len=30, settings: HEADER_TABLE_SIZE=4096, MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=65535, MAX_FRAME_SIZE=16384, MAX_HEADER_LIST_SIZE=16384

2018/09/13 11:18:04 http2: Transport received SETTINGS len=30, settings: HEADER_TABLE_SIZE=4096, MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=65535, MAX_FRAME_SIZE=16384, MAX_HEADER_LIST_SIZE=16384

...

2018/09/13 11:18:49 http2: Transport encoding header "content-type" = "application/json"

2018/09/13 11:18:49 http2: Transport encoding header "content-length" = "1547"

2018/09/13 11:18:49 http2: Transport encoding header "accept-encoding" = "gzip"

2018/09/13 11:18:49 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"

2018/09/13 11:18:49 http2: Framer 0xc42037b180: wrote HEADERS flags=END_HEADERS stream=17191 len=56

2018/09/13 11:18:49 http2: Transport readFrame error on conn 0xc420418e00: (http.http2ConnectionError) connection error: PROTOCOL_ERROR

2018/09/13 11:18:49 http2: Framer 0xc42037b180: wrote GOAWAY len=8 LastStreamID=0 ErrCode=PROTOCOL_ERROR Debug=""

If we reduce the RPS to <100, the error does not occur. 

My modest understand of the protocol and how the go http client *should* work is that the client and server negotiate MaxConcurrentStreams. My read of the code is that the client's default is 250 streams; however, that should be negotiated down to 100, given this is the maximum the server supports (is that right?). My understanding of current behavior is that the client should block if MaxConcurrentStreams is exceeded. 

Any insights or help would be greatly appreciated. 

[Eddie Shvartsman]

Correction: current default client MaxConcurrentStreams is 1,000