stream error: stream ID XXXXXX; NO_ERROR

[Stan Tsouvallas]

Hi,

I'm seeing some pretty weird behaviour when sending HTTP requests between two services that I run. 

When I scale up traffic (approx 600k requests/min) I start to see a lot of occurrences of stream error: stream ID XXXXXX; PROTOCOL_ERROR on my client (resp, err := httpClient.Do(req)), and stream error: stream ID XXXXXX; NO_ERROR on my server (bodyBytes, err := ioutil.ReadAll(r.Body)). 

I can't work out what is causing this. I'm running a non-tls HTTP2 client as follows, with Envoy proxying the requests between the services:

httpClient := &http.Client{
    Transport: &http2.Transport{
        AllowHTTP: true,
        DialTLS: func(netw, addr string, cfg *tls.Config) (net.Conn, error) {
            return dialer.Dial(netw, addr)
        },
    },
    Timeout: defaultTimeout,
}

Whats really weird, is when I make the request using resp, err = server.httpClient.Transport.RoundTrip(req), then the issue basically disappears. After that I only see the error when pending requests are terminated mid-flight which is fine.

Does anyone know whats going on here? I'm far from an expert on the Golang network stack.

Thanks in advance!