Devtools protocol: Network.responseReceived, encoded data bigger then non encoded

1,044 views
Skip to first unread message

Renault John Lecoultre

unread,
Oct 21, 2015, 11:56:44 AM10/21/15
to Chromium-dev
Hi,

I'm trying to analyse what the devtools gives me and I'm surprised to see that inside the Network.responseReceived object, the encoded data length is bigger then the data length:

  {
    "message": {
      "params": {
        "timestamp": 5873.373919, 
        "encodedDataLength": 13616, 
        "requestId": "320.3", 
        "dataLength": 8325
      }, 
      "method": "Network.dataReceived"
    }, 
    "webview": "00381C36-DE4D-42DA-BF4E-72D2D97C98F9"
  }, 
  {
    "message": {
      "params": {
        "timestamp": 5873.374212, 
        "encodedDataLength": 0, 
        "requestId": "320.3", 
        "dataLength": 5179
      }, 
      "method": "Network.dataReceived"
    }, 
    "webview": "00381C36-DE4D-42DA-BF4E-72D2D97C98F9"
  }, 
  {
    "message": {
      "params": {
        "timestamp": 5873.372868, 
        "encodedDataLength": 13616, 
        "requestId": "320.3"
      }, 
      "method": "Network.loadingFinished"
    }, 
    "webview": "00381C36-DE4D-42DA-BF4E-72D2D97C98F9"
  }, 

encodedDataLength: 13616
dataLength: 13504
Advertised Content-Length: 13504

There is no Content-Encoding in the response headers. Is there an explanation for the 112 bytes difference or is it a bug?

Thanks,
Renault

PS: for sake fo completeness, here is the response message:
{
    "message": {
      "params": {
        "timestamp": 5873.373595, 
        "frameId": "320.2", 
        "requestId": "320.3", 
        "loaderId": "320.3", 
        "type": "Image", 
        "response": {
          "mimeType": "image/png", 
          "remotePort": 443, 
          "connectionId": 15, 
          "protocol": "h2", 
          "requestHeaders": {
            ":path": "/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png", 
            "accept-language": "en-US,en;q=0.8", 
            "accept-encoding": "gzip, deflate, sdch", 
            "accept": "image/webp,image/*,*/*;q=0.8", 
            ":authority": "www.google.com", 
            ":scheme": "https", 
            "cookie": "PREF=ID=1111111111111111:FF=0:TM=1445294416:LM=1445294416:V=1:S=2HuxIpXFBX815OPe; NID=72=oYivFg_a_0SW1T2BKY0F7iReh5N3io8mJBwXgN8_T7XIUVG-HbD9jbUV-jwdBMyip91WyvTtOsesxwhhtHSQwH4iIibS1tZwolsg81Y8yHfDFmSpt8Knv1wmzYBeM8D4", 
            ":method": "GET", 
            "user-agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
          }, 
          "encodedDataLength": -1, 
          "connectionReused": true, 
          "fromServiceWorker": false, 
          "timing": {
            "dnsEnd": -1, 
            "receiveHeadersEnd": 26.3009999998758, 
            "sslEnd": -1, 
            "sendStart": 0.395000000025902, 
            "connectEnd": -1, 
            "proxyStart": -1, 
            "workerStart": -1, 
            "requestTime": 5873.345631, 
            "workerReady": -1, 
            "sendEnd": 0.591999999414838, 
            "sslStart": -1, 
            "dnsStart": -1, 
            "connectStart": -1, 
            "proxyEnd": -1
          }, 
          "headers": {
            "status": "200", 
            "alternate-protocol": "443:quic,p=1", 
            "content-length": "13504", 
            "x-xss-protection": "1; mode=block", 
            "x-content-type-options": "nosniff", 
            "expires": "Mon, 19 Oct 2015 22:40:16 GMT", 
            "server": "sffe", 
            "last-modified": "Fri, 04 Sep 2015 22:33:08 GMT", 
            "cache-control": "private, max-age=31536000", 
            "date": "Mon, 19 Oct 2015 22:40:16 GMT", 
            "alt-svc": "quic=\"www.google.com:443\"; p=\"1\"; ma=600,quic=\":443\"; p=\"1\"; ma=600", 
            "content-type": "image/png"
          }, 
          "fromDiskCache": false, 
          "status": 200, 
          "statusText": "OK", 
          "remoteIPAddress": "74.125.224.16", 
          "securityState": "secure", 
          "securityDetails": {
            "keyExchange": "AES_128_GCM", 
            "certificateId": 1, 
            "protocol": "TLS 1.2", 
            "cipher": "ECDHE_RSA"
          }
        }
      }, 
      "method": "Network.responseReceived"
    }, 
    "webview": "00381C36-DE4D-42DA-BF4E-72D2D97C98F9"
  }

Matthew Menke

unread,
Oct 21, 2015, 2:53:02 PM10/21/15
to Chromium-dev
How big are the headers in the response?  It may be including them.

Renault John Lecoultre

unread,
Oct 23, 2015, 12:12:32 PM10/23/15
to Chromium-dev
Hi Matthew,

This was my first idea, however, I've discarded it for the following reason:

To my understanding, response headers are not encoded (compressed). Given the fact that there is approximatively 650 characters in the response headers (give or take) and that headers are using ISO-8859-1 char encoding, they should take the same amount of bytes on the wire. This cannot explain the 112 bytes discrepancy that I'm seeing.

Renault

Matthew Menke

unread,
Oct 23, 2015, 1:39:16 PM10/23/15
to Chromium-dev
The request headers you posted were for HTTP2 or QUIC, which do compress headers.

Renault John Lecoultre

unread,
Oct 23, 2015, 2:05:29 PM10/23/15
to Chromium-dev
That's interesting!

Now, given the headers size, 650 bytes, and the observed data discrepancy, 112 bytes, we would have a compression factor of 83%. It sounds like a lot, no?

Do you know, or anyone else, for the matter, if encodedDataLength should contain the response headers and the body or just the body length?

PhistucK

unread,
Oct 23, 2015, 2:11:17 PM10/23/15
to renault....@appdynamics.com, Chromium-dev

On Fri, Oct 23, 2015 at 9:05 PM, Renault John Lecoultre <renault....@appdynamics.com> wrote:
Now, given the headers size, 650 bytes, and the observed data discrepancy, 112 bytes, we would have a compression factor of 83%. It sounds like a lot, no?

​I think HTTP2 has some known shorthands for common headers (and values?), that makes the compression ratio high.



☆PhistucK

Matthew Menke

unread,
Oct 23, 2015, 7:04:20 PM10/23/15
to Chromium-dev, renault....@appdynamics.com
HTTP2 has magic where it basically compresses headers across requests.  So if you send "content-encoding: sploggy" in 47 different responses, you'd just say "Repeat that header key again, repeat that header value again" most times.  I believe it actually uses a dictionary that can be mutated as needed, rather than referring to headers from previous requests directly, but I could be misremembering.  May also have build-in short-hand for common headers, not sure.  Look up HPACK encoding, if you're interested.
Reply all
Reply to author
Forward
0 new messages