SPDY/2 request problems

[Jamie Hall]

Hi all,

I've been working on a SPDY library in Go and have got SPDY/3 working fine. Having got the SPDY/2 server code working just as well, I've been having some issues with SPDY/2 requests. When I request https://www.google.co.uk/ or https://www.facebook.com/, I get error responses. For comparison, I set up a SPDY/2 server and captured the request from Google Chrome, but it appears the same. I'll paste below the relevant details of the two sessions.

Raw binary (excluding headers, since they'd be compressed):

Chrome:  128 2 0 1 1 0 1 177 0 0 0 1 0 0 0 0 0 0

My client: 128 2 0 1 1 0 0  67 0 0 0 1 0 0 0 0 0 0

The only difference I can see here is the lengths, which both match up to their data.

Parsed SYN_STREAMs:

Chrome (cookie header redacted, user-agent shortened):

SYN_STREAM {

Version:              2

Flags:                FLAG_FIN 

Stream ID:            1

Associated Stream ID: 0

Priority:             0

Headers:              Headers {

Accept-Charset  ISO-8859-1,utf-8;q=0.7,*;q=0.3

Accept-Language en-US,en;q=0.8

Cache-Control   max-age=0

Host            localhost

Method          GET

Url             /

User-Agent      ... Chrome/26.0.1410.65 ...

Version         HTTP/1.1

Accept          text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Encoding gzip,deflate,sdch

Cookie          uid="xxxxxxx"; auth="xxxxxxxx"

Scheme          https

}

}

My client:

SYN_STREAM {

Version:              2

Flags:                FLAG_FIN 

Stream ID:            1

Associated Stream ID: 0

Priority:             0

Headers:              Headers {

Method          GET

Url             /

Version         HTTP/1.1

Host            www.google.com:443

Scheme          https

}

}

In response, Google sends a 400 Bad Request response, while Facebook sends a GOAWAY with the Last Known Good Stream ID set to 0.

From what I can see, they're essentially identical. The only conclusion I can come to is that there's a header I'm not sending, but the spec says:

The first line of the request is unfolded into name/value pairs like other HTTP headers and must be present:
 - "method" - the HTTP method for this request (e.g. "GET", "POST", "HEAD", etc)
 - "scheme" - the scheme portion of the URL for this request (e.g. "https")
 - "url" - the absolute path for this request (e.g. "/foo/bar.html")
 - "version" - the HTTP version of this request (e.g. "HTTP/1.1")

If a client sends a SYN_STREAM without all of the method, url, and version headers, the server must reply with a HTTP 400 BAD REQUEST reply.

Any help would be greatly appreciated.

Jamie

[Jamie Hall]

I should probably note that the header names are only displayed in title case; they are sent in lower-case as the spec demands. Also, my pretty printing does not print the frame length, since it isn't stored, but calculated on send and parsed on receive.

[William Chan (陈智昌)]

You're best off providing a decrypted packet trace so we can use spdyshark to debug it. What do the response headers say in the Google response? Can you dump the raw bytes after SSL decryption from your Go spdy framer?

Btw, did you read http://dev.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3#TOC-7.-Incompatibilities-with-SPDY-draft-2? Make sure you have the updated dictionary and endianness and stuff like that or you're going to have a bad time.

On Wed, May 22, 2013 at 9:30 AM, Jamie Hall <the.sl...@googlemail.com> wrote:

I should probably note that the header names are only displayed in title case; they are sent in lower-case as the spec demands. Also, my pretty printing does not print the frame length, since it isn't stored, but calculated on send and parsed on receive.

--
 
---
You received this message because you are subscribed to the Google Groups "spdy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spdy-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jamie Hall]

You're best off providing a decrypted packet trace so we can use spdyshark to debug it. What do the response headers say in the Google response? Can you dump the raw bytes after SSL decryption from your Go spdy framer?

The packet dump is as follows:

SETTINGS (sent)

00000000  80 02 00 04 00 00 00 0c  00 00 00 01 04 00 00 00  |................|

00000010  00 00 03 e8                                       |....|

SYN_STREAM (sent)

00000000  80 02 00 01 01 00 00 5c  00 00 00 01 00 00 00 00  |.......\........|

00000010  00 00 78 f9 df a2 51 b2  62 60 65 60 cb 05 e6 c3  |..x...Q.b`e`....|

00000020  fc 14 06 66 77 d7 10 06  66 90 20 a3 3e 03 3b 54  |...fw...f. .>.;T|

00000030  0d 03 07 4c 2b 03 0b 28  37 31 08 01 53 83 5e 7a  |...L+..(71..S.^z|

00000040  7e 7e 7a 4e aa 5e 72 7e  ae 95 89 89 31 03 5b 31  |~~zN.^r~....1.[1|

00000050  30 be 73 53 19 58 33 4a  4a 0a 8a 19 b0 02 00 00  |0.sS.X3JJ.......|

00000060  00 00 ff ff                                       |....| 

Google's response is as follows:

SYN_REPLY {

Version:              2

Flags:                [NONE]

Stream ID:            1

Headers:              Headers {

Connection      close

Content-Length  925

Content-Type    text/html; charset=UTF-8

Date            Wed, 22 May 2013 12:41:58 GMT

Server          GFE/2.0

Status          400 Bad Request

Version         HTTP/1.1

}

}

and a DATA frame containing:

"<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n  <title>Error 400 (Bad Request)!!1</title>\n  <style>\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}\n  </style>\n  <a href=//www.google.com/><img src=//www.google.com/images/errors/logo_sm.gif alt=Google></a>\n  <p><b>400.</b> <ins>That’s an error.</ins>\n  <p>Your client has issued a malformed or illegal request.  <ins>That’s all we know.</ins>\n"

Btw, did you read http://dev.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3#TOC-7.-Incompatibilities-with-SPDY-draft-2? Make sure you have the updated dictionary and endianness and stuff like that or you're going to have a bad time.

Yeah, I've been through that bit fairly thoroughly and I'm pretty sure I'm following everything correctly. The client works fine with my own SPDY/2 server, and another written in Go, but produces the 400 from Google and GOAWAY from Facebook.

Thanks,

Jamie

[William Chan (陈智昌)]

I misread your earlier email. I thought you had SPDY/2 and were trying to get SPDY/3 working. This is interesting. Does Chrome work with your SPDY/2 server? If not, I know how to debug that more easily.

For debugging a client, I think we might need Roberto to advise you on using the open source flip server, or perhaps if you ran mod_spdy or nginx you could test against their open source impls and see what is going wrong.

PS: We'd love to hear your implementation experience here. Would like to know what you found difficult or confusing and what not.

PPS: Are you writing an external Go library or do you hope to upstream it into their libraries? I wrote a lame one that I think they put in experimental since I wasn't really maintaining it. I'm excited to hear there's work on this front.

--

[Simone Bordet]

Hi,

Just a thought, are you using the right dictionary for compression ?
It changed between SPDY/2 and SPDY/3.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.

[Jamie Hall]

I misread your earlier email. I thought you had SPDY/2 and were trying to get SPDY/3 working. This is interesting. Does Chrome work with your SPDY/2 server? If not, I know how to debug that more easily.

Chrome works fine with my SPDY/2 server, but so does my client, so I'm not sure where the issue is. Thinking about it more, I have an idea. Chrome has been updated with the changes to SPDY/2 outlined in SPDY/3, such as correcting the priority order and making everything network byte order. Accordingly, I did the same in my implementation. It's possible that Google's servers are using the original SPDY/2 specification, which might not have been noticed, since Chrome will now use SPDY/3 when connecting to those servers.

Does anyone have any further insight into how the Google servers handle SPDY/2?

 

For debugging a client, I think we might need Roberto to advise you on using the open source flip server, or perhaps if you ran mod_spdy or nginx you could test against their open source impls and see what is going wrong.

Thanks, I'll see what I can do to get one up and running locally so I can test against that.

PS: We'd love to hear your implementation experience here. Would like to know what you found difficult or confusing and what not.

Implementing SPDY has been an interesting challenge. I already had some experience with this kind of protocol, since my just-finished undergraduate dissertation involved implementing my own protocol inspired by SPDY/3, but replacing HTTP, rather than supplementing it.

The parts I found tricky were getting header compression working (originally, I misinterpreted the compression state as being stream-specific, not connection-specific), and getting flow control right. I still don't completely understand CREDENTIAL frames, but I get the impression they're not used enormously in the public domain, so this is not too important.

Looking ahead, adding support for SPDY/4 will be a challenge, since it's so different from SPDY/3 and SPDY/2. Furthermore, the new compression algorithm is a little daunting.

The general protocol details are fine and getting the basic details working was fine.

 

PPS: Are you writing an external Go library or do you hope to upstream it into their libraries? I wrote a lame one that I think they put in experimental since I wasn't really maintaining it. I'm excited to hear there's work on this front.

It's an external Go library, although it uses net/http to integrate HTTP support. I saw your work in exp, but I felt that since I would add the full protocol logic, it made sense to re-implement the frame structures too to reduce the dependencies. It's available here, and I welcome help and contributions.

Just a thought, are you using the right dictionary for compression ?
It changed between SPDY/2 and SPDY/3.

Yeah, it took a little work, but I have got (de)compression working fine, since the pretty printing has decoded the headers sent by Chrome.

Thanks,

Jamie

[Roberto Peon]

Yea, don't worry about CREDENTIAL frames for SPDY/2 or SPDY/3.

-=R

Thanks,

Jamie

--

[Sunil Panigrahi]

Hey Jamie,

I am trying to write my own parser to be able to convert the SPDY frames to HTTP. I guess you have achieved this already :).

It would be highly appreciated if you could help me a little with my issue. As far as I know, SPDY uses zlib library for compression and decompression of data frames.
When I am trying to decompress the Name/Value pairs of the SYN_STREAM frame using inflate() it is giving me a return value 2( Z_NEED_DICT).
Then I am using
inflateSetDictionary(&inflate_stream, SPDY_dictionary_txt, sizeof(SPDY_dictionary_txt));
to set the dictionary mentioned on "dev.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3#TOC-2.6.10-Name-Value-Header-Block"
Now this function (inflateSetDictionary) is returning me -3(Z_DATA_ERROR).

I am passing the data after the "Pri|Unused | Slot " bits till the end with the data size of received length - 18(prior header size).

Kindly guide me where I am going wrong.

Thanks in advance. :)

[Jamie Hall]

Hey Sunil,

Could I ask which language you're doing this in? It doesn't look like Go, which is the language I used in my library, and I'm afraid I don't have any experience with compression in other languages.

Thanks,

Jamie

[Sunil Panigrahi]

Hey,

Thanks for the reply. I appreciate it.
I am doing it using C/C++. Could you kindly tell me how you uncompressed the data that you received in the Name/Value pair.

 i.e. the compressed data after:

+------------------------------------+
|1|    version    |         1        |
+------------------------------------+
|  Flags (8)  |  Length (24 bits)    |
+------------------------------------+
|X|           Stream-ID (31bits)     |
+------------------------------------+
|X| Associated-To-Stream-ID (31bits) |
+------------------------------------+
| Pri|Unused | Slot |                |
+-------------------+   

Thanks Again :)

On Wednesday, May 22, 2013 5:53:07 PM UTC+5:30, Jamie Hall wrote:

[Jamie Hall]

Hey,

Thanks for the reply. I appreciate it.
I am doing it using C/C++. Could you kindly tell me how you uncompressed the data that you received in the Name/Value pair.

 i.e. the compressed data after:

+------------------------------------+
|1|    version    |         1        |
+------------------------------------+
|  Flags (8)  |  Length (24 bits)    |
+------------------------------------+
|X|           Stream-ID (31bits)     |
+------------------------------------+
|X| Associated-To-Stream-ID (31bits) |
+------------------------------------+
| Pri|Unused | Slot |                |
+-------------------+   

Thanks Again :)

 

The compressed data is in the form:

+------------------------------------+
| Number of Name/Value pairs (int32) |
+------------------------------------+
|     Length of name (int32)         |
+------------------------------------+
|           Name (string)            |
+------------------------------------+
|     Length of value  (int32)       |
+------------------------------------+
|          Value   (string)          |
+------------------------------------+
|           (repeats)                |

I decompress using Go's encoding/zlib package. Once the data is decompressed, I loop through the decompressed data, reading it into an http.Header structure.

The code for this is available here, but I'm afraid I only have the Go version; I don't have any experience using compression in C++.

Thanks,

Jamie

[Tatsuhiro Tsujikawa]

On Fri, May 24, 2013 at 8:38 PM, Sunil Panigrahi <sunil.pan...@gmail.com> wrote:

Hey,

Thanks for the reply. I appreciate it.
I am doing it using C/C++. Could you kindly tell me how you uncompressed the data that you received in the Name/Value pair.

Hi,

If you are interested in C implementation of SPDY protocol, take a look at

https://github.com/tatsuhiro-t/spdylay

zlib inflate/deflate code is in lib/spdylay_zlib.{h,c}

frame pack/unpack are in lib/spdylay_frame.{h,c}

Best regards,

Tatsuhiro Tsujikawa

 

--

[Jamie Hall]

Having done a near-complete rewrite of my library, I've experimented with SPDY/2 a fair amount more and I think I may have some more pieces to the puzzle now.

It would appear that Chrome (v27) has an implementation of SPDY/2 that has some behaviours defined in SPDY/3. For example, Chrome requests images at priority 4 (despite SPDY/2 only having 2-bit priority, so max priority 3), and does not send the additional number of name/header values that were originally specified in SYN_REPLY and HEADERS frames.

As a result, it's possible that other oddities with draft 2 exist. If anyone has any specific insight into the implementation in Chrome and/or google.com, that would be very helpful.

I'll try experimenting with various things, but for now I'll mark SPDY/2 requests as experimental.

[Sunil Panigrahi]

Hey Tatsuhiro Tsujikawa,

Thanks a lot of the link. The code was really very useful and was able to find out where I was actually going wrong.
Help greatly appreciated :)

Regards
Sunil Panigrahi.

[Tatsuhiro Tsujikawa]

Hi,

On Wed, May 29, 2013 at 8:16 AM, Jamie Hall <the.sl...@googlemail.com> wrote:

Having done a near-complete rewrite of my library, I've experimented with SPDY/2 a fair amount more and I think I may have some more pieces to the puzzle now.

It would appear that Chrome (v27) has an implementation of SPDY/2 that has some behaviours defined in SPDY/3. For example, Chrome requests images at priority 4 (despite SPDY/2 only having 2-bit priority, so max priority 3), and does not send the additional number of name/header values that were originally specified in SYN_REPLY and HEADERS frames.

Did your server implementation get priority 4 in SPDY/2 SYN_STREAM?

It is very strange, and it is clearly impossible to get 4 from just 2 bit field as you wrote.

I did quick check server log and saw that chrome sends SYN_STREAM for gif with priority 2.

Best regards,

Tatsuhiro Tsujikawa

 

As a result, it's possible that other oddities with draft 2 exist. If anyone has any specific insight into the implementation in Chrome and/or google.com, that would be very helpful.

I'll try experimenting with various things, but for now I'll mark SPDY/2 requests as experimental.

--

[Jamie Hall]

Did your server implementation get priority 4 in SPDY/2 SYN_STREAM?

It is very strange, and it is clearly impossible to get 4 from just 2 bit field as you wrote.

I did quick check server log and saw that chrome sends SYN_STREAM for gif with priority 2.

Yeah, my mistake, I double-checked my code and I hadn't changed the extraction of the priority value from SPDY/3, so it was off by one bit. D'oh!

Nevertheless, Chrome still appears not to send the extra number of name/header values that were in the SPDY/2 spec, but have since been removed.

Thanks,

Jamie 

[Sunil Panigrahi]

Hey Jamie,

I need your help again. Is it necessary that the SYN_REPLY and the following data be sent in different packets? Is it not possible to just give the length of SYN_REPLY stream and follow it with a data frame in the same packet.

What is happening is, when I am sending the reply back to chrome, it is sending me FIN. Totally confused where I am going wrong :(

[Jamie Hall]

Hey Sunil,

No, the SYN_REPLY and DATA frames should remain completely separate. If Chrome is ending the stream, then something else is going wrong. Presumably the SYN_REPLY is causing the error. Make sure that the SYN_REPLY contains headers for ":version" and ":status", as described in SPDY/3 section 3.2.2.

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "spdy-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spdy-dev/aA69aSj8Fmc/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to spdy-dev+u...@googlegroups.com.

[Sunil Panigrahi]

Hi,

This is the final name value pair on which I am calling deflate.

0 '\000'        7 '\a'  118 'v' 101 'e' 114 'r' 115 's' 105 'i' 111 'o'
 110 'n' 0 '\000'        8 '\b'  104 'h' 116 't' 116 't' 112 'p' 47 '/'
 49 '1'  46 '.'  49 '1'  0 '\000'        6 '\006'        115 's' 116 't' 97 'a'
 116 't' 117 'u' 115 's' 0 '\000'        6 '\006'        50 '2'  48 '0'  48 '0'
 32 ' '  111 'o' 107 'k' 0 '\000'        16 '\020'       120 'x' 45 '-'  120 'x'
 115 's' 115 's' 45 '-'  112 'p' 114 'r' 111 'o' 116 't' 101 'e'
 99 'c'  116 't' 105 'i' 111 'o' 110 'n' 0 '\000'        16 '\020'       49 '1'
 59 ';'  32 ' '  109 'm' 111 'o' 100 'd' 101 'e' 61 '='  98 'b'
 108 'l' 111 'o' 99 'c'  107 'k' 0 '\000'        15 '\017'       120 'x' 45 '-'
 102 'f' 114 'r' 97 'a'  109 'm' 101 'e' 45 '-'  111 'o' 112 'p'
 116 't' 105 'i' 111 'o' 110 'n' 115 's' 0 '\000'        15 '\017'       115 's'
 97 'a'  109 'm' 101 'e' 111 'o' 114 'r' 105 'i' 103 'g' 105 'i'
 110 'n' 0 '\000'        13 '\r' 99 'c'  97 'a'  99 'c'  104 'h' 101 'e'
 45 '-'  99 'c'  111 'o' 110 'n' 116 't' 114 'r' 111 'o' 108 'l'
 0 '\000'        13 '\r' 112 'p' 114 'r' 105 'i' 118 'v' 97 'a'  116 't'
 101 'e' 44 ','  32 ' '  109 'm' 97 'a'  120 'x' 45 '-'  97 'a'
 103 'g' 101 'e' 61 '='  48 '0'  0 '\000'        16 '\020'       99 'c'  111 'o'
 110 'n' 116 't' 101 'e' 110 'n' 116 't' 45 '-'  101 'e' 110 'n'
 99 'c'  111 'o' 100 'd' 105 'i' 110 'n' 103 'g' 0 '\000'        16 '\020'
 103 'g' 122 'z' 105 'i' 112 'p' 0 '\000'        12 '\f' 99 'c'  111 'o'
 110 'n' 116 't' 101 'e' 110 'n' 116 't' 45 '-'  116 't' 121 'y'
 112 'p' 101 'e' 0 '\000'        12 '\f' 116 't' 101 'e' 120 'x' 116 't'
 47 '/'  104 'h' 116 't' 109 'm' 108 'l' 59 ';'  32 ' '  99 'c'
 104 'h' 97 'a'  114 'r' 115 's' 101 'e' 116 't' 61 '='  117 'u'
 116 't' 102 'f' 45 '-'  56 '8'  0 '\000'        4 '\004'        100 'd' 97 'a'
 116 't' 101 'e' 0 '\000'        4 '\004'        116 't' 104 'h' 117 'u' 44 ','
and so on.

I have added the length correctly after compression of the data. The length in the headers is 8 bytes + size of compressed data. I don't understand why I am getting FIN. :(

[Jamie Hall]

Hi,

I'm afraid I can't really make any sense of that. Could you include just the headers text? Make sure the data is being encoded into the HEADER format of 32-bit name length, name, 32-bit value length, value, ....

[Sunil Panigrahi]

Hi,

My fault, I should have mentioned what that is actually.

I am actually doing it for SPDY/2. So the HEADER format needs to be of the format of 16-bit name length, name, 16-bit value length, value, ....

That is what I had pasted above, the above pasted is the memory representation of the name/value header pair that is sent for compression before sending to chrome. The first 2 bytes(characters) here are 0 and 7(ie 7) for "version" follower by the next two bytes 0 and 8 for "http/1.1" and so on..

If you can share any sniff for SYN_REPLY or a received packet response or something like that then help would be highly appreciated. :)

Thanks a lot :)

[Jamie Hall]

Ah, I see. I think the problem might be that you used "http/1.1", not "HTTP/1.1", but here's the full headers from a SPDY/2 session:

Request:

accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-encoding: gzip,deflate,sdch
accept-language: en-US,en;q=0.8
cache-control: max-age=0
host: [xxx]
if-modified-since: [xxx]
method: GET
scheme: https
url: /
user-agent: [xxx]
version: HTTP/1.1

Response:

status: 304
version: HTTP/1.1

To unsubscribe from this topic, visit https://groups.google.com/d/topic/spdy-dev/aA69aSj8Fmc/unsubscribe.

[Sunil Panigrahi]

Hey Jamie,

I finally realized where I was going wrong. Actually I was not compressing the number of name/value pairs. In SPDY 3 this part is not compressed(as per what I understood from the draft) whereas in SPDY2 it is. Chrome must be trying to decompress the 2 bytes in which I had placed the number of name/value(which was already uncompressed). The code finally worked. Now I think I can manage with the rest of it(hopefully).

Thanks for the continuous help. It is highly appreciated. :)