SPDY command line client

[Tatsuhiro]

I started to implement my own spdy/2 implementation in C just for my
fun.
https://github.com/tatsuhiro-t/spdylay

In the course of the development, I created SPDY command line client,
named spdycat.
It is a dead simple curl/wget like downloader but speaks SPDY only.

With verbose output, you can see SPDY frame is back and forth:

$ ./spdycat -vn https://www.google.com/
[ 0.029] NPN select next protocol: the remote server offers:
* spdy/2
* http/1.1
[ 0.040] recv SETTINGS frame <version=2, flags=0, length=12>
(niv=1)
[4(1):100]
[ 0.040] send SYN_STREAM frame <version=2, flags=1, length=107>
(stream_id=1, assoc_stream_id=0, pri=3)
host: www.google.com:443
method: GET
scheme: https
url: /
user-agent: spdylay/0.0.0
version: HTTP/1.1
[ 0.087] recv SYN_REPLY frame <version=2, flags=0, length=580>
(stream_id=1)
cache-control: private, max-age=0
content-type: text/html; charset=ISO-8859-1
date: Wed, 01 Feb 2012 15:43:00 GMT
expires: -1
server: gws
status: 200 OK
version: HTTP/1.1
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
[ 0.087] recv DATA frame (stream_id=1, flags=0, length=4096)
[ 0.088] recv DATA frame (stream_id=1, flags=0, length=2617)
[ 0.094] recv DATA frame (stream_id=1, flags=0, length=4096)
[ 0.094] recv DATA frame (stream_id=1, flags=1, length=828)
[ 0.094] send GOAWAY frame <version=2, flags=0, length=4>
(last_good_stream_id=0)

It might be useful for debugging. Just FYI. Have fun.

Best regards,

Tatsuhiro Tsujikawa

[Roberto Peon]

Looks interesting!

-=R

[Ma Chi]

sounds cool

[James A. Morrison]

Thanks, I've managed to use your code to mostly make an obj-c
version of spdycat[1]!

[1] https://github.com/sorced-jim/SPDY-for-iPhone

--
Thanks,
Jim
http://phython.blogspot.com

[Tatsuhiro Tsujikawa]

2012/02/03 7:37 "James A. Morrison" <jim.mo...@gmail.com>:

>
> Thanks, I've managed to use your code to mostly make an obj-c
> version of spdycat[1]!
>
>
> [1] https://github.com/sorced-jim/SPDY-for-iPhone
>

Spdycat on iPhone? That is Great!

For Android 2.x users like me, you can compile spdycat using ndk (For me ndk7 is not working. libsupc++ not found. I used ndk6b).

Here is a spdycat android binary with openssl-1.0.1b2 statically linked.

http://sourceforge.net/projects/spdylay/files/contrib/?

[Gautam Dewan]

Tatsuhiro: I am using spdycat to test my experimental SPDY webserver implementation.

I call it from a simple perl script to send thousands of SPDY requests to my webserver.

It is very useful.

Thanks,

Gautam.

[Tatsuhiro Tsujikawa]

2012/02/14 14:42 "Gautam Dewan" <gde...@gmail.com>:

:)
It is great to hear that my tiny piece of code is actually  helping people.
It really encourages the development effort.

Best regards ,

Tatsuhiro Tsujikawa

[Suyambulingam R M]

Hi   Tatsuhiro ,

I am new to SPDY, am trying to use your  library. But when i connect to localhost from client like ( ./spdycat -Oasv23 https://localhost:3000/   )it is working fine. If i connect to google.com like what u have mentioned (  ./spdycat -vn https://www.google.com/  ) am not getting any reply from server. Could you pls help me to figure this out. Thanks a ton in advance.

Regards,

Suyambu

[Tatsuhiro Tsujikawa]

Hi,

On Thu, Jul 5, 2012 at 10:52 PM, Suyambulingam R M
<lingam....@gmail.com> wrote:
> Hi Tatsuhiro ,
>
> I am new to SPDY, am trying to use your library. But when i connect to
> localhost from client like ( ./spdycat -Oasv23 https://localhost:3000/ )it
> is working fine. If i connect to google.com like what u have mentioned (
> ./spdycat -vn https://www.google.com/ ) am not getting any reply from
> server. Could you pls help me to figure this out. Thanks a ton in advance.
>

I have no problem with https://www.google.com/
Please make sure that you are using latest git master branch of spdylay.
Does getting no reply mean that spdycat blocks and no command prompt returns?

Best regards,

Tatsuhiro Tsujikawa

[Suyambulingam R M]

Hi   Tatsuhiro ,

Am using latest code oly...
client : ./spdycat -nv https://localhost:3000/ oly works for me.If i give https://www.google.com client is not responding at all.
server ./spdyd -v 3000 server.key server.crt

Is there a different procedure to run in command line tool. Am running server and client in the same machine. Can you please give me info that how u are running, Thanks in advance. Awaiting ur reply.

[Tatsuhiro Tsujikawa]

2012/07/06 1:30 "Suyambulingam R M" <lingam....@gmail.com>:

>
>
> Hi   Tatsuhiro ,
>
> Am using latest code oly...
> client : ./spdycat -nv https://localhost:3000/ oly works for me.If i give https://www.google.com client is not responding at all.
> server ./spdyd -v 3000 server.key server.crt
>
> Is there a different procedure to run in command line tool. Am running server and client in the same machine. Can you please give me info that how u are running, Thanks in advance. Awaiting ur reply.
>

My command line is the same one you use. Are you behind a proxy? If so, spdycat does not work because it does not support proxy at the moment.

Best regards,

Tatsuhiro Tsujikawa

[Suyambulingam R M]

Oh.. thats it.. i tried in some other machine yesterday it was working :) thanks a lot. But without running the server itself, the client can able to produce the results for google.com. How s it possible ?

Regards,
Suyambulingam.R.M

[Suyambulingam R M]

Hi  Tatsuhiro  ,

Now am able to figure it out. Am able to run the server and it can able to reply the requested files to the clients. I was struggling as i was using it first time. I am trying to implement spdy server in a way when a new request comes, am spawning a new worker process and making it to handle the request. Is your design compatible for this ? Or is it possible to share the objects between different process. If you have suggestions can u please suggest me. Thanks in advance

Regards,

Suyambu

[Tatsuhiro Tsujikawa]

On Fri, Jul 6, 2012 at 2:42 PM, Suyambulingam R M

<lingam....@gmail.com> wrote:
> Hi Tatsuhiro ,
>

> Now am able to figure it out. Am able to run the server and it can able to
> reply the requested files to the clients. I was struggling as i was using it
> first time. I am trying to implement spdy server in a way when a new request
> comes, am spawning a new worker process and making it to handle the request.
> Is your design compatible for this ? Or is it possible to share the objects
> between different process. If you have suggestions can u please suggest me.
> Thanks in advance

You can use spdylay in your design.
The spdylay library is thread safe as long as same spdylay_session
object is accessed by only one thread at a time. If you need to access
to the spdylay_session object from more than 2 threads, you need lock.
The simplest setting is that in each spawned process, create
spdylay_session object and interact with client.

Here is the rough outline to create spdy server:

1. spawn worker process (or assign pooled process) and pass client socket
2. Setup SSL/TLS object.
3. Perform SSL/TLS handshake, and check NPN result (spdy version is
determined here)
4. Create spdylay_session object using spdylay_session_server_new()
with the version
negotiated in step 3. For callbacks, check the spdyd.cc and
SpdyServer.cc in example
directory.
5. Interact with client using spdylay_session_send() and spdylay_session_recv().

You need private key and certificate for the server. Here is the link
which describes how to create self-signed certificate:
http://httpd.apache.org/docs/2.2/ssl/ssl_faq.html#selfcert

[Suyambulingam R M]

Thanks a lot for your kind help :) )

Regards,

Suyambu

[Suyambulingam R M]

HI Tatsuhiro,

I am looking for ur help again. I am trying with spdy reverse proxy (shrpx). Can you please explain to run it in terminals.  am getting BAD gateway error in client.

Am Running like this

client: ./spdycat -Oav https://localhost:3000/
server: ./shrpx  -b www.google.com,443 -n 3 -L INFO  server.key server.crt

IS it the right way. ? i need to modify some thing? Thanks in advance. Looking for ur reply.

Regards,
Suyambu

[Tatsuhiro Tsujikawa]

On Sun, Jul 15, 2012 at 3:11 AM, Suyambulingam R M
<lingam....@gmail.com> wrote:
> HI Tatsuhiro,
>
> I am looking for ur help again. I am trying with spdy reverse proxy (shrpx).
> Can you please explain to run it in terminals. am getting BAD gateway error
> in client.
>
> Am Running like this
>
> client: ./spdycat -Oav https://localhost:3000/
> server: ./shrpx -b www.google.com,443 -n 3 -L INFO server.key server.crt
>
> IS it the right way. ? i need to modify some thing? Thanks in advance.
> Looking for ur reply.
>

The backend server should not be the outside server like www.google.com.
It is typically the same pc or another pc but in the same server farm
and explicitly configured to work with them. The example usage is use
your local apache2 server as backend.
For example:

$ shrpx -b localhost,80 server.key server.crt

Best regards,

Tatsuhiro Tsujikawa

[Suyambulingam R M]

Thanks Tatsuhiro i got it...

Regards,
Suyambulingam.R.M

[Tatsuhiro Tsujikawa]

On Thu, Jun 13, 2013 at 7:10 PM, Sunil Panigrahi <sunil.kr.pa...@gmail.com> wrote:

Hi  Tatsuhiro  ,

I am in need of your help too. I am trying to implement my own parsers for HTTP and SPDY. When I send the reply back to chrome it sends me a FIN. No idea why. Can I not send the SYN_REPLY and the data in the same packet?? I am specifying the length in the SYN_REPLY packet and the stream ends after the compressed name value pairs. After that I start the data frame in the same packet. and send the data received by the web server. Kindly help

Do you mean FLAG_FIN  in SYN_STREAM or DATA from Chrome by FIN?

If so, it just means that chrome does not send any further DATA frame.

Remember that SPDY stream is bidirectional and FLAG_FIN from chrome means stream is in half-closed state. The server can still send SYN_REPY and DATA.

Also check the required headers are present in the SYN_REPLY:

http://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft2#TOC-Response

Best regards,

Tatsuhiro Tsujikawa

 

Thanks in advance;

Sunil Panigrahi

On Tuesday, June 4, 2013 12:01:43 PM UTC+5:30, ramswish wrote:

Hi, I am new to C++ code. i am tried compiling C SPDY library. now i need to test that library.

i see the command like ./spdycat -vn https://www.google.com/ 

I am struggling to compile spdycat executable... i placed spdylay include folder in /usr/local/include and i tired compiling using gcc spdylay.cc 

but i am getting so many errors like 

in function spdylay::print_usage(std::basic_ostream<char, std:: char_traits<char> > &)':

like this i am gettting many errors saying .text+0x5c1 .text+0x5c2 ..etc

Can any one pls help me to solve this problem and give me the steps to setup the client and server to test C SPDY libary.

Thanks in advance

ramswish

On Wednesday, 1 February 2012 21:35:59 UTC+5:30, Tatsuhiro Tsujikawa wrote:

--
 
---
You received this message because you are subscribed to the Google Groups "spdy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spdy-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sunil Panigrahi]

Hi  Tatsuhiro  ,

I am in need of your help too. I am trying to implement my own parsers for HTTP and SPDY. When I send the reply back to chrome it sends me a FIN. No idea why. Can I not send the SYN_REPLY and the data in the same packet?? I am specifying the length in the SYN_REPLY packet and the stream ends after the compressed name value pairs. After that I start the data frame in the same packet. and send the data received by the web server. Also, as far as what I know, SPDY doesn't support Transfer encoding chunked (correct me if I am wrong). So what might be the possible consequences of sending chunked data back to chrome??? Is there any library that converts HTTP response to SPDY response. Any help highly appreciated.

Thanks in advance;

Sunil Panigrahi

[Tatsuhiro Tsujikawa]

On Thu, Jun 13, 2013 at 9:39 PM, Sunil Panigrahi <sunil.kr.pa...@gmail.com> wrote:

Hi  Tatsuhiro  ,

Thanks for the previous reply. :)

I think its a TCP fin. IP 111 is my handset and 78 is the gateway. The 1st packet received is SETTING frame followed by the SYN_STREAM. The GW is sending back the response in SYN_REPLY. I am not sure if the packet in which I am sending SYN_REPLY should contain only the headers and (compressed) name/value pairs or I can attach the data too in the same packet. Also, how about the removal of chunked data? Is it necessary? Should I just remove the "length\r\n" at the beginning and "\r\n" at the end and send the remaining data as it has come from web server or I will have to change that as well??  

The packet thing should not matter. Actually, you cannot control the number of packets along the way to the client; they may be coalesced or divided along the way.

In SPDY header block, "\r\n" does not appear in HTTP header name. It is just a boundary of HTTP/1.x headers. Did you layout name/value pairs conforming to the SPDY spec and deflate it properly?

Best regards,

Tatsuhiro Tsujikawa

[Sunil Panigrahi]

Yes I have done that, The 1st packet I am sending back to chrome is in this format:

 +----------------------------------+

  |1|        2        |        2     |

  +----------------------------------+

  | Flags (8)  |  Length (24 bits)   |

  +----------------------------------+

  |X|          Stream-ID (31bits)    |

  +----------------------------------+

  | Unused        |                  |

  +----------------                  |
  |     Name/value header block      |
  |              ...                 |

There are a total of 11 Name/value blocks and I have deflated it properly. The data past this frame is the problem I guess. The data I am receiving from the web server is a HTTP reply in transfer encoding chunked format. The next thing I am trying to do after sending the SYN_REPLY is to send the data to chrome. Should I remove the "length\r\n" in the chunk or should I send it as I receive it? I still am stuck finding the reason for the FIN I am getting. I have attached a screenshot of the sniff. 24.111 is chrome IP and 28.78 is the gateway. 1st packet from 24.111 is SETTING followed by SYN_STREAM. When 28.78 is sending back the response to 24.111(Chrome) it sends back FIN.

[Patrick McManus]

On Thu, Jun 13, 2013 at 8:39 AM, Sunil Panigrahi <sunil.kr.pa...@gmail.com> wrote:

data too in the same packet. Also, how about the removal of chunked data?

you need to remove http/1 chunking, yes.

chunked encoding is no longer valid (spdy/2)

Transfer-encoding headers are not valid and MUST not be sent. (spdy/3).. 

...

There are a number of translations between headers that need to be done to gateway http/1 back and forth through spdy correctly. "Accept-Encoding: gzip" is another one. (it is implicit in each request).

[Sunil Panigrahi]

 x-xss-protection: 1; mode=block
 x-frame-options: sameorigin
 cache-control: private, max-age=0
 connection: keep-alive
 content-encoding: gzip
 content-type: text/html; charset=utf-8
 date: thu, 13 jun 2013 13:18:12 gmt
 expires: -1
 server: gws

These are the headers I am sending. The name value pair before compression looks like this:

0 '\000'        7 '\a'  118 'v' 101 'e' 114 'r' 115 's' 105 'i' 111 'o'
 110 'n' 0 '\000'        8 '\b'  104 'h' 116 't' 116 't' 112 'p' 47 '/'
 49 '1'  46 '.'  49 '1'  0 '\000'        6 '\006'        115 's' 116 't' 97 'a'
 116 't' 117 'u' 115 's' 0 '\000'        6 '\006'        50 '2'  48 '0'  48 '0'
 32 ' '  111 'o' 107 'k' 0 '\000'        16 '\020'       120 'x' 45 '-'  120 'x'
 115 's' 115 's' 45 '-'  112 'p' 114 'r' 111 'o' 116 't' 101 'e'
 99 'c'  116 't' 105 'i' 111 'o' 110 'n' 0 '\000'        16 '\020'       49 '1'
 59 ';'  32 ' '  109 'm' 111 'o' 100 'd' 101 'e' 61 '='  98 'b'
 108 'l' 111 'o' 99 'c'  107 'k' 0 '\000'        15 '\017'       120 'x' 45 '-'
 102 'f' 114 'r' 97 'a'  109 'm' 101 'e' 45 '-'  111 'o' 112 'p'
 116 't' 105 'i' 111 'o' 110 'n' 115 's' 0 '\000'        15 '\017'       115 's'
 97 'a'  109 'm' 101 'e' 111 'o' 114 'r' 105 'i' 103 'g' 105 'i'
 110 'n' 0 '\000'        13 '\r' 99 'c'  97 'a'  99 'c'  104 'h' 101 'e'
 45 '-'  99 'c'  111 'o' 110 'n' 116 't' 114 'r' 111 'o' 108 'l'
 0 '\000'        13 '\r' 112 'p' 114 'r' 105 'i' 118 'v' 97 'a'  116 't'
 101 'e' 44 ','  32 ' '  109 'm' 97 'a'  120 'x' 45 '-'  97 'a'
 103 'g' 101 'e' 61 '='  48 '0'  0 '\000'        16 '\020'       99 'c'  111 'o'
 110 'n' 116 't' 101 'e' 110 'n' 116 't' 45 '-'  101 'e' 110 'n'
 99 'c'  111 'o' 100 'd' 105 'i' 110 'n' 103 'g' 0 '\000'        16 '\020'
 103 'g' 122 'z' 105 'i' 112 'p' 0 '\000'        12 '\f' 99 'c'  111 'o'
 110 'n' 116 't' 101 'e' 110 'n' 116 't' 45 '-'  116 't' 121 'y'
 112 'p' 101 'e' 0 '\000'        12 '\f' 116 't' 101 'e' 120 'x' 116 't'
 47 '/'  104 'h' 116 't' 109 'm' 108 'l' 59 ';'  32 ' '  99 'c'
 104 'h' 97 'a'  114 'r' 115 's' 101 'e' 116 't' 61 '='  117 'u'
 116 't' 102 'f' 45 '-'  56 '8'  0 '\000'        4 '\004'        100 'd' 97 'a'
 116 't' 101 'e' 0 '\000'        4 '\004'        116 't' 104 'h' 117 'u' 44 ','
and so on.

I think the issue is with the next packets that I am sending, I guess I will have to convert chunked response to content length format.
Is content length a mandatory header??

[Patrick McManus]

On Thu, Jun 13, 2013 at 9:28 AM, Sunil Panigrahi <sunil.kr.pa...@gmail.com> wrote:

 x-xss-protection: 1; mode=block
 x-frame-options: sameorigin
 cache-control: private, max-age=0
 connection: keep-alive
 content-encoding: gzip
 content-type: text/html; charset=utf-8
 date: thu, 13 jun 2013 13:18:12 gmt
 expires: -1
 server: gws

[..]

You can't send the connection header over spdy.

 

I think the issue is with the next packets that I am sending,

could be!

 

I guess I will have to convert chunked response to content length format.
Is content length a mandatory header??

response content-length is purely advisory. SPDY is self delimiting.

I would suggest taking the time to read http://dev.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3 (and/or draft2) from top to bottom again. There are some things that don't sink in the first time through.

hth.

[Tatsuhiro Tsujikawa]

So, you are creating http/1.1 <-> SPDY gateway..

You need to parse chunked response and only send the un-chunked response body to the client.

I don't think content-length is mandatory in SYN_REPLY. But you need to set correct length in each DATA frame you are sending back to the client.

Best regards,

Tatsuhiro Tsujikawa

 

On Thursday, June 13, 2013 6:46:01 PM UTC+5:30, Patrick McManus wrote:

On Thu, Jun 13, 2013 at 8:39 AM, Sunil Panigrahi <sunil.kr.pa...@gmail.com> wrote:

data too in the same packet. Also, how about the removal of chunked data?

you need to remove http/1 chunking, yes.

chunked encoding is no longer valid (spdy/2)

Transfer-encoding headers are not valid and MUST not be sent. (spdy/3).. 

...

There are a number of translations between headers that need to be done to gateway http/1 back and forth through spdy correctly. "Accept-Encoding: gzip" is another one. (it is implicit in each request).

[Sunil Panigrahi]

Thanks for the response,

What I finally realized is that the browser(chrome) is sending me a FIN (TCP FIN) as a response for the 1st packet(SYN_REPLY) I am sending back to the gateway. After the first packet there is another packet that I am sending which is the 1st data packet. The gateway sends me a reset for this packet as it has already sent the FIN for the 1st packet. I am sure that the packet which I am sending is in the right format(as per SPDY). What could be the possible reason for the FIN?? 

[Tatsuhiro Tsujikawa]

On Fri, Jun 14, 2013 at 2:37 PM, Sunil Panigrahi <sunil.kr.pa...@gmail.com> wrote:

Thanks for the response,

What I finally realized is that the browser(chrome) is sending me a FIN (TCP FIN) as a response for the 1st packet(SYN_REPLY) I am sending back to the gateway. After the first packet there is another packet that I am sending which is the 1st data packet. The gateway sends me a reset for this packet as it has already sent the FIN for the 1st packet. I am sure that the packet which I am sending is in the right format(as per SPDY). What could be the possible reason for the FIN?? 

That means Chrome does not like SYN_REPLY.

I found some strange things in your memory dump of headers.

For example,

0 '\000'        16 '\020'       99 'c'  111 'o'
 110 'n' 116 't' 101 'e' 110 'n' 116 't' 45 '-'  101 'e' 110 'n'
 99 'c'  111 'o' 100 'd' 105 'i' 110 'n' 103 'g' 0 '\000'        16 '\020'
 103 'g' 122 'z' 105 'i' 112 'p' 

That is supposed to be an encoded form of name/value pair: "content-encoding" and "gzip".

"content-encoding" looks good; its length 16 matches the header: 16.

But the length of "gzip" is completely messed up; it must be 3 but the above text says it is 16.

If the pasted text comes from your implementation without modification, I'm sure that something wrong in your implementation. I think you have header decoder because you need to parse

the headers from Chrome, so I recommend to write a test case which checks the encoded header block by your encoder can be parsed by the decoder.

[Sunil Panigrahi]

Hey Tatsuhiro,

I finally realized where I was going wrong. Actually I was not compressing the number of name/value pairs. In SPDY 3 this part is not compressed(as per what I understood from the draft) whereas in SPDY2 it is. Chrome must be trying to decompress the 2 bytes in which I had placed the number of name/value(which was already uncompressed). The code finally worked. Now I think I can manage with the rest of it(hopefully).

Thanks for the continuous help. It is highly appreciated. :)

[Patrick McManus]

On Wed, Jun 19, 2013 at 8:12 AM, Sunil Panigrahi <sunil.pan...@gmail.com> wrote:

Hey Tatsuhiro,

I finally realized where I was going wrong. Actually I was not compressing the number of name/value pairs. In SPDY 3 this part is not compressed(as per what I understood from the draft) whereas in SPDY2 it is.

The number of name/value pairs is part of the compressed name/value header block in both spdy/2 and spdy/3.

[Ting OneDing]

Hello Tatsuhiro,

I am trying to compile your spdycat for Android, but have been unsuccessful. I read: "The dependent libraries, such as OpenSSL and libevent should be built with the toolchain and installed under$ANDROID_HOME/usr/local. We recommend to build these libraries as static library to make the deployment easier." on https://github.com/tatsuhiro-t/spdylay but I don't know how to compile these libraries. I did not find any detailed instructions on the site. Can you help? I just need the binary to do some testing. If you could send me the binary right away that would be great too.

Thanks a lot!

Best, 

Ting

[laila daniel]

Hello

Kindly let me know where I should change in the spdylay code to implement TCP Fast Open (TFO)?

The changes to the server and the client for TFO is given in this article
https://lwn.net/Articles/508865/

In the make_listen_socket(const std::string& host, uint16_t port, int family)

between the bind and listen I placed the following code for server

 int qlen = 5;                            // Value to be chosen by application
    setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));

But I do not know how to place

sendto(sfd, data, data_len, MSG_FASTOPEN, 
 (struct sockaddr *) &server_addr, addr_len);

Kindly give some advice
Best regards
laila daniel