How to implement a new server side?

[Ngoc Dao]

For the server side, SockJS seems to be easiser to implement than
Socket.IO.

I'd like to add SockJS feature to Xitrum (http://
ngocdaothanh.github.com/xitrum/).
Is there a doc (simple, short is OK) on how to implement a new server
side?
I've searched but couldn't find any.

Thanks.

[Fugiman]

https://github.com/sockjs/sockjs-protocol defines how a server should work. Read that a few times and you should be able to come up with an implementation ;)

[Ngoc Dao]

Thanks a lot.
This is beautiful:
http://sockjs.github.com/sockjs-protocol/sockjs-protocol-0.3.html

[Carl Byström]

The tests included in the protocol are great. Running them while coding your server is very helpful.

I've ported SockJS to Java/Netty this way.

--

Carl

[Ngoc Dao]

Thank you for your instructions guys.

I have implemented SockJS for my web framework like this:
https://github.com/ngocdaothanh/xitrum/tree/master/src/main/scala/xitrum/sockjs

More info:
http://ngocdaothanh.github.com/xitrum/guide/async.html#sockjs
http://groups.google.com/group/xitrum-framework/browse_thread/thread/59a0c07d092e0c21

[Marek Majkowski]

Cool, were you able to run SockJS protocol tests against your server?

Marek

[Ngoc Dao]

> Cool, were you able to run SockJS protocol tests against your server?

The spec and tests are very useful to understand SockJS.

Currently I only use the echo test. I do not use others like
disabled_websocket_echo and cookie_needed_echo, because currently I
always set websocket = true and cookie_needed = false. I will allow
customizing those options in later version of Xitrum web framework.

[Marek Majkowski]

Ok, please try to implement other services also and attempt to
pass all the sockjs protocol tests. This is important for interoperability,
and generally, it's likely that you missed something important
if you don't pass all the tests.

Marek

[Ngoc Dao]

> Ok, please try to implement other services also and attempt to
> pass all the sockjs protocol tests. This is important for interoperability,

I have this problem when trying to pass
HandlingClose.test_abort_xhr_streaming and
HandlingClose.test_abort_xhr_polling:
http://sockjs.github.com/sockjs-protocol/sockjs-protocol-0.3.3.html#section-149

The test client closes the first connection:
r1.close()

Then immediately establishes another connection:
r3 = POST_async(url + '/xhr_streaming')

And asserts that this new connection should not be "Another connection
still open". But due to TCP nature, the server does not immediately
know that the first connection has been closed!

I'm using Netty on Mac. Netty only know that the first connection has
been closed when it tries to send the keep alive message to the first
connection (after 25 seconds).

My question is: Are the two "abort" tests themselfves broken? Is there
anything I can do to pass the tests?

Thanks.

[Ngoc Dao]

I have another problem.

When running these tests against my server on Mac and Ubuntu:
XhrStreaming.test_response_limit
EventSource.test_response_limit
HtmlFile.test_response_limit

The tests never end and the Python test client process consumes 100%
CPU.

I have manually checked my server side with telnet and curl. My server
side does close connection after sending messages and sent bytes
exceed 4KB.

You can reproduce the problem:

Install latest Xitrum SNAPSHOT:
git clone git://github.com/ngocdaothanh/xitrum.git
cd xitrum
sbt/sbt publish-local

Run the test server:
git clone git://github.com/ngocdaothanh/xitrum-sockjs-test.git
cd xitrum-sockjs-test
sbt/sbt run

There's no problem when running against the Nodejs server:
https://github.com/sockjs/sockjs-node

But I still guess there's problem with the Python test client.
Specifically, the cause is "r.read()" below, it does not return:

# The connection should be closed after enough data was
# delivered.
self.assertFalse(r.read())

[Marek Majkowski]

On Sat, Nov 17, 2012 at 7:37 AM, Ngoc Dao <ngocda...@gmail.com> wrote:
>> Ok, please try to implement other services also and attempt to
>> pass all the sockjs protocol tests. This is important for interoperability,
>
> I have this problem when trying to pass
> HandlingClose.test_abort_xhr_streaming and
> HandlingClose.test_abort_xhr_polling:
> http://sockjs.github.com/sockjs-protocol/sockjs-protocol-0.3.3.html#section-149
>
> The test client closes the first connection:
> r1.close()
>
> Then immediately establishes another connection:
> r3 = POST_async(url + '/xhr_streaming')
>
> And asserts that this new connection should not be "Another connection
> still open". But due to TCP nature, the server does not immediately
> know that the first connection has been closed!

Yes, but it will be closed half a millisecond later.

> I'm using Netty on Mac. Netty only know that the first connection has
> been closed when it tries to send the keep alive message to the first
> connection (after 25 seconds).

That's a bug in Netty, sorry. You should be able to react to network
connection being broken immediately. It's possible (with a bit of hackery)
in node, python, erlang, etc. It must be possible in netty.

> My question is: Are the two "abort" tests themselfves broken? Is there
> anything I can do to pass the tests?

Yes. Make Netty react to an event when underlying network connection
gets closed.

Use Wireshark / Tcpdump to see when actually the FIN packet was sent
by the protocol tests.

Marek

[Marek Majkowski]

On Mon, Nov 19, 2012 at 2:39 AM, Ngoc Dao <ngocda...@gmail.com> wrote:
> I have another problem.
>
> When running these tests against my server on Mac and Ubuntu:
> XhrStreaming.test_response_limit
> EventSource.test_response_limit
> HtmlFile.test_response_limit
>
> The tests never end and the Python test client process consumes 100%
> CPU.
>
> I have manually checked my server side with telnet and curl. My server
> side does close connection after sending messages and sent bytes
> exceed 4KB.

Okay... Are curl is sending exactly the request that
sockjs-protocol sends?

> You can reproduce the problem:
>
> Install latest Xitrum SNAPSHOT:
> git clone git://github.com/ngocdaothanh/xitrum.git
> cd xitrum
> sbt/sbt publish-local
>
> Run the test server:
> git clone git://github.com/ngocdaothanh/xitrum-sockjs-test.git
> cd xitrum-sockjs-test
> sbt/sbt run
>
> There's no problem when running against the Nodejs server:
> https://github.com/sockjs/sockjs-node
>
> But I still guess there's problem with the Python test client.
> Specifically, the cause is "r.read()" below, it does not return:

If it doesn't return than why it uses 100% CPU?

> # The connection should be closed after enough data was
> # delivered.
> self.assertFalse(r.read())

Again, please use tools to see what's really happening on the
wire and why your server reacts differently than sockjs-node.
I recommend Tcpdump / Wireshark, or even better Tcpflow.

Marek

[Ngoc Dao]

> I recommend Tcpdump / Wireshark, or even better Tcpflow.

Thanks for your advice.
I'm new to Tcpflow but trying to use it.

I'm using this on Ubuntu:
sudo tcpflow -v -i any port 8081

Its output does not show any information about "close" event. I'm
searching but haven't found the command option that displays that
information.

Can you show me the Tcpflow command that should be used?

Thanks.

[Marek Majkowski]

Tcpflow saves tcp/ip streams in files. Run tcpflow
and run a single sockjs-protocol test against sockjs-node,
then repeat but against your server.

That way you'll be able to compare what was the difference
between response of sockjs-node and your server.

Please show us four files:
- tcpflow inboud and outbound results of running
a single test of sockjs-protocol that fails against
your server
- the same but against sockjs-node (and passes)

If you want to see tcp/ip packets, use tcpdump, for
example:
tcpdump -s0 -n -i any port 8081 -X

And take a deep look at FIN packets.

Marek

[Ngoc Dao]

Marek,
Thanks a lot, your advice is super useful!

About the "test_response_limit" problem:

After investigating with tcpflow, I found out that before closing the
connection, I forgot to send the last chunk (the "0" chunk, as in this
example: http://en.wikipedia.org/wiki/Chunked_transfer_encoding#Encoded_data).

[Marek Majkowski]

Well done! :)

Marek

[Ngoc Dao]

There are only 2 tests that my server side implementation still has
not passed:
* test_xhr_server_encodes
* test_haproxy

I have 2 questions:
* What's the implication of not passing the above 2 tests? For
example, will clients have problem with decoding some messages?
* How to make a Java server pass the test_xhr_server_encodes?

Thanks.

[Marek Majkowski]

You can safely ignore test_haproxy if you don't care about running
your service behind a load balancer. Or if you don't care about
running websockets behind a loadbalancer.
This test failing means that iPads / iPhones with iOS < 6 won't
be able to use ws if your server is behind a good load balancer.

test_xhr_server_encodes is much worse - failing that means
that if an application sends a valid unicode character, it will
be mangled by the browser and delivered as something else.

To pass this test you usually need to run a regular expression
on top of encoded sockjs frames and replace "invalid" unicode
characters with their "\uXXXX" replacement. It's quite simple:
https://github.com/sockjs/sockjs-node/blob/master/src/utils.coffee#L87-L109

Marek

[Ngoc Dao]

> https://github.com/sockjs/sockjs-node/blob/master/src/utils.coffee#L87-L109

Thanks a lot. My implementation can pass the test_xhr_server_encodes
now. But to make sure, because you write "The browser must escape
quite a list of chars, this is due to browser mangling outgoing chars
on transports like XHR.", I want to ask that in addition to XHR, what
transports should be quoted?

[Marek Majkowski]

The escaping routine is the same for every transport. This is for
simplicity. Escaping weird characters is for sure required for
all xhr-based transports, jsonp and sse. Websockets might
work without the weird escaping but we do it for the sake
of consistency.

[Ngoc Dao]

I want to report that can the SockJS implementation in Xitrum can now
pass the abort tests:
https://github.com/ngocdaothanh/xitrum-sockjs-test

As a result, the only test left that Xitrum still fails is
WebsocketHixie76.test_haproxy. I've sent pull requests to include
Xitrum to the list of SockJS implementations:
https://github.com/sockjs/sockjs-protocol/pull/71
https://github.com/sockjs/sockjs-client/pull/115

[serg.s...@gmail.com]

Greetings,

Is there implementation for Jetty or Tomcat?

[Chandan Pandey]

Spring has an implementation for tomcat/jetty. .in fact any container which supports servlet spec 3.0 and higher. .