GSoC Project: Support for WebSockets

[robkoch86]

Dear members,

my name is Robert Koch and I was selected by Mozilla for this Google
Summer of Code. I'm a master student on the Technical University of
Vienna, Austria. My task is adding support for WebSockets into ZAP.

The upcoming WebSockets standard (http://dev.w3.org/html5/websockets/
| http://tools.ietf.org/html/rfc6455) allows for bi-directional
communication with very low overhead. The protocol is based on TCP.

I want my outcome to have the following features:
* capturing WebSocket-messages
- messages are collected from various fragments
- dealing not only with “ws://” but also with secure “wss://”
WebSocket connections
* display messages
- filterable by
- out- & ingoing messages
- message type: beside text frames, there are also binary frames
and several control frames (ping/pong/close).
* breakpoints:
- set via sites panel, where WS connection is shown
- set some custom Regex, that has to be fulfilled by a message in
order to trigger the breakpoint
- set via arrows “all incoming messages” and/or “all outgoing
messages”
* filters:
- adding filters like the “Replace HTTP request body using defined
pattern” for WebSockets as “Replace outgoing WebSockets messages using
defined pattern”
- another filter for incoming messages
* various options for the settings dialog (as they can be foreseen
now) to:
- capture only specific types – e.g.: no ping/pong
- capture blacklist – avoid capturing WebSocket messages for high-
volume communication on specific domains (could also be implemented as
opt-in, i.e. whitelist)
* inject arbitrary messages (e.g. to close a connection)


I will setup a status site soon, such that everybody can follow
progress. I am also open for advices and comments.
Best wishes,
Robert

[robkoch86]

I used this day to read into the code. I knew that the response of the HTTP Upgrade request (that is done for WebSockets as shown in https://en.wikipedia.org/wiki/WebSockets#WebSocket_protocol_handshake) is not processed as there is some code in the HTTPMethodBase class of the HttpClient-library (http://hc.apache.org/httpclient-3.x/) ignoring status codes between 99 & 200:
            int status = this.statusLine.getStatusCode();
            if ((status >= 100) && (status < 200)) {
                if (LOG.isInfoEnabled()) {
                    LOG.info("Discarding unexpected response: " + this.statusLine.toString());
                }
                this.statusLine = null;
            }

It times out after a while and the response with status code 101 for switching protocols is ignored. My solution proposals:

1. First, I thought that updating to the current version of the HttpClient library could solve this problem. Unfortunately it is not API compatible (it was also renamed to Apache HttpComponents), which would require a lot of work for rewriting Paros.
   
2. Extending the class GetMethod was the next idea. Unfortunately the attribute statusLine is private. I looked up the latest version of HttpClient 3.x (3.1 in fact, while 3.0 is in use) and it has got that attribute protected, which would be a viable option for extension.
3. Use custom http client classes or the current version of HttpComponents (http://hc.apache.org/) only for WebSockets connection. This would require a lot of work too, as the answer to a WebSockets Upgrade-request could also be a Redirect (3xx) or a Failure (4xx) instead of a Switching Protocol (101). Moreover I do not know if further steps in connection handling are easier to be done with version 4.x instead of version 3.1.
   

My favourite is proposal 2. However, I do not know if I could use this way to establish a WebSockets connection, but I have to examine that in small steps. The next step in thinking through the stuff will be to keep the TCP connection open after the request and handle it over to some custom class (that finishes the handshake on WebSockets level).

What do you think about that? I need your help. Is an upgrade from HttpClient 3.0 to 3.1 a viable option?

Best wishes Robert

[robkoch86]

I replaced the commons-httpclient-3.0.jar with ommons-httpclient-3.1.jar. I had to catch an CloneNotSupportedException whenever an org.apache.commons.httpclient.URI was cloned. Then I was able to compile and run the proxy. Requests and responses got tracked so far, but I did not test it in detail. The changed methods can be looked up in the release notes at: http://archive.apache.org/dist/httpcomponents/commons-httpclient/RELEASE_NOTES.txt

As a result I was able to exchange the GetMethod class with my own ZapGetMethod that allows status code 101. I would like to know if I/we can go this way of upgrading to (the still old) version 3.1?

[thc202]

Hi.

The proposal #2 is the one that requires less time/effort, so I think that one
is the best (for now).

Updating Commons Httpclient from version 3.0 to 3.1 is viable, there is only
the need to change a couple of classes, so I think we could give it a try.

Have you take a look into the Monsoon project [1] in the Apache Labs? maybe it
could be helpful, though it uses HttpComponents Client.
Monsoon is now on Google Project Hosting [2] and has dropped the use of
HttpComponents Client (it came from other project [3] than the one of the
Apache Labs).

[1] https://svn.apache.org/repos/asf/labs/monsoon/
[2] https://code.google.com/p/monsoon
[3] https://code.google.com/p/websocket-draft-eval/


Best regards.

[psiinon]

I agree - updating HttpClient to 3.1 looks like the easiest option and it would be well worth having a good look at the Monsoon project.

Cheers,

Simon

[robkoch86]

Ok, fine. Thanks for your response.

Yeah, I will definitely take a look at Monsoon - thank you thc202 for that tip. I stumbled over various other implementations, such as Atmosphere (https://github.com/Atmosphere/atmosphere).

Best wishes
Robert

[robkoch86]

In the last days I designed the initial part of the WebSockets extension. Unfortunately I ran into problems. Starting the class WebSocketsThread in non threaded mode in ExtensionWebSocket on line 223 works. When I run the echo demo from http://www.websocket.org/echo.html, communication is recognized (see DEBUG output on command line). But If I use a Thread to start this processing, everything fails. Could someone please take a look at the code in the WebSockets-branch (http://code.google.com/p/zaproxy/source/detail?r=1426). I'd be glad for any advice what I can do to find out what is going wrong. Maybe I have missed something with Java's thread model.

Best wishes,
Robert

[thc202]

Hi.

Could you provide the steps that you have used to run the echo demo?

Best regards.

[robkoch86]

Yes, of course.

In order to see debug output, you have to set the following lines in the log4j.properties file:

log4j.logger.org.parosproxy.paros=DEBUG
log4j.logger.org.zaproxy.zap=DEBUG

My related log4j file is placed at ~/.ZAP/log4j.properties. Then start ZAP, redirect the traffic from the browser through port 8080 and surf to: http://www.websocket.org/echo.html

After clicking on the "connect" button you can read on the command line (or console in eclipse) something like:

250424 [ZAP-ProxyThread] DEBUG org.parosproxy.paros.core.proxy.ProxyThread  - Got WebSockets upgrade request. Handle socket connection over to WebSockets extension.
250424 [ZAP-ProxyThread] DEBUG org.zaproxy.zap.extension.websocket.ExtensionWebSocket  - Got WebSockets channel from /127.0.0.1 port 42420 to echo.websocket.org/174.129.224.73 port 80
250424 [ZAP-ProxyThread] DEBUG org.zaproxy.zap.extension.websocket.ExtensionWebSocket  - No Sec-Websocket-Version header was provided - try version 13
250432 [ZAP-ProxyThread] DEBUG org.zaproxy.zap.extension.websocket.WebSocketProxy  - Create WebSockets proxy for version '13'.
250434 [ZAP-ProxyThread] DEBUG org.zaproxy.zap.extension.websocket.WebSocketProxy  - Register this WebSocket-proxy on given selector.

After clicking the "Send"-button further debug output is print:

337697 [ZAP-ProxyThread] DEBUG org.zaproxy.zap.extension.websocket.WebSocketProxy  - Got WebSockets-frame: 1
337697 [ZAP-ProxyThread] DEBUG org.zaproxy.zap.extension.websocket.WebSocketProxyV13  - Length of current frame payload is: 28
337698 [ZAP-ProxyThread] INFO org.zaproxy.zap.extension.websocket.WebSocketProxyV13  - got payload: Rock it with HTML5 WebSocket

and for the echoed message of the server:

337897 [ZAP-ProxyThread] DEBUG org.zaproxy.zap.extension.websocket.WebSocketProxy  - Got WebSockets-frame: 1
337897 [ZAP-ProxyThread] DEBUG org.zaproxy.zap.extension.websocket.WebSocketProxyV13  - Length of current frame payload is: 28
337898 [ZAP-ProxyThread] INFO org.zaproxy.zap.extension.websocket.WebSocketProxyV13  - got payload: Rock it with HTML5 WebSocket


When using the threaded version, the connection fails (as reported in the echo-demo of the browser) and no message runs through the proxy.

[thc202]

Thanks.

I've asked for the steps because I'm getting some exceptions while trying to test the echo demo (doesn't happens always).

Sometimes Mozilla Firefox tries to "CONNECT":

 CONNECT echo.websocket.org:80 HTTP/1.1
 User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:12.0) Gecko/20100101 Firefox/12.0
 Proxy-Connection: keep-alive
 Host: echo.websocket.org

Leading to an exception: "javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?".

Other times is doesn't and the request is correctly handled by ZAP, it sends:

 GET http://echo.websocket.org/?encoding=text HTTP/1.1
 Host: echo.websocket.org
 User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:12.0) Gecko/20100101 Firefox/12.0
 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
 Accept-Language: en
 Accept-Encoding: gzip, deflate
 Sec-WebSocket-Version: 13
 Origin: http://www.websocket.org
 Sec-WebSocket-Key: x3JJHMbDL1EzLkh9GBhXDw==
 Pragma: no-cache
 Cache-Control: no-cache
 Upgrade: websocket
 Connection: Upgrade

Anyway, regarding the issue you have pointed, the "problem" happens (the channel "browser<->ZAP" is closed) when the stream (httpIn) is closed in the method ProxyThread.disconnect() that is executed after the thread WebSocketsThread is started.
Probably because of (from the documentation of the method java.io.Closeable.close()):
 "Closes this stream and releases any system resources associated with it."

I hope it helps.

Best regards.

[robkoch86]

On Thursday, May 10, 2012 9:20:11 PM UTC+2, thc202 wrote:

Sometimes Mozilla Firefox tries to "CONNECT":

 CONNECT echo.websocket.org:80 HTTP/1.1
 User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:12.0) Gecko/20100101 Firefox/12.0
 Proxy-Connection: keep-alive
 Host: echo.websocket.org

It is interesting that it chooses a random method. I'm aware of the fact that CONNECT is a valid way for setting up a WebSockets connection. However, I thought that the browser won't do it in the case for a transparent proxy (like the one configured in Firefox).

From RFC6455:

If the client is configured to use a proxy when
using the WebSocket Protocol to connect to host /host/ and port
/port/, then the client SHOULD connect to that proxy and ask it
to open a TCP connection to the host given by /host/ and the port
given by /port/.

For now, I have not take care about this connection method. I will add it to my todo list.

Anyway, regarding the issue you have pointed, the "problem" happens (the channel "browser<->ZAP" is closed) when the stream (httpIn) is closed in the method ProxyThread.disconnect() that is executed after the thread WebSocketsThread is started.
Probably because of (from the documentation of the method java.io.Closeable.close()):
 "Closes this stream and releases any system resources associated with it."

Thank you, that helped me a lot! Previously I thought that keeping the socket open, would suffice as I did:

 // ZAP: Keep socket connection from browser <-> ZAP alive if boolean set
if (keepSocketAfterDisconnect == false) {
    HttpUtil.closeSocket(inSocket);
}

Now I do not call the whole disconnect() method from ProxyThread anymore after a valid WebSockets handshake was done and it works fine.

Thanks a lot!

[robkoch86]

Hello,

I got stuck with my implementation of WebSockets support in ZAP.

What I've done so far (which works):

Created sockets via SocketChannel.open().socket() that allows me to retrieve the SocketChannel later and use that for processing WebSockets traffic. As a result I can take advantage of Java's NIO features. I do only need one thread for all WebSocket connections. There is no blocking for in- or outgoing messages.
So I collect these channels in case there is a successful handshake and pass it to my extension in the ProxyThread class. It works and you can even play BrowserQuest (http://browserquest.mozilla.org/) as all WebSocket frames are forwarded. This version is already checked in into the WebSockets-branch.

The problem:

Now I wanted to add support for secure WebSockets connections over SSL. Unfortunately I have got huge problems. A SSLSocket is used to wrap the former socket and manages all SSL stuff. I'm still able to retrieve the SocketChannel from this SSLSocket, but there is no such SSLSocketChannel class. I would have to take care about SSL myself using the SSLEngine class. Here I'm stuck. I was not able to get the SSL working at this point. I searched Google for 3 days now and I was not able to find out how an existing SSLSocket connection can be used further with a SocketChannel+SSLEngine. Is there someone out there, that has experience with SSL and/or NIO?

Possible solutions:

1. Take over SSL handling from SSLSocket with SSLEngine (I did not find anything that goes in that direction).
2. Use a custom SSLSocket class that uses an SSLEngine internally (I did not find any implementation of that - I even do not know if that is possible).
   
3. Do not use Java's NIO features for processing WebSocket-messaging. Would require to spawn more threads as I would have to use blocking reads.
   
4. Rewrite core to NIO and avoid using Sockets
• use of Netty that would also support SPDY - http://www.javacodegeeks.com/2012/05/netty-using-spdy-and-http-transparently.html
• upgrade to HTTPComponents (current version 4.2 has NIO support)
  

If no one comes up with a solution or advice, I am forced to continue with solution#3. Contact me if you want to take a look at the source code.

Best wishes,

Robert

[robkoch86]

As it is hard to see the problem, I've provided a standalone example that does a WebSockets handshake via an SSLSocket and then wants to send a ping frame over the wrapped SocketChannel.

You can find it here: http://pastebin.com/vqZ3iwWp
Best wishes
Robert

[robkoch86]

As always, after posting I got one step further. I managed to reuse the existing session, but the error in the end remains the same :-(

Updated paste: http://pastebin.com/PpCMr072

[thc202]

Hi.

Have you take a look at http://docs.oracle.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#SSLENG ?

Shouldn't the data read from the channel, that is being done in the Read class, be unwrapped?

Best regards.

[robkoch86]

Yes, you're right. I had to call the read not on the channel directly, but on the wrapped object. Unfortunately this makes no difference. The first call to unwrap for the write, triggers a ClientHello. I think that this is done due to the initial (uninitialized) state of the SSLEngine. However, after endless trial&error I think that resuming an existing session (from an SSLSocket) with SSLEngine is not possible. The answer to the resendt ClientHello is always some TLSv1Alert, resulting in an error.

I wanted to use Java's NIO features as I thought that would be the way to go. However, it turns out that I have to use blocking I/O for my task.
Thx Robert

[robkoch86]

Dear members,

I'm happy to announce the merge of WebSocket features into trunk. Feel free to try it out. You can visit Mozilla's BrowserQuest to view communication over WebSockets in ZAP.

Note: I had to upgrade the library commons-httpclient from version 3.0 to 3.1. So you have to re-configure the Build Path in Eclipse first.

Although my implementation is not yet feature complete, I want to give our community the chance to give some feedback before the Google Summer of Code ends - don't worry, I want to stay in the community afterwards to maintain & improve the new features.

A list of what is missing so far, can be found under: http://code.google.com/p/zaproxy/wiki/GSoC2012_WebSockets#Weekly_Status_Updates

If you've got some questions, recommendations, bug reports, etc. - feel free to contact me or open a new issue.

Best wishes,
Robert

[psiinon]

Thats great :D

The next major version of ZAP will have some really good enhancements, mostly thanks to the 3 GSOC projects.

Thanks for all of your hard work on this Robert!

Simon

[robkoch86]

Dear members,

some minutes ago I have merged my WebSocket branch into trunk again.

What has changed since the last merge in revision r1953 from July 23, 2012?

There was a change that will affect all of us: I've upgraded to HSQLDB version 2.2.9 (former it was 1.8). It has real support for BLOBs & CLOBs. Moreover its queries should execute faster. As a result of the upgrade, it might happen, that some columns are to small (size definitions are now required). Please report any bugs.

Regarding WebSockets, there are some new features:
 - Fuzzing Support: Select some payload and fuzz it like you know it from fuzzing HTTP requests.
 - Options: Control how breakpoints are set.
 - Session Properties:
   # Exclude specific WebSocket channels from ZAP
   # Support for new Scope & Mode
 - Support for binary messages

As always, please report any issues you've experienced via the issues list, Google Group or just drop me a mail. I hope you like the new feature set and I'm looking forward to any kind of feedback.

Last but not least, a big thanks to thc202 for preparing some HTTP specific core parts to be able to deal with WebSocket messages.

Best wishes,
Robert