FYI: Enabling WebSockets when putting Sync Gateway behind nginx

[Jens Alfke]

If you put your Sync Gateway(s) behind an nginx proxy, you'll need to configure nginx to pass WebSocket connections. Andrew Reslan found instructions here on the nginx website.

(Come to think of it, you'll also need to configure nginx to pass through the Server: response header, otherwise Couchbase Lite won't know it's talking to Sync Gateway and won't even try to use WebSockets (nor several other replication optimizations.)

We should have an article in our docs on using nginx with Sync Gateway, but for now I'm posting this.

—Jens

[Jeremy Kelley]

I've been wrestling with putting the sync gateway behind nginx today and am having no luck.

Did anyone actually ever get this working?  Any pointers you can share?

Thanks,

Jeremy

[Traun Leyden]

A related ticket in case you haven't seen it:

https://github.com/couchbase/sync_gateway/issues/439#issuecomment-57867740

Afaik Andy Reslan should be adding this soon.

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/4234717f-e7d1-47dc-8ab0-0bd280e17d37%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeremy Kelley]

I actually just got it working. I'll paste a gist soon to help others. 

Sent from my iPad

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/8C72996D-76DE-472B-AF43-80BC4C6C747B%40gmail.com.

[Jeremy Kelley]

Here's the location stanzas I'm using with nginx in front of sync gateway.  Hope it's helpful to someone.

https://gist.github.com/nod/0e12567203e94e3646c1

-j

On Saturday, October 4, 2014 7:35:04 PM UTC-5, Jens Alfke wrote:

[Marcus Roberts]

On Thursday, 23 October 2014 05:35:08 UTC+1, Jeremy Kelley wrote:

Here's the location stanzas I'm using with nginx in front of sync gateway.  Hope it's helpful to someone.

https://gist.github.com/nod/0e12567203e94e3646c1

-j

Thanks for this.  It works great for http.  Has anyone got it working with https with nginx being the SSL endpoint?    From what I can see of my similar config (but with ssl switched on) the first web socket messages go through, but later ones seem to get logged as new connections (a bunch of encrypted rubbish appears in the logs) instead of carrying on the same connection (within seconds so it's not a proxy timeout issue).  If I don't pass the Server header back, the app doesn't use web sockets and everything works fine.  It also works great over http, just https is the problem.

[Jeremy Kelley]

HTTPS is on my "list" to do in the next couple of days. Haven't
tackled that yet, as we're not rolled out in production.

> --
> You received this message because you are subscribed to the Google Groups
> "Couchbase Mobile" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mobile-couchba...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mobile-couchbase/247b3b63-1b14-45b3-8097-29ee45310316%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.



--

The Christian ideal has not been tried and found wanting;
it has been found difficult and left untried – G. K. Chesterton

[ajres]

@Marcus

I have been collating nginx config from @Jeremy here and our own demo cluster as input to a wiki page on nginx configuration for sync_gateway.

But right now I am facing a similar issue to you, SSL works great, Websockets work great, but Websockets over SSL results in the following errors in my Couchbase Lite client:

CFNetwork SSLHandshake failed (-9847)

I'll post here as soon as I have a solution.

Andy

[Jeremy Kelley]

I also should mention I've been considering doing ssl termination with
HAProxy in front of couchbase, nginx, etc.

Does anyone have any experience putting the sync gateway behind HAProxy?

-j

> --
> You received this message because you are subscribed to the Google Groups
> "Couchbase Mobile" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mobile-couchba...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mobile-couchbase/e2e8c80c-acc2-435b-868b-c7b01dbf7550%40googlegroups.com.

[Marcus Roberts]

But right now I am facing a similar issue to you, SSL works great, Websockets work great, but Websockets over SSL results in the following errors in my Couchbase Lite client:

CFNetwork SSLHandshake failed (-9847)

I'll post here as soon as I have a solution.

Andy, that's the exact same error message as me.  If you remove the  Server proxy pass so that it falls back to long polling it works fine.

From the nginx logs, my reading of it is that the follow up web socket messages are seen as new connections, because I get a bunch of what looks like encrypted data, and to it responds with a 400 error.

From my reading around there is a wss:// setting which I think means web-socket secure, and I wonder if we need to be setting web sockets into a secure mode at a lower level (i.e. inside the lite lib) rather than just sending it over an SSL connection.

[Jens Alfke]

On Oct 27, 2014, at 12:32 AM, Marcus Roberts <marcus....@gmail.com> wrote:

If I don't pass the Server header back, the app doesn't use web sockets and everything works fine.  

FYI (in case I haven't mentioned it on this thread yet) you can explicitly disable WebSockets in Couchbase Lite/iOS by setting the CBLReplicator's customProperties to @{@"websocket": @NO}. This is a better workaround than not passing through the Server header, because it will let CBL use its other server-specific replication optimizations.

—Jens

[Marcus Roberts]

On Monday, 27 October 2014 14:03:55 UTC, Jeremy Kelley wrote:

I also should mention I've been considering doing ssl termination with
HAProxy in front of couchbase, nginx, etc.

Does anyone have any experience putting the sync gateway behind HAProxy?

Not yet, but that was going to be my next thing to test.  Let me know if you have any success and I will do the same. 

[Marcus Roberts]

FYI (in case I haven't mentioned it on this thread yet) you can explicitly disable WebSockets in Couchbase Lite/iOS by setting the CBLReplicator's customProperties to @{@"websocket": @NO}. This is a better workaround than not passing through the Server header, because it will let CBL use its other server-specific replication optimizations.

Thanks Jens, that's good to know.  I think for the moment SSL is going to outweigh the advantages of web sockets so I will do that, but I'd like to get both working if I can, so I'll let you know what I find out. 

[Jens Alfke]

On Oct 27, 2014, at 7:00 AM, ajres <an...@couchbase.com> wrote:

But right now I am facing a similar issue to you, SSL works great, Websockets work great, but Websockets over SSL results in the following errors in my Couchbase Lite client:

CFNetwork SSLHandshake failed (-9847)

I'll post here as soon as I have a solution.

Hm, I do have a unit test in CBL for WebSocket-over-SSL direct to Sync Gateway, so the issue must be specific to the proxy.

This may have something to do with the SSL/TLS protocol version negotiation during the handshake, although the logic should be the same for the WebSocket as for the default CBLSocketChangeTracker — as of 1.0.3, both accept only TLS 1.0.

I just looked up error -9847 in <SecureTransport.h> — it's "errSSLRecordOverflow". No idea what that means, except that it sounds like a low level protocol error…

Another thing to try would be to find some other WebSocket client (maybe a tiny node.js script) and have it connect to SG in the same configuration.

—Jens

PS: I guess this is covered under #471? But this seems more like a bug than a lack of docs; SSL should work.

[Jens Alfke]

On Oct 27, 2014, at 9:13 AM, Marcus Roberts <marcus....@gmail.com> wrote:

From my reading around there is a wss:// setting which I think means web-socket secure,

I believe that's just a URL scheme used to indicate WebSocket-over-SSL.

Couchbase Lite's WebSocket library doesn't need a special URL scheme, though, it just takes a regular http: or https: URL.

and I wonder if we need to be setting web sockets into a secure mode at a lower level (i.e. inside the lite lib) rather than just sending it over an SSL connection.

WebSocket-over-SSL should work just like regular HTTP, since it opens the connection in HTTP mode and then uses the Upgrade: header to negotiate a switch to the WebSocket protocol.  This is supposed to be invisible to a proxy, if the proxy is just passing through the undecrypted SSL traffic to the server — the proxy doesn't even know there's WebSockets going on.

So perhaps the issue here is that (I think) you're instead using nginx to do the SSL encryption itself? In that case nginx will need to manage the WebSocket protocol, open a WebSocket connection to SG, and relay messages back and forth. Maybe it needs special configuration to do that?

—Jens

[ajres]

I have created ticket #472 to track this issue.

[Marcus Roberts]

WebSocket-over-SSL should work just like regular HTTP, since it opens the connection in HTTP mode and then uses the Upgrade: header to negotiate a switch to the WebSocket protocol.  This is supposed to be invisible to a proxy, if the proxy is just passing through the undecrypted SSL traffic to the server — the proxy doesn't even know there's WebSockets going on.

So perhaps the issue here is that (I think) you're instead using nginx to do the SSL encryption itself? In that case nginx will need to manage the WebSocket protocol, open a WebSocket connection to SG, and relay messages back and forth. Maybe it needs special configuration to do that?

We are using nginx to do the SSL decryption/encryption.   Nginx is supposed to handle the websocket - you used to have to make a tcp proxy configuration but nginx does support the protocol natively now - if you switch to non-ssl it works perfectly with the web socket packets even when proxying.    I'm sure the issue here is nginx rather than sync gateway / couchbase lite. 

I suspect the issue is that nginx is missing some inspection of the later packets (because they are encrypted) and sees them as a new connection.  Doing a little research that specific iOS error is often triggered when an SSL session is mistakenly delivered to port 80 and then redirected to port 444 - the HTTP redirect is seen as bogus data and causes the overflow.  It's not the same issue here, but something similar.   

I'm away from home until tomorrow but I'll do some tcpdump and then take this to the nginx list and see if we can find the issue there as I'm sure the issue is a proxy one.  I'll update you on the ticket  you created.

[Marcus Roberts]

On Monday, 27 October 2014 16:37:55 UTC, Jens Alfke wrote:

WebSocket-over-SSL should work just like regular HTTP, since it opens the connection in HTTP mode and then uses the Upgrade: header to negotiate a switch to the WebSocket protocol.  This is supposed to be invisible to a proxy, if the proxy is just passing through the undecrypted SSL traffic to the server — the proxy doesn't even know there's WebSockets going on.

I had a quick try of doing it this way - specifying https in the connection, and proxying that through nginx to a gateway with the SSL cert and key specified in the config file.  ngninx doesn't see any of the http content, just logs this sort of thing:

85.13.70.247 - - [27/Oct/2014:22:50:22 +0000] "\x16\x03\x01\x00\xC0\x01\x00\x00\xBC\x03\x03TN\xCC0\xF8\xB9\xE1\x02\x88\xFF7\x1A,\xB8\x22\x09\xA1'\xF75\xFF\x19L\xD0\x9F\x7Ff\xE7]\xB9\x8Cy\x00\x00J\x00\xFF\xC0$\xC0#\xC0" 400 181 "-" "-"

which to me looks like the proxy has to be the SSL endpoint to get to see the Upgrade: header?

 

[Jens Alfke]

Well, at this point we get into the details of how HTTP proxying works, which his frankly not something I know about… :/

—Jens

[Marcus Roberts]

For the moment I took the easy way out.

I installed stud (instructions here: http://alexnj.com/blog/installing-and-configuring-stud-to-serve-ssl-requests.html) to terminate all my SSL connections and pass them through to my nginx server (with its SSL bindings removed).    Everything is working fine again (albeit with one extra level of complexity).

I'll try and dig deeper into nginx's handling of https web sockets in the future but for now my app is working fine over SSL and web sockets, so that's a win for me.

Marcus

[Marcus Roberts]

stud doesn't work - I'd accidentally left web sockets disabled - as soon as I enabled them I saw the exact same behaviour from stud as from nginx.

I'm setting up HAProxy at the moment - I have that passing https non-websocket traffic to the sync gateway.  Just tweaking the config to switch to the web socket protocol, and I will report back how that goes.

  

[Scott Ahten]

We have an instance of SyncGateway running behind nginx. One additional issue we ran into is a 1mb incoming limit set by default on nginx. This caused our app to get updates from other mobile clients and our web app, but prevented pushing updates of larger docs to the gateway. 

[Marcus Roberts]

Scott,

Do you have it running over https or http?

I can get http to work fine, but not https.  If you have a working https config you could share that would be really useful.

[Marcus Roberts]

On Thursday, 30 October 2014 07:47:18 UTC, Marcus Roberts wrote:

Scott,

Do you have it running over https or http?

I can get http to work fine, but not https.  If you have a working https config you could share that would be really useful.

Also, do you have web sockets enabled?  With web sockets disabled this all works perfectly over SSL through any of the proxies too.  I'm just doing some tcpdumps to see what's happening when the upgrade to web sockets happens.

 

[Marcus Roberts]

I think I've found the cause of the problem.

Watching the packets, everything was coming in over https except the web socket data which came in on port 80.

In WebSocketClient, in _connect the code says 

                             onPort:  (UInt16)(url.port ?url.port.intValue : 80)

which I think is defaulting it to port 80 if a port isn't specified.

Secondly, the URL is tested to determine whether to use TLS or not

    if (_tlsSettings || [url.scheme caseInsensitiveCompare: @"https"] == 0) {

So my URL of https://my.server.com/my-database

is going to port 80, but with TLS enabled.      I need it to go to port 443 with TLS disabled.

I hardcoded 443 and TLS disabled and it now works perfectly with websockets  through HAProxy.  I swapped back to nginx and that worked fine too.

I've updated the ticket with these findings.  Let me know what you think.

Marcus

[Scott Ahten]

I'm primarily focused on the mobile side of things but, IIRC, we ran into a similar problem on my current project, which was resolved via making a change to the config of nginx. I'll see what I can track down.  

[Jens Alfke]

On Oct 30, 2014, at 1:54 AM, Marcus Roberts <marcus....@gmail.com> wrote:

In WebSocketClient, in _connect the code says 

                             onPort:  (UInt16)(url.port ?url.port.intValue : 80)

which I think is defaulting it to port 80 if a port isn't specified.

That was definitely the problem! I've fixed it in CBL and pushed a new revision to master, 1ccd928.

—Jens

[Marcus Roberts]

On Thursday, 30 October 2014 18:25:41 UTC, Jens Alfke wrote:

That was definitely the problem! I've fixed it in CBL and pushed a new revision to master, 1ccd928.

Working just fine in my app now, thanks for the quick fix Jens!

Marcus