Nginx Considered Harmful

[Aaron Friel]

I am working on a project that currently uses Keter for deployment and a modified Yesod framework for subdomain routing and web proxying. I've successfully demonstrated the efficacy of this approach as can be observed here:

http://h.80.www.google.com.ergo.test.168.62.192.130.xip.io:3000/

That is the yesod devel instance of the proxy as currently deployed. The keter instance is available on port 80 (and hosted as a separate process run by Keter):

http://h.80.www.google.com.ergo.test.168.62.192.130.xip.io/

You will notice that one of these works really, really well. The other - not so much. The problem, traced with Fiddler on Windows and a great many searches today yielded that Nginx is not actually HTTP/1.1 compliant, has never been and may not be for A Long Time(TM). I'm not sure what happened here - they're a fantastic project and I've loved using Nginx in the past as a proxy, but I have to question the direction of the project when draft SPDY support is higher on the todo list than HTTP/1.1 support.

Given that Yesod/Wai support HTTP/1.1 and chunked transfer encoding, this can lead to problems that only show up with Keter. That's really unfortunate, and could lead to bugs appearing late in development and preventing certain features. Chunked transfer encoding is also necessary for some devices, as allegedly mobile OSes tend to use chunked POSTs.

I as well did some preliminary searching for possible solutions and proxies that might be suitable in the future.

One solution might be to have the proxy be a Yesod instance. I would be willing to contribute substantial parts of my project code for this to happen. Wai is HTTP/1.1 compliant and my web proxy is more than capable of sitting in front of Yesod, as demonstrated by this absurd example:

http://h.3000.h.80.www.google.com.ergo.test.168.62.192.130.xip.io.ergo.test.168.62.192.130.xip.io:3000/

(Queue Inception references.)

Another solution would be to take advantage of another project like Varnish and the additional features offered therein. Incorporating Varnish support in Keter would mean that responses would be cached "for free" and ensure better utilization of resources on servers hosting Yesod websites. I believe Varnish supports HTTP/1.1.

Among the ones I have ruled out are:

• Nginx - 1.3 support seems to be prioritizing SPDY over HTTP/1.1
• Pound
• Node.js

Services that may work, but I couldn't find a definitive answer on support for:

• Apache
• lighttpd

I'm inclined to believe the answer here is to stay home-grown as Varnish does not, and will not be supported on Windows in the near future. Apache and Lighttpd can be deployed on Windows. Ironically it looks like IIS may work as a reverse proxy on Windows - but of course it lacks portability.

[Greg Weber]

Nginx supports a lot of functionality, what are you trying to use it for?
why did your rule out Pound?

[Michael Snoyman]

Firstly: cool project. Are you using http-conduit for this? 

Which HTTP 1.1 feature is missing from nginx which is causing problems? I'm surprised to see this, since I've used nginx as a reverse proxy for almost all of my deployments, and haven't seen any issues yet.

I'm not opposed to pushing the proxy features directly into Keter (I think Greg has been wanting me to do that since the beginning). I've been hesitant to do so, since we'd be dropping a battle tested, highly efficient server (nginx) for something homegrown. But if nginx isn't up to the task, maybe it's time to make the move, at least as a separate branch.

I don't think we'd want to use a full-blown web server for this, probably some relatively simple code that looks like:

routeConnection :: Source IO ByteString -> Sink ByteString IO () -> Map Hostname Port -> IO ()

routeConnection src sink hostmap = do

    (vhost, rsrc) <- src $$+ getVhost

    case lookup vhost hostmap of

        Nothing -> response200 "Welcome to Keter. The virtual host you requested was not found."

        Just port -> do

            socket <- lift $ getSocket port

            tid1 <- forkIO $ rsrc $$+- sinkSocket socket

            tid2 <- forkIO $ sourceSocket socket $$ sink

            waitForThreads [tid1, tid2]

This way, we can avoid keter doing a lot of work, and hopefully get something working. I'm not sure how far your code is from a solution along these lines.

Michael

[Michael Snoyman]

Here's a more fleshed out reverse proxy: https://gist.github.com/3265522

Michael

[Aaron Friel]

From searching for terms like [Pound proxy transfer encoding] and [Pound "HTTP/1.1" chunked] it looks like others have had the same problem - HTTP/1.1 back ends cause problems.

[Aaron Friel]

Yes I am using http-conduit for speed and all the peripheral benefits.

Specifically the problem is that Nginx does not support sending incoming HTTP/1.1 requests using chunked transfer encoding (i.e.: a POST with a large body as commonly sent by mobile phones) or proxying a back-end response using chunked transfer encoding. Neither work.

A very simple proxy like the one you suggested would also work - but I suspect we want the proxy to be able to do things like set or filter headers and possibly route based on the Host header, yes? The actual "proxy" part of my code is impressively small. Most of the code is the subdomain routing, which may be useful for Keter, and the HTML parser that fixes URIs in pages.

[Tristan Seligmann]

On Sun, Aug 5, 2012 at 5:04 PM, Michael Snoyman <mic...@snoyman.com> wrote:
> Which HTTP 1.1 feature is missing from nginx which is causing problems? I'm
> surprised to see this, since I've used nginx as a reverse proxy for almost
> all of my deployments, and haven't seen any issues yet.

As far as I know, nginx has HTTP/1.1 support on the front end that
works just fine; that is, a client can talk HTTP/1.1 to an nginx
server without any issues. However, HTTP/1.1 is completely unsupported
by nginx when proxying to a backend server; thus no chunked encoding
support and so on when nginx is talking to a backend server.
--
mithrandi, i Ainil en-Balandor, a faer Ambar

[Aaron Friel]

Correct.

[Michael Snoyman]

On Sun, Aug 5, 2012 at 7:07 PM, Aaron Friel <aaron...@gmail.com> wrote:

Yes I am using http-conduit for speed and all the peripheral benefits.

Specifically the problem is that Nginx does not support sending incoming HTTP/1.1 requests using chunked transfer encoding (i.e.: a POST with a large body as commonly sent by mobile phones) or proxying a back-end response using chunked transfer encoding. Neither work.

I have no reason to doubt you about chunked request bodies, but I'm surprised to hear about the chunked response bodies. That is a serious problem.

 

A very simple proxy like the one you suggested would also work - but I suspect we want the proxy to be able to do things like set or filter headers and possibly route based on the Host header, yes? The actual "proxy" part of my code is impressively small. Most of the code is the subdomain routing, which may be useful for Keter, and the HTML parser that fixes URIs in pages.

For a more general purpose proxy, those things are necessary. For Keter's purposes, we only need the host header parsing (which my sample code handles). Remember that in the case of Keter, we're making a request to www.example.com, and the server thinks it's serving requests for www.example.com. Your proxy is doing something different: the requested host and the actual host are different.

Michael

[Aaron Friel]

Agreed - one caveat I can see with moving to a pure Yesod proxy is SSL support. That's breaking for anyone that wants to support secure log in without rolling their own SSL or SSL Middleware.

[Michael Snoyman]

On Mon, Aug 6, 2012 at 5:37 PM, Aaron Friel <aaron...@gmail.com> wrote:

Agreed - one caveat I can see with moving to a pure Yesod proxy is SSL support. That's breaking for anyone that wants to support secure log in without rolling their own SSL or SSL Middleware.

I've started a branch of Keter that includes the built-in reverse proxy:

https://github.com/snoyberg/keter/tree/builtin-proxy

Anyone who has the chance to, please check it out.

As for SSL support, I see two choices:

1. Continue as we have it right now: some kind of external server (Nginx or Pound) would reverse proxy to Keter, and Keter knows nothing about SSL.

2. Write a Keter module that adds SSL support. This would actually be very easy, due to Vincent's awesome tls package (just look at how small the warp-tls package is).

So I don't see this as a serious obstacle.

Michael

[Lyndon Maydwell]

+100 to SSL support without requiring a 3rd party server!

[Aaron Friel]

I'll work on a Wai Middleware for SSL then. Should be easy, right? :D

[Michael Snoyman]

Actually, a WAI middleware wouldn't help here. (Plus, a WAI middleware is impossible due to what WAI is. Consider, for example, what a WAI SSL middleware would mean for CGI.) What we need here is an alternate to network-conduit's runTCPServer that uses SSL. My basic approach for adding the support to keter was:

* Added support to the config file for specifying the SSL port to listen on and the SSL certificate.
* If the SSL information is provided, provide that information to runTCPSSLServer (or whatever it's called), and start listening in the background just like we do for non-SSL.

Michael

[Max Cantor]

I've been using Amazon ELBs as a reverse proxy/SSL endpoint for a long time with multiple projects.  I'm very happy with this solution. I  know it locks you into EC2 (although I believe Rackspace and others provide a similar tool), but its dead simple SSL support, as secure as anything else in the world, and I can get email or SMS notices if the number of dead instances behind the ELB exceeds a certain threshold. 

[Kazu Yamamoto]

Hello,

Just FYI.

mighttpd2 on Hackage has a feature of reverse proxy. Though I'm not sure
you like it, it uses http-conduit to implement reverse proxy. mighttpd2 runs
on Unix, not on Windows.

It does not support SSL/TLS yet but a friend of mine is interested in implementing them.

--Kazu

[Michael Snoyman]

As it stands now, Keter only runs on Linux (it uses hinotify). However, once Mark's file watching code is released, Keter will probably switch over to that, allowing it to run on any OS. I'm not sure if we really care about Windows support for Keter (any comments on that?), so I'm not sure if that's a blocker.

I guess my main reason to implement it in-house for Keter is that our reverse proxy needs are incredibly specific, and as such can be implemented in a very small bit of code. Are there cases that I haven't taken into account that you think mighttpd2 is providing for?

Also, for the SSL support: is warp-tls insufficient?

Michael

[Michael Snoyman]

OK, done :)

https://github.com/snoyberg/keter/commit/0cb1d5b2a0b125630da330b0d58e2503365fbc66

*Very* untested code, anyone who can, please give it a whirl. Note that you need to specify your SSL settings in the keter config file. Have a look at https://github.com/snoyberg/keter/blob/0cb1d5b2a0b125630da330b0d58e2503365fbc66/keter-config.yaml for more information.

Michael

[Blake Rain]

Just given the builtin-proxy branch of keter a whirl. I haven't been able to test any SSL yet, as I don't have any on the server. A few thoughts:

1. I will have to switch back to nginx (or something) as I have two sites (PHP and static HTML) that need to be served;

2. Keter's built-in proxy seems to give considerably faster response times than nginx. What have you done?

Lots of love,

- Blake.

[Felipe Almeida Lessa]

On Thu, Aug 9, 2012 at 12:50 PM, Blake Rain <blake...@gmail.com> wrote:
> 2. Keter's built-in proxy seems to give considerably faster response times
> than nginx. What have you done?

Michael's a type sorcerer!

--
Felipe.

[Michael Snoyman]

On Aug 9, 2012 6:50 PM, "Blake Rain" <blake...@gmail.com> wrote:
>
> Just given the builtin-proxy branch of keter a whirl. I haven't been able to test any SSL yet, as I don't have any on the server. A few thoughts:
>
> 1. I will have to switch back to nginx (or something) as I have two sites (PHP and static HTML) that need to be served;

Actually, for the static HTML, you could just create a warp-static Keter bundle and serve it behind Keter. For the PHP site, you could use mighttpd2 as a Keter bundle and set it up to reverse proxy to your nginx server. I'm certain there's a yo dawg in there somewhere, or perhaps P R O X C E P T I O N.

Not sure if that's an improvement or not.

> 2. Keter's built-in proxy seems to give considerably faster response times than nginx. What have you done?

Thanks for that feedback :)

[Kazu Yamamoto]

Hi,

Unfortunately, mighttpd2 cannot handle PHP since it does not support
FastCGI. A few years ago, I investigated how difficult to implement
FastCGI (server side, not client side). I appeared it took a long time
and I don't use PHP. So, I gave up supporting FastCGI.

--Kazu

[Blake Rain]

Well, the solution is obviously to re-write the PHP site using Yesod.