lua-resty-http

[James Hurst]

Hi all,

I've been working on a lua-resty-http implementation recently, and since this seems to be coming up on the list more and more often, I thought it sensible to share.

https://github.com/pintsized/lua-resty-http

I started where Brian left off with lua-resty-http-simple, extending it with a more complete and generic "cosocket" interface, and adding support for streaming the response body with an iterator. I kept a "simple" URI based interface similar to Brian's, which just wraps the generic one.

It's passing tests, so for most cases I would imagine you can just drop this in and start using it. I'm not running it in production just yet - though it'll be tested amongst our other apps soon to see if this exposes any issues. First I thought it would be good to get feedback on the design / interface before setting it in stone.

The main thing I wanted to achieve was streaming response bodies into Lua, regardless of whether they were encoded as chunked or not, for obvious memory management reasons. More generally it's been said before that we need a "blessed" resty-http module, so if anyone has any criticisms / suggestions / patches I'm all ears!

Cheers,

James.

[Brian Akins]

Interesting. I like the option to stream the response. Wondering if it'd be useful to stream request bodies as well. thinking of large POST/PUTs.

[James Hurst]

On 22 October 2013 13:13, Brian Akins <br...@akins.org> wrote:

Interesting. I like the option to stream the response. Wondering if it'd be useful to stream request bodies as well. thinking of large POST/PUTs.

Yeah, I think it would. Chunked transfer encoding for request bodies would be good, as well as simply being able to write data to the socket as it arrives (from wherever) rather than buffering it all to Lua first. I was also thinking of a helper called proxy_pass or something, which feeds the current request into the connection for you, since that's probably a common case.

On a different topic, what are your thoughts on normalising the request headers? I noticed you had this in your version, but I'm still not sure how I feel about it - whether it should be something the user deals with (i.e. don't sent bad requests), or whether it's just not cool for a http client to accidentally treat headers case sensitively? I'm leaning to the latter, but the problem seems like a corner case so it feels a little wasteful to be always normalising, just in-case the user's input is wrong?

[Yichun Zhang (agentzh)]

Hello!

On Tue, Oct 22, 2013 at 6:13 AM, James Hurst wrote:
> as well as simply being able to write data to the socket as it
> arrives (from wherever) rather than buffering it all to Lua first.

Yes, such features would require aviramc's patches below to enhance
the existing ngx_lua cosocket API:

https://github.com/chaoslawful/lua-nginx-module/pull/290

They actually built a real forward proxy in pure Lua :)

BTW, I'm really glad to see all the collaborations involved in the
community implementations of lua-resty-http :)

Thanks for your work!

Best regards,
-agentzh

[Brian Akins]

I normalized the headers because in my use case I knew I was getting "un-normalized" headers.

My idea concerning "streaming" request bodies is based somewhat on how libcurl does it - you register a callback and it calls it when it needs to read from that side, then curl writes that data to the socket. You can do this for non-chunked requests as well.  You can also do the "here's a full request body" option as well.   I have a few use cases with large request bodies that I don't necessarily want to buffer into Lua. 

Of course, one could do something similar to how nginx core does it - buffer in memory for up to X size and use a tempfile for larger bodies. The http client could just take a filehandle or something that behaved like one. Of course, if it just needed to "behave" like a filehandle, you could do all of what I just described easily.

Sorry, was a stream of thought email ;)