[ANN] lua-resty-http v0.11 released

[James Hurst]

Hi all,

Just a heads up for those using lua-resty-http, that the new version regrettably breaks backwards compatibility with regards to underscores in header field names.

Please carefully test your upgrades, and if you're concerned please read on.

# History

Originally we normalised headers to be case insensitive, but also normalised hyphens to underscores (considering underscore to the be the lower case hyphen), since this allowed syntax such as `h.content_type` to be equivalent to `h["Content-Type"].

This was for parity with how header normalisation was handled in `ngx.header`, and mostly worked because underscores in header names are often unsupported in the real world.

# Reason for change

However, underscores are indeed valid characters in header field names (despite being rare in the wild), and this behaviour breaks certain APIs. Having looked at the options, it was clear that a HTTP client which got in the way of setting valid header field names was just broken, and so retaining default behaviour with some kind of "raw" option didn't sit right.

In addition, since the underscore normalisation wasn't specifically documented, I feel reasonably confident that few people will have taken explicit advantage of how it worked, and it's actually quite possible that the old version could present unexpected surprises in existing deployments.

# New behaviour

We no longer interfere with the header field in any semantic sense. Alpha characters remain case insensitive.

These are all equivalent:

h["content-type"]

h["CONTENT-TYPE"]

h["cOnTENt-tyPe"]

but h["content_type"] is now an entirely different header field.

# Impact

If you use lua-resty-http, and you specify or read header names using underscores as shorthand for hyphens: (e.g. h.content_type when you mean h["Content-Type"]), your code will break, potentially in difficult to track ways.

# Advice

Test your systems against v0.11 properly before upgrading, and grep for hints of underscored usage.

# Feedback

If this affects you in horrible ways and you have a reasonable suggestion for supporting the old behaviour, please bring this up over at: https://github.com/pintsized/lua-resty-http/issues

Regards,

James

[Daurnimator]

On 25 July 2017 at 00:20, James Hurst <ja...@pintsized.co.uk> wrote:
> # New behaviour
>
> We no longer interfere with the header field in any semantic sense. Alpha
> characters remain case insensitive.
>
> These are all equivalent:
>
> h["content-type"]
> h["CONTENT-TYPE"]
> h["cOnTENt-tyPe"]

I've found many servers are case sensitive for header names (even
though they shouldn't be): wouldn't it make more sense to not change
case at all?

[James Hurst]

​I think not. The spec is the spec and that's what we must adhere to IMO.

By case sensitive for header names, do you mean explicitly only checking for one case, or reading different meaning for the same header in different cases?

We preserve the case given by the user - so if a server expects exactly `X-FOO-BAR`, then set `X-FOO-BAR` and all is well. The only way you can get into trouble is setting both `X-FOO-BAR` and 'X-Foo-Bar' for example, where the value of the second will be set to both. Which is of course, what you'd expect.

Also this is as much about reading response headers as it is setting request headers. I think it's important that the behaviours are consistent, and also that the user need not be aware of the case that a server responds with. In other words, the server *can* send `CONTENT-TYPE`, but we must be able to read it with `h["Content-Type"]`.

Funnily enough, Mark Nottingham (HTTP working group chair) tweeted this a few hours ago: https://twitter.com/mnot/status/892983532766625792

James