Why aren't http headers just a simple array?

1,274 views
Skip to first unread message

Joe Bowman

unread,
Apr 14, 2010, 11:38:47 AM4/14/10
to nodejs
Hi,

I'm curious why the approach for headers was to use a key/value pair
for headers, instead of doing an array of header strings? The key/
value pair approach has caused problems for people with setting
cookies, and there's been several hacks/patches put out there to
convert them to arrays. In my case, I just moved to using an array for
all headers.

I can see why changing this would break existing applications, but
would it be worth it to just move to using an array of properly
formatted header strings for the http module? You'd also save a little
bit of processing power by not having to do any string concatenation.

Nikhil Marathe

unread,
Apr 14, 2010, 12:17:52 PM4/14/10
to nod...@googlegroups.com

I don't think so. Headers are not meant to iterate over. Each header
has a particular job, and you usually want to query if a particular
header exists and what its value is, even for custom headers. A object
literal is ideal for the job. With an array you would just have to do
a linear time operation of indexOf. And then how would you get the
header value?

headers[headers.indexOf('user-agent')][1]
or headers[headers.indexOf('user-agent')].value

is very inconvenient. So I think object literals are the way to go.
What is your use case for headers?

To set a cookie, why not just put in the appropriate header when doing
writeHead() ?

Nikhil

Mikeal Rogers

unread,
Apr 14, 2010, 12:48:51 PM4/14/10
to nod...@googlegroups.com
This brings me back to something I've wanted for a while.

When you're writing a proxy you would ideally like to send and return the headers exactly as they were sent from the client and server. node changes all the headers to lowercase and sticks them in an object which cannot have duplicates so you can't do this.

Rather that try to change the default API, which is quite nice for everything except this particular use case, could we get a method or attribute in the ServerRequest and ClientResponse that returns the headers exactly as they can from the server, ideally just in a string. Also, we would need the HTTPClient.request to take, as it's header argument, that string headers to send untouched and we would need ServerResponse.writeHead to also take a string for it's headers argument.

This would allow people, when they wanted to, to just work around the niceties of the regular API to handle these specific use cases without changing the convenience of the default API.

-Mikeal


--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.


Isaac Schlueter

unread,
Apr 14, 2010, 1:03:08 PM4/14/10
to nod...@googlegroups.com
+1 for response.headers = {key:[value, value, value]}
+1 for response.rawHeaders === "key: value\nkey:value\nkey : value\n...."

--i

Dean Landolt

unread,
Apr 14, 2010, 1:05:15 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 12:48 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:
This brings me back to something I've wanted for a while.

When you're writing a proxy you would ideally like to send and return the headers exactly as they were sent from the client and server. node changes all the headers to lowercase and sticks them in an object which cannot have duplicates so you can't do this.

JSGI handles duplicate headers by allowing an array of strings for a particular header. The server can test for this and output multiple headers -- this would be a nice feature of node. It has the side benefit of being forgiving -- arrays toString with a comma join -- which is technically the Right Thing in HTTP so in most cases if you screw up and += your header value that's already an array, you'll be fine. I realize this blows up with cookies -- so, umm, don't do that -- but it's a nice little feature.

The other thing that headers-as-lists offers that headers-as-objects cannot is control over order but in the case of a proxy this shouldn't matter -- you'll build the header object from the source headers, most likely in order, so you should be good. Does node respect the order of header keys?
 

Rather that try to change the default API, which is quite nice for everything except this particular use case, could we get a method or attribute in the ServerRequest and ClientResponse that returns the headers exactly as they can from the server, ideally just in a string. Also, we would need the HTTPClient.request to take, as it's header argument, that string headers to send untouched and we would need ServerResponse.writeHead to also take a string for it's headers argument.

This would allow people, when they wanted to, to just work around the niceties of the regular API to handle these specific use cases without changing the convenience of the default API.

Alternatively perhaps node could take a case map somehow (as config or at runtime) that allows you to choose your header case. This wouldn't give you completely control but people really seem to care about header case.
 

Dean Landolt

unread,
Apr 14, 2010, 1:06:11 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 1:03 PM, Isaac Schlueter <i...@izs.me> wrote:
+1 for response.headers = {key:[value, value, value]}
+1 for response.rawHeaders === "key: value\nkey:value\nkey : value\n...."

Nice. Would setting rawHeaders supercede anything set on headers?

Isaac Schlueter

unread,
Apr 14, 2010, 1:19:22 PM4/14/10
to nod...@googlegroups.com

I would suggest that:

.rawHeaders on incoming message is read-only. (Not in an
Object.defineProperty sense, perhaps, but in a "changes don't actually
matter" sense.)
.rawHeaders on outgoing message is a getter/setter, such that setting
it loses any changes on .headers
var rawHeaderString = null;
Object.defineProperty(this, "rawHeaders",
{ set : function (h) { this.headers = parse(h); rawHeaderString = h }
, get : function () { return rawHeaderString }
});

Then, when the message is actually sent, if message.rawHeaders ===
null, then build the header string from the .headers object. So, once
you set message.rawHeaders, you lose message.headers. (Maybe
message.headers should be a getter/setter that throws if
rawHeaderString !== null?)

Just kinda sketching ideas here. Honestly, I'd be perfectly happy if
it just had one override the other, and the docs said "don't use
both".

--i

Dean Landolt

unread,
Apr 14, 2010, 1:28:16 PM4/14/10
to nod...@googlegroups.com

That makes good sense to me -- though if you're going the getter/setter route I'd say go a step further and have the rawHeaders setter set a private _rawHeaders and the rawHeaders getter return _rawHeaders || toRawHeaders(headers)...that way you can use the headers obj up until you need to dip your toe into the raw pool. And yeah, throwing in the headers setter makes sense if _rawHeaders has been set.

Joe Bowman

unread,
Apr 14, 2010, 1:50:13 PM4/14/10
to nodejs
Well the main problem is the set-cookie header, when acting as a
server. I can see where it would be preferable to access headers when
you want to read them, using the existing method. When doing set-
cookie, my experience is works best across different browsers if there
is a separate set-cookie header for each cookie.

I believe the +1 for set-cookie as an array, and/or raw headers goes
for me too.

On Apr 14, 12:17 pm, Nikhil Marathe <nsm.nik...@gmail.com> wrote:

Dean Landolt

unread,
Apr 14, 2010, 1:54:44 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 1:50 PM, Joe Bowman <bowman...@gmail.com> wrote:
Well the main problem is the set-cookie header, when acting as a
server. I can see where it would be preferable to access headers when
you want to read them, using the existing method. When doing set-
cookie, my experience is works best across different browsers if there
is a separate set-cookie header for each cookie.

I believe the +1 for set-cookie as an array, and/or raw headers goes
for me too.

Absolutely -- what we're proposing is to allow any header to be an array and have node interpret that as a distinct header line for each array value. The raw headers thing would just be a bonus.

Aaron Heckmann

unread,
Apr 14, 2010, 2:00:29 PM4/14/10
to nod...@googlegroups.com
Yep. A separate set-cookie header works best.

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.




--
Aaron

Mikeal Rogers

unread,
Apr 14, 2010, 2:02:40 PM4/14/10
to nod...@googlegroups.com
The problem with the "array for duplicate" headers is that it means you have to do a type check every time you read or write to the headers object.

The current API is simple and works great for the vast majority of use cases. I don't think we should do anything that complicates it, it's great the way it is.

In the case that you want to get the *exact* headers string from a request/response or you want to send headers with duplicates and/or capitalization we should just expose a raw API like the one isaacs suggests.

-Mikeal

Isaac Schlueter

unread,
Apr 14, 2010, 2:22:10 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 11:02, Mikeal Rogers <mikeal...@gmail.com> wrote:
> The problem with the "array for duplicate" headers is that it means you have
> to do a type check every time you read or write to the headers object.

That's why I suggest that the headers object always be a {key:[value]}
style, even when it's not a duplicate.

--i

Dean Landolt

unread,
Apr 14, 2010, 2:23:06 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 2:02 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:
The problem with the "array for duplicate" headers is that it means you have to do a type check every time you read or write to the headers object.

No you don't. Again, array coerces to a comma-join...it's a neat side effect. Plus, if you have no need for duplicate headers, just don't use them. It's that easy...

Joe Bowman

unread,
Apr 14, 2010, 2:33:11 PM4/14/10
to nodejs
Gotcha, and yea, I think that's the best approach. The reason I was
suggesting that any all headers be strings in array is because I
wasn't sure if set-cookie was the only header that would create this
issue. The approach you guys are proposing I think gives the best of
both worlds, as it still allows individual header access via a key as
well. Awesome engineering job there.

On Apr 14, 1:54 pm, Dean Landolt <d...@deanlandolt.com> wrote:


> On Wed, Apr 14, 2010 at 1:50 PM, Joe Bowman <bowman.jos...@gmail.com> wrote:
> > Well the main problem is the set-cookie header, when acting as a
> > server. I can see where it would be preferable to access headers when
> > you want to read them, using the existing method. When doing set-
> > cookie, my experience is works best across different browsers if there
> > is a separate set-cookie header for each cookie.
>
> > I believe the +1 for set-cookie as an array, and/or raw headers goes
> > for me too.
>

> Absolutely -- what we're proposing is to allow *any* header to be an array

Mikeal Rogers

unread,
Apr 14, 2010, 3:26:43 PM4/14/10
to nod...@googlegroups.com
If they are *always* {key:[value]} that would work but if they are *sometimes* an array and sometimes a string you couldn't do things like

headers['set-cookie'] = headers['set-cookie'] + ',asdf'

You would have to check if it's an array first.

Personally, I think making the current headers object us arrays for values is annoying to work with, I *hate* doing:

headers.name[0]

instead of

headers.name

all the time, not to mention this means I have to check for undefined first

if (headers.name) {headers.name[0]}

I just don't like it. The existing API is clean and simple, leave it be. Give me and the other 10 people that need raw header access the lowest level API possible which seams like the raw string API and let everyone else keep using the easy, simple, clean thing.

-Mikeal

Isaac Schlueter

unread,
Apr 14, 2010, 3:39:01 PM4/14/10
to nod...@googlegroups.com
I don't like testing for undefined either.

The problem is that it's hard to send multiple values of the same
header key, which is a fairly common use case. The existing API makes
it unnecessarily tricky to set more than one cookie, for instance.

I'm kinda hoping one of the geniuses lurking on this list will come up
with an elegant solution if we just keep bitching about it long
enough. ;)

--i

Mikeal Rogers

unread,
Apr 14, 2010, 4:09:56 PM4/14/10
to nod...@googlegroups.com
That's just it, I don't think it's that common of a use case.

I certainly need to do it, but that's because I'm building a proxy. If I was building a web application, or a file server, or something like that, I could write things sanely and not use multiple headers of the same name.

Even for cookie setting, I've never needed to do multiple set-cookie headers on the server, I've only had to do multiple set-cookie headers when dealing with someone else's crazy infrastructure like Facebook.

If for some silly reason I did want my web application to return two headers of same key I wouldn't expect this behavior to be encouraged by the API and I'd be happy with the raw string API that's been proposed. For the proxy use case, the raw string API is perfect because you want to send the headers block identically to what was sent to the proxy.

-Mikeal

Dean Landolt

unread,
Apr 14, 2010, 4:22:52 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 3:26 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:
If they are *always* {key:[value]} that would work but if they are *sometimes* an array and sometimes a string you couldn't do things like

headers['set-cookie'] = headers['set-cookie'] + ',asdf'

Dude -- seriously -- try it. For all the WTFs in javascript, this is like the opposite...
 

You would have to check if it's an array first.

Nope -- it'll coerce, and the coercion behavior is exactly what we want.
 

Personally, I think making the current headers object us arrays for values is annoying to work with, I *hate* doing:

headers.name[0]

instead of

headers.name

all the time, not to mention this means I have to check for undefined first

if (headers.name) {headers.name[0]}

I just don't like it. The existing API is clean and simple, leave it be. Give me and the other 10 people that need raw header access the lowest level API possible which seams like the raw string API and let everyone else keep using the easy, simple, clean thing.

I agree -- the existing api is very clean and this is an edge case. But if the edge case also happens to work perfectly, w/o anyone changing their code even, why not support it?

Dean Landolt

unread,
Apr 14, 2010, 4:26:33 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 4:22 PM, Dean Landolt <de...@deanlandolt.com> wrote:


On Wed, Apr 14, 2010 at 3:26 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:
If they are *always* {key:[value]} that would work but if they are *sometimes* an array and sometimes a string you couldn't do things like

headers['set-cookie'] = headers['set-cookie'] + ',asdf' 

Dude -- seriously -- try it. For all the WTFs in javascript, this is like the opposite...

One caveat to this (that would apply whether the source is a string or an array), you'd probably want to test for existence before prepending that comma. Of course, the array case isn't burdensome here either -- something like:

headers['set-cookie'] = headers['set-cookie'] && "," || "" && "asdf";

Mikeal Rogers

unread,
Apr 14, 2010, 4:27:26 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 1:22 PM, Dean Landolt <de...@deanlandolt.com> wrote:


On Wed, Apr 14, 2010 at 3:26 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:
If they are *always* {key:[value]} that would work but if they are *sometimes* an array and sometimes a string you couldn't do things like

headers['set-cookie'] = headers['set-cookie'] + ',asdf'

Dude -- seriously -- try it. For all the WTFs in javascript, this is like the opposite...
 

You would have to check if it's an array first.

Nope -- it'll coerce, and the coercion behavior is exactly what we want.

It'll coerce to a string, which means that it *was* and array, *now* it's a string, if another user of the API somewhere else, maybe on a listener after mine, tries to use it as an array now it will break, or worse, they'll try to use headers.name[0] and get the first character of the new string.
 
 

Personally, I think making the current headers object us arrays for values is annoying to work with, I *hate* doing:

headers.name[0]

instead of

headers.name

all the time, not to mention this means I have to check for undefined first

if (headers.name) {headers.name[0]}

I just don't like it. The existing API is clean and simple, leave it be. Give me and the other 10 people that need raw header access the lowest level API possible which seams like the raw string API and let everyone else keep using the easy, simple, clean thing.

I agree -- the existing api is very clean and this is an edge case. But if the edge case also happens to work perfectly, w/o anyone changing their code even, why not support it?
 

--

Dean Landolt

unread,
Apr 14, 2010, 4:27:41 PM4/14/10
to nod...@googlegroups.com
ugh...FAIL...perhaps i shouldn't get cute w/ short circuits...but you get the idea

Dean Landolt

unread,
Apr 14, 2010, 4:34:51 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 4:27 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:


On Wed, Apr 14, 2010 at 1:22 PM, Dean Landolt <de...@deanlandolt.com> wrote:


On Wed, Apr 14, 2010 at 3:26 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:
If they are *always* {key:[value]} that would work but if they are *sometimes* an array and sometimes a string you couldn't do things like

headers['set-cookie'] = headers['set-cookie'] + ',asdf'

Dude -- seriously -- try it. For all the WTFs in javascript, this is like the opposite...
 

You would have to check if it's an array first.

Nope -- it'll coerce, and the coercion behavior is exactly what we want.

It'll coerce to a string, which means that it *was* and array, *now* it's a string, if another user of the API somewhere else, maybe on a listener after mine, tries to use it as an array now it will break, or worse, they'll try to use headers.name[0] and get the first character of the new string.

Doctor, it hurts when I don't ducktest...

But no worries -- it'll work in jsgi-land for me and I think it's a nice feature, but like you said, the current API + raw headers would give everyone what they need.

Matthew Ranney

unread,
Apr 14, 2010, 5:22:10 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 1:09 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:
I certainly need to do it, but that's because I'm building a proxy. If I was building a web application, or a file server, or something like that, I could write things sanely and not use multiple headers of the same name.

Even if you are building a web application, you still end up needing to deal with multiple values per header from those __utm 3rd party cookies that many people send.

I like that headers are broken out into properties, but I like one property per header.  Parsing the cookie header is a special case, not just because of the multiple cookies, but because each cookie value itself needs to be parsed.  Let's just get some sort of cookie header parsing library to decode the whole thing.

billywhizz

unread,
Apr 14, 2010, 6:07:00 PM4/14/10
to nodejs
there's a lot of people here who seem to be saying "it works okay in
most cases". as far as i undestand it, node.js is supposed to be a low
level platform that leaves as much as possible open to build on top
of. so, we should be looking for a solution here that works in all
possible scenarios, and the current one doesn't. it's broken for the
case where you have multiple lines of the same header key,
particularly in the case of Set-Cookie, as the values for this can
contain commas, which means i have to do expensive and specific
parsing on the headers.

I have brought this issue up multiple times on here, but have never
got much feedback. Here is how i fix it in my node-httpclient project:

http.IncomingMessage.prototype._addHeaderLine = function (field,
value) {
if (field in this.headers) {
this.headers[field].push(value);
} else {
this.headers[field] = [value];
}
};

This is a hashtable with the header as the key. Each value stored in
the hashtable is an array. For headers that can only be specified
once, there will only be one element in the array. for headers that
have multiple values, you can iterate over the array using forEach...

you can see this in action here:

http://github.com/billywhizz/node-httpclient/blob/master/lib/httpclient.js#L149

more discussion on this issue here:
http://groups.google.com/group/nodejs/browse_thread/thread/bd772aa35fc69612

if you combined this solution with a simple way to get at the raw
headers, you should be able to handle any scenario...

On Apr 14, 10:22 pm, Matthew Ranney <m...@ranney.com> wrote:

r...@tinyclouds.org

unread,
Apr 14, 2010, 6:18:45 PM4/14/10
to nod...@googlegroups.com
There just isn't a good solution. I'll note that for several months
node used only arrays of tuples to represent headers - and that this
is still a format to write headers. I moved away from this for the
incoming headers because 1) it confused everyone and 2) it was slow -
lots of extra little arrays had to be created.

I guess we can do { 'Set-Cookie': ['A', 'B'], 'Content-Length': '123'
}. And everyone will just have to check the type of every header
value, always. It's just going to be ugly. We can do { 'Set-Cookie':
['A', 'B'], 'Content-Length': ['123'] }, but I assure you it will
confuse 90% of the newbies - plus it creates a usually unnecessary
object for every header-line. A hybrid solution might be that we
guarantee that certain headers will be arrays and certain ones will be
single values. For example, it makes no sense to have two
'Content-Length' header lines - HTTP specifies that you should over
write the previous 'Content-Length'. 'Set-Cookie', really the only
problem, could always be an array value - even when only one
'Set-Cookie' line is present.

billywhizz

unread,
Apr 14, 2010, 6:51:52 PM4/14/10
to nodejs
thanks. i understand what you are saying about extra allocations. do
you know how much overhead an array would be compared to a
concatenated string?

from the options above, i would say a +1 for making certain headers
arrays and leaving the rest as strings. Set-Cookie seems to be causing
the most pain at the moment and would be a prime candidate. a lot of
applications won't be using this header so it shouldn't affect
performance in this case.

on another, not unrelated, point - am wondering would it be a good
idea to move the whole of http out of node.js core and make a separate
addon for http client and http server? it just feels to me like http
is too high level and complex a protocol for node core and this is why
we are seeing these issues.

btw - let me know if you need to allocate any work on the core stuff.
i am getting up to speed on how v8 works and i am in heavy development
over next few weeks on a system based on top of node.js so will be in
a position to help out if you need it...

billywhizz

unread,
Apr 14, 2010, 6:54:02 PM4/14/10
to nodejs
also, there is nothing really stopping anyone overriding the default
node.js behaviour by changing the prototype after loading http.js:

http.IncomingMessage.prototype._addHeaderLine = function (field,
value) {
if (field in this.headers) {
this.headers[field].push(value);
} else {
this.headers[field] = [value];
}
};

i have only tried this on http client so am not sure if it will break
http server...

r...@tinyclouds.org

unread,
Apr 14, 2010, 6:59:13 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 3:51 PM, billywhizz <apjo...@gmail.com> wrote:
> thanks. i understand what you are saying about extra allocations. do
> you know how much overhead an array would be compared to a
> concatenated string?
>
> from the options above, i would say a +1 for making certain headers
> arrays and leaving the rest as strings. Set-Cookie seems to be causing
> the most pain at the moment and would be a prime candidate. a lot of
> applications won't be using this header so it shouldn't affect
> performance in this case.
>
> on another, not unrelated, point - am wondering would it be a good
> idea to move the whole of http out of node.js core and make a separate
> addon for http client and http server? it just feels to me like http
> is too high level and complex a protocol for node core and this is why
> we are seeing these issues.

No. HTTP is fundamental to the internet and I want Node to support it very well.

> btw - let me know if you need to allocate any work on the core stuff.
> i am getting up to speed on how v8 works and i am in heavy development
> over next few weeks on a system based on top of node.js so will be in
> a position to help out if you need it...

Cool. I'll post some tasks soon that need doing.

billywhizz

unread,
Apr 14, 2010, 7:02:48 PM4/14/10
to nodejs
looks like this issue exists in c# land also:

http://www.velocityreviews.com/forums/t77773-set-cookie-header-broken-in-webheadercollection.html

the recommended practice for this in .Net is to use a CookieContainer
(which must have access to the raw headers behind the scenes). This
could be a valid solution for node also. any thoughts? i guess it
could be an option on the http Client and behind the scenes node.js
could populate it if it exists from the raw http headers...

Mikeal Rogers

unread,
Apr 14, 2010, 7:03:44 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 3:18 PM, <r...@tinyclouds.org> wrote:
There just isn't a good solution. I'll note that for several months
node used only arrays of tuples to represent headers - and that this
is still a format to write headers. I moved away from this for the
incoming headers because 1) it confused everyone and 2) it was slow -
lots of extra little arrays had to be created.

I guess we can do { 'Set-Cookie': ['A', 'B'], 'Content-Length': '123'
}. And everyone will just have to check the type of every header
value, always. It's just going to be ugly. We can do  { 'Set-Cookie':
['A', 'B'], 'Content-Length': ['123'] }, but I assure you it will
confuse 90% of the newbies - plus it creates a usually unnecessary
object for every header-line. A hybrid solution might be that we
guarantee that certain headers will be arrays and certain ones will be
single values. For example, it makes no sense to have two
'Content-Length' header lines - HTTP specifies that you should over
write the previous 'Content-Length'.

I've actually run in to this with some headers that aren't allowed to have duplicates and it turns out that different browsers enforce a different header (either first or last) and some really terrible server applications rely on this behavior :(

In the case you're writing a proxy you really want to just pass the headers *exactly* or with only specific changes like adding proxy headers. This is why I would advocate the raw string API because it can handle any use case anyone would have without making the current API more complex.
 
'Set-Cookie', really the only
problem, could always be an array value - even when only one
'Set-Cookie' line is present.

Mikeal Rogers

unread,
Apr 14, 2010, 7:05:20 PM4/14/10
to nod...@googlegroups.com
I've been spending a lot of time with the current headers API and I love it 99% of the time and the worst thing we could do would be to complicate that and make me love it less 99% of the time in order to cover that 1%.

Dean Landolt

unread,
Apr 14, 2010, 7:39:06 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 6:18 PM, <r...@tinyclouds.org> wrote:
There just isn't a good solution. I'll note that for several months
node used only arrays of tuples to represent headers - and that this
is still a format to write headers. I moved away from this for the
incoming headers because 1) it confused everyone and 2) it was slow -
lots of extra little arrays had to be created.

I guess we can do { 'Set-Cookie': ['A', 'B'], 'Content-Length': '123'
}. And everyone will just have to check the type of every header
value, always.

But, but, but...you don't have to check unless you really care (set-cookie) -- and at that point, it's not onerous, and you'll want to take extra care with that one anyway.
 
It's just going to be ugly. We can do  { 'Set-Cookie':
['A', 'B'], 'Content-Length': ['123'] }, but I assure you it will
confuse 90% of the newbies - plus it creates a usually unnecessary
object for every header-line. A hybrid solution might be that we
guarantee that certain headers will be arrays and certain ones will be
single values. For example, it makes no sense to have two
'Content-Length' header lines - HTTP specifies that you should over
write the previous 'Content-Length'. 'Set-Cookie', really the only
problem, could always be an array value - even when only one
'Set-Cookie' line is present.

It seems like it'd be a good convention for folks to always use an array with set-cookie, though it doesn't seem like it needs to be a required. Enforcing some arbitrary set of headers that must be arrays seems wrong to me -- and wholly unnecessary.

billywhizz

unread,
Apr 14, 2010, 7:45:49 PM4/14/10
to nodejs
Mikeal i think you are ignoring the http client side of things.
cookies are used in a lot of "web services" (especially for
authentication/session handling) and it's important to get them right.

have just had a thought - wouldn't the best solution be use \n instead
of a comma as the delimiter when their are multiple headers? you can
be guaranteed that an individual header value will never contain \n as
this is used as part of the \r\n pair to separate the headers in the
raw message.

At least then we can leave things pretty much as they are - nice and
simple - but it will also be easy to parse out multi-valued headers by
doing a simple string.split...

This would change the _addHeaderLine function to:

IncomingMessage.prototype._addHeaderLine = function (field, value) {
if (field in this.headers) {

this.headers[field] += "\n" + value;
} else {
this.headers[field] = value;
}
};


thoughts?

On Apr 15, 12:05 am, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> I've been spending a lot of time with the current headers API and I love it
> 99% of the time and the worst thing we could do would be to complicate that
> and make me love it less 99% of the time in order to cover that 1%.
>
>
>
> On Wed, Apr 14, 2010 at 3:59 PM, <r...@tinyclouds.org> wrote:

> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>

Dean Landolt

unread,
Apr 14, 2010, 7:54:02 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 7:45 PM, billywhizz <apjo...@gmail.com> wrote:
Mikeal i think you are ignoring the http client side of things.
cookies are used in a lot of "web services" (especially for
authentication/session handling) and it's important to get them right.

have just had a thought - wouldn't the best solution be use \n instead
of a comma as the delimiter when their are multiple headers? you can
be guaranteed that an individual header value will never contain \n as
this is used as part of the \r\n pair to separate the headers in the
raw message.

At least then we can leave things pretty much as they are - nice and
simple - but it will also be easy to parse out multi-valued headers by
doing a simple string.split...

This would change the _addHeaderLine function to:

IncomingMessage.prototype._addHeaderLine = function (field, value) {
 if (field in this.headers) {
   this.headers[field] += "\n" + value;
 } else {
   this.headers[field] = value;
 }
};


thoughts?


Heh. HTTP can be a strange beast. Headers are allowed to wrap multiple lines -- so if \r\n is followed by any number of spaces the header is considered multiline -- this would break that functionality.

billywhizz

unread,
Apr 14, 2010, 8:20:35 PM4/14/10
to nodejs
but won't those CR/LF pairs be stripped out by the time we get to
_addHeaderLine?

it looks like from the HTTP 1.1 spec that anything spanning multiple
lines can have whitespace stripped from it:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

After reading the spec, i can see that the way node.js currently works
is really correct. the spec clearly states this:

"Multiple message-header fields with the same field-name MAY be
present in a message if and only if the entire field-value for that
header field is defined as a comma-separated list [i.e., #(values)].
It MUST be possible to combine the multiple header fields into one
"field-name: field-value" pair, without changing the semantics of the
message, by appending each subsequent field-value to the first, each
separated by a comma."

In this case, it's really cookies that are broken (because they allow
commas within the value) not the way node.js handles headers.

I think the cleanest thing to do is leave the existing process intact
but add a cookiecontainer object to http client which can be used
optionally.

On Apr 15, 12:54 am, Dean Landolt <d...@deanlandolt.com> wrote:

Matthew Ranney

unread,
Apr 14, 2010, 8:25:15 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 5:20 PM, billywhizz <apjo...@gmail.com> wrote:
I think the cleanest thing to do is leave the existing process intact
but add a cookiecontainer object to http client which can be used
optionally.

We have a url library for parsing URL.  Let's just get a cookie library for parsing cookies.

That way if you don't care about cookies, you don't incur any extra parsing cost for them.

Elijah Insua

unread,
Apr 14, 2010, 8:34:41 PM4/14/10
to nod...@googlegroups.com
+1 for a userland library

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.

Dean Landolt

unread,
Apr 14, 2010, 8:37:36 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 8:34 PM, Elijah Insua <tmp...@gmail.com> wrote:
+1 for a userland library

 
This isn't a matter of userland libraries -- it cannot be solved in userland if node doesn't support it at all. This conversation is about figuring out how to get node to support it (preferably efficiently and unobtrusively) -- then libraries can be written to make it pretty.

Dean Landolt

unread,
Apr 14, 2010, 8:40:03 PM4/14/10
to nod...@googlegroups.com
On Wed, Apr 14, 2010 at 8:20 PM, billywhizz <apjo...@gmail.com> wrote:
but won't those CR/LF pairs be stripped out by the time we get to
_addHeaderLine?

it looks like from the HTTP 1.1 spec that anything spanning multiple
lines can have whitespace stripped from it:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

After reading the spec, i can see that the way node.js currently works
is really correct. the spec clearly states this:

"Multiple message-header fields with the same field-name MAY be
present in a message if and only if the entire field-value for that
header field is defined as a comma-separated list [i.e., #(values)].
It MUST be possible to combine the multiple header fields into one
"field-name: field-value" pair, without changing the semantics of the
message, by appending each subsequent field-value to the first, each
separated by a comma."

In this case, it's really cookies that are broken (because they allow
commas within the value) not the way node.js handles headers.

Indeed, but that's the way it is, and the way it will be for a long time. Duplicate headers, one way or another, is the only workable solution.
 

I think the cleanest thing to do is leave the existing process intact
but add a cookiecontainer object to http client which can be used
optionally.

That's one option. I wouldn't call that the cleanest though -- drawing from the .NET well probably won't ever yield the cleanest solution :)

billywhizz

unread,
Apr 14, 2010, 9:17:27 PM4/14/10
to nodejs

On Apr 15, 1:40 am, Dean Landolt <d...@deanlandolt.com> wrote:

> the .NET well probably won't *ever* yield the *cleanest* solution :)

at the end of the day cookies are broken in terms of the HTTP spec
because they don't assume that multiple headers can be put into a
comma delimited list, so i think we are going to have to come up with
a higher level/less clean solution to handle cookies.

For me, this solution works:

http.IncomingMessage.prototype._addHeaderLine = function (field,
value) {

if(field == "set-cookie") {
if (field in this.headers) {
this.headers[field].push(value);
} else {
this.headers[field] = [value];
}
}
else {
if (field in this.headers) {
this.headers[field] += ", " + value;


} else {
this.headers[field] = value;
}
}
};

and it doesn't break anything. anyone who needs easy cookie support
can just put this in their code after the require of http.js... ahh...
the wonders of dynamic languages... =)

PS - MS don't get everything wrong - the CLR/Net/Mono is a pretty nice
ecosystem for a wide range of problems...

r...@tinyclouds.org

unread,
Apr 14, 2010, 9:22:23 PM4/14/10
to nod...@googlegroups.com
Let's just have message.headers['set-cookie'] always an array.

billywhizz

unread,
Apr 14, 2010, 10:37:59 PM4/14/10
to nodejs
sounds good to me...

neocoder

unread,
Apr 15, 2010, 4:03:58 AM4/15/10
to nodejs
Haven't tested 0.1.90 yet, but if headers is an object now I totally
agree with Joe.
We need a way to set multiple 'Set-Cookie' headers which is not
possible with
headers object.


On Apr 14, 6:38 pm, Joe Bowman <bowman.jos...@gmail.com> wrote:
> Hi,
>
> I'm curious why the approach for headers was to use a key/value pair
> for headers, instead of doing an array of header strings? The key/
> value pair approach has caused problems for people with setting
> cookies, and there's been several hacks/patches put out there to
> convert them to arrays. In my case, I just moved to using an array for
> all headers.
>
> I can see why changing this would break existing applications, but
> would it be worth it to just move to using an array of properly
> formatted header strings for the http module? You'd also save a little
> bit of processing power by not having to do any string concatenation.

Jorge

unread,
Apr 15, 2010, 8:24:59 AM4/15/10
to nod...@googlegroups.com
On 15/04/2010, at 00:07, billywhizz wrote:
> (...)
> if (field in this.headers) {
> (...)

That should be:

if ( this.headers.hasOwnProperty(field) ) {

Because `propName in object` does return true for inherited propName properties, while `object.hasOwnProperty(propName)` doesn't.
--
Jorge.

Reply all
Reply to author
Forward
0 new messages