HOLY CRAP. nearly all nodejs http servers are vulnerable to DoS and apparently, the V8 guys seem to not care much

[Jann Horn]

http://www.youtube.com/watch?v=R2Cq3CLI6H8

Technical explaination 0m-19m or so, part about nodejs at 40m or so.

Basically, because v8 uses weak hashes for objects, you can fill up
one slot of the hashtable with many entries, e.g. using a POST
containing a querystring with many keys with the same hash. Operating
on those keys (inserting and reading) then becomes slow as hell which
allows you to bring a nodejs server to 100% CPU usage for a long time
(blocking the event loop completely) with one moderately large POST
request. This is bad.

Those guys say they told Google October 18th, they got through to the
v8 guys in November, and they said they don't care sooo much about DoS
attacks on v8 because they're mainly interested in browserside stuff.

This is bad for us.

[Mikeal Rogers]

The POST body isn't parsed until it's all in memory. Servers should set a max size for POST bodies to avoid this.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

[Jann Horn]

2011/12/29 Mikeal Rogers <mikeal...@gmail.com>:

> The POST body isn't parsed until it's all in memory. Servers should set a max size for POST bodies to avoid this.

Well... but what max size? With a dead-simple collision that produces
very long keys, I got 7-10s for creating the object. With keys of the
same length that don't collide, it only took less than 0.5s. Both with
10000 entries in the object. If we assume that an attacker can find
collisions with a length of 8 bytes, he'll need maybe 10 bytes per
entry in the querystring or so (yes, I know, I'm somewhat guessing
around now). So, a string of 100kB might already be sufficient to
block the server for ten seconds.

[jmar777]

Max sizes for POST/PUT/etc. aren't sufficient - the attack can be non-
negligible with only a few kilobytes. Ideally this would be addressed
via randomized hashing in V8, but the problem can at least be
mitigated by frameworks. E.g., express or whatever could provide a
method interface for querystring and body values that auto-prefixes
keys with a random value, busting pre-computed hash collisions... but
that's nasty. You could also use a custom request parser that has a
whitelist of keys for your app, but that's so stupid-annoying no one
would do it.

I don't know the proper channels to escalate visibility and priority
of this with the V8 guys, but this definitely warrants some noise.

[Phoscur]

This can be fixed by limiting the (recursive) amount of entries in
POST/etc. right?
I see no reason why this should be fixed by the V8 guys, this has to be
fixed in querystring.

What usecase needs more than a 100 POST arguments?

I see how this is a vulnerability produced by hashfunctions, but it is
not a problem which has to be fixed in the hashfunctions itself.

[Phoscur]

These guys exploit the worstcase runtime of a hashtable, is it possible
to avoid this case by checking for too many collisions on each insert?

[Mark Hahn]

>   is it possible to avoid this case by checking for too many collisions on each insert? 

That would be a change to v8.  There is a much better fix for v8 which is to randomize the hashes. 

[Andi]

This is the solution. I know that V8 has very fast hashes. And to
overwrite a large object's key does not mean that V8 iterates over a
long list. I think the whole story is overrated. Just give your post
body size a limit that makes sense (can be very small depending on
what you expect), then try to parse the string. I think it's no
problem to deal with this issue.
But it's important to set the limit!

[Dean Landolt]

On Thu, Dec 29, 2011 at 8:35 AM, Andi <test...@gmx.de> wrote:

This is the solution. I know that V8 has very fast hashes. And to
overwrite a large object's key does not mean that V8 iterates over a
long list. I think the whole story is overrated. Just give your post
body size a limit that makes sense (can be very small depending on
what you expect), then try to parse the string. I think it's no
problem to deal with this issue.
But it's important to set the limit!

What you're describing is what others might call a "footgun". Better to lobby for a proper fix in v8.

[José de Zárate]

and how the heck are we supposed to "lobby"?? flooding google's v8 hq with emails? I'm absolutely in for that ;-)

--

Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
uh, oh.

[Zachary Scott]

@mikeal, @phoscur What's the best way to set the max size? Is this just for POST requests?

2011/12/29 José de Zárate <jza...@gmail.com>

[Dave Clements]

Let me just see if I understand:

1) receiving POST data without imposing limits causing a vulnerability
to DoS attacks
2) we can limit the size of post request
3) we can limit how many fields

so on a per request basis, that's no problem (e.g. you can't overload
it with one request)

However, it can be overloaded by multiple requests, say by using XSS
attacks to formulate many automated Javascript post requests

So what about the final suggestion, (as a precaution prior to a fix)
limiting the CPU processing? We could use os.cpus()[0].model.times?

[Arnout Kazemier]

2/3 those are work arounds, they do not fix the underlaying problem.

This is not just an issue for POST request but also for parsing JSON.

or anything that gets converted to an object that would be received from

an unknown source.

[chjj]

On Dec 28, 9:48 pm, jmar777 <jmar...@gmail.com> wrote:
> E.g., express or whatever could provide a
> method interface for querystring and body values that auto-prefixes
> keys with a random value, busting pre-computed hash collisions... but
> that's nasty.

That is the most disgusting thing I've ever heard. Please apologize
for suggesting it!

[jmar777]

> That is the most disgusting thing I've ever heard. Please apologize
> for suggesting it!

Haha. Ok, I...

1) ...apologize for the suggestion
2) ...am open to better framework-level suggestions (that aren't just
arbitrary size limits... somewhere, sometime, an application is going
to need 8MB+ POSTs that don't brick the server)
3) ...did say that it was nasty :p

[Mark Hahn]

>  That is the most disgusting thing I've ever heard.  

I've heard woise.  And I'd do this in a heartbeat if I was being attacked.

 

[Tim Caswell]

I wonder if harmony proxies can be used to somehow intercept key
settings on all objects. It would slow stuff down, but would make it
secure till V8 fixed their hash algorithm.

On Thu, Dec 29, 2011 at 1:39 PM, Mark Hahn <ma...@hahnca.com> wrote:
>>  That is the most disgusting thing I've ever heard.
>
> I've heard woise.  And I'd do this in a heartbeat if I was being attacked.
>
>
>

[Arnout Kazemier]

The fastest workaround for this would just be adding a counter to the decoder, and have it break out the `forEach` loop after

it has stored about 1k keys in the `obj` https://github.com/joyent/node/blob/master/lib/querystring.js#L163-187

If you are doing a lot of JSON parsing you can just use the extra `reviver` parameter in JSON.parse to limit

the amount of keys that you are returning.

These are both simple and quick monkey patch that would minimize the impact of this attack, until V8

fixes this, if it would ever get fixed.

[Dominic Tarr]

it sounds like this attack could be used in any way that an untrusted user can get data onto your server.

thus, the only realistic solution is to fix it in v8, so that it's impossible for an attacker to predict what hash a key will get. that should be as simple as giving each v8 instance a randomized salt to hash with?

I don't know the V8 code at all, but it sounds like it might be a 1-5 line fix?

[maga]

> Those guys say they told Google October 18th, they got through to the
v8 guys in November, and they said they don't care sooo much about DoS
attacks on v8 because they're mainly interested in browserside stuff.

Why do they think it doesn't affect clients? What if I send compromised JSON to browser and parse it there with JSON.parse()? Especially if I'm an API provider and can send it through another sites? 

[arunoda.s...@gmail.com]

I think Joyent and node core team must have a look on this and patch the v8
And release a fixed version of node.  
I think this will be the way to do that.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

--

Arunoda Susiripala

@arunoda

https://github.com/arunoda

http://www.linkedin.com/in/arunoda

[Ryan Dahl]

On Fri, Dec 30, 2011 at 2:39 AM, arunoda.s...@gmail.com
<arunoda.s...@gmail.com> wrote:
> I think Joyent and node core team must have a look on this and patch the v8
> And release a fixed version of node.
> I think this will be the way to do that.

Yep. Hang tight. v0.6.7 is coming up soon.

[arunoda.s...@gmail.com]

Great.  
That's great and we can have trust on Node :-)

[Liam]

More discussion here:

https://plus.google.com/103939784105159993090/posts/Sh2Nxw8p3WB
https://plus.google.com/116910304844117268718/posts/BfTdM25d23y

[MK2]

Out team member had created the attack querystring.
And I test them in my macbook, there is the test results.

http://fengmk2.github.com/mk2blog/blog/2011/hac-in-nodejs-results.html

[dhruvbird]

Great! I hope this fixes javascript hashes to have a worst case O(log
n) behaviour rather than a worst case O(n) behaviour.

Regards,
-Dhruv.

On Dec 30 2011, 6:08 am, Ryan Dahl <r...@tinyclouds.org> wrote:
> On Fri, Dec 30, 2011 at 2:39 AM, arunoda.susirip...@gmail.com

[Joe Stein]

A WAF is always an option also https://www.owasp.org/index.php/Web_Application_Firewall

as are IDS too if you are so worried/inclined/required to

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop
*/

[Michael Jackson]

That's only true if you're using a library that buffers the entire POST body, like Express.

[tjholowaychuk]

Express doesn't make you do anything, you can opt-in to that behaviour
if it's what you want. Most people dont like streaming JSON because
etc it's terrible to work with

[Ryan Dahl]

Update. Bert Belder is doing the work to fix V8. He has a preliminary
patch which works but does not with snapshots. He's working with Erik
Corry from V8 to get the snapshot support in place.
http://codereview.chromium.org/9083001
Bert is hoping to get this resolved in the next day or two, after
which we will release v0.6.7. If the patch ends up taking longer, we
will instead apply an easy fix for v0.6.7 (a randomized prefix to the
querystring and http headers objects) until the full V8 fix is done.

[Nicolas Chambrier]

Could you provide the code you used to reproduce AND fix the issue ? That would help a lot of people to understand all this shit :-)

[Mihamina Rakotomandimby]

On 01/04/2012 03:47 PM, Nicolas Chambrier wrote:
> Could you provide the code you used to reproduce AND fix the issue ?

http://packetstormsecurity.org/files/108294/HashtablePOC.py.txt

--
RMA.

[chjj]

Looks like it landed in the master branch today:

https://github.com/v8/v8/commit/0a9c3fb1b51d501c5ad0ed71bf88faca1edf0697

[mscdex]

On Jan 4, 3:49 pm, chjj <chjjeff...@gmail.com> wrote:
> Looks like it landed in the master branch today:
>
> https://github.com/v8/v8/commit/0a9c3fb1b51d501c5ad0ed71bf88faca1edf0697

Looks like it's only effective if you compile without snapshots

[Vyacheslav Egorov]

Even if you compile with snapshots one has to guess seed embedded into _your_ snapshot to generate colliding data.

--
Vyacheslav Egorov

[Ilya Dmitrichenko]

On 4 January 2012 22:02, Vyacheslav Egorov <veg...@chromium.org> wrote:
> Even if you compile with snapshots one has to guess seed embedded into
> _your_ snapshot to generate colliding data.
>
> --
> Vyacheslav Egorov

Of course, and by the way - most linux systems save and restore their
random seed, which is, as I understand, used by most of the
applications out there, unless one desires to implement a more tricky
thing - pick up a second of noise from a sound card for example and do
some computation on that.

As a side question, can this fix possibly be implemented in C++ OR as
an exclusive piece of javascript, i.e. that will be not included in
the snapshot?
... or may one could simply reload this implementation and hence
redefine the functions, am I correct on this?

[Dave Clements]

> pick up a second of noise from a sound card for example

that would be..so..cool.

[Ryan Schmidt]

On Jan 5, 2012, at 18:39, Dave Clements wrote:

>> pick up a second of noise from a sound card for example
>
> that would be..so..cool.

Of course not all physical or virtual servers have sound hardware.

[Dave Clements]

*shrug* graceful degradation?... there are a LOT of virtual servers
obviously minus sound hardware though