Circuit breaker and DOS attack

[Adam Dyga]

Hello,

I've got a question about circuit breaker in hystrix. Please tell me if my doubts are correct or not.

When a hystrix command throws an exception from run() method, the event counts against failure metrics. If the error occurs "often enough" the circuit breaker finally trips the circuit and all subsequent command invocations will fail immediately (for some time, until the circuit is closed again).

I'm wondering if this mechanism can be abused to perform a DOS attack. Let's say that an attacker, for instance, finds a way to trigger a NullPointerException in the command (or any other error eg. by passing special combination of parameters to a web-service which uses hystrix command). 

Then by repeating the request constantly, the circuit will get tripped  and remain open "cutting off" *all* legitimate users from accessing the target resource, which results in a DOS. Is that correct? 

What can be done in the code to mitigate this? One of solutions is to let out only those exceptions from HystrixCommand.run() method which are *really* known to be external system failures and catch and silence everything else, for instance, using HystrixBadRequestException (ie. catch(Exception e) { throw HystrixBadRequestException(e); } in almost every command). Is this the way how real-life commands should be implemented?

Regards,

Adam Dyga

[Ben Christensen]

Yes, if someone were to figure out how to cause your command to fail every time with certain request arguments they could DOS it.

The approach you suggest could work, but in my experience is the opposite I pursue since no one can ever know all “real” failures - that’s kind of the point of Hystrix. Complex systems fail in all kinds of ways. If you only bulkhead against failures you know, then you are vulnerable to failures you don’t know about, and those are the ones that take down production systems.

I suggest instead doing argument validation, similar to how SQL-injection attacks are prevented by validating input. This way you can fail a user before even executing the command, or if you have to go back to the server (execute the command) before knowing if input is invalid, then the run() method can throw a HystrixBadRequestException when it is a user error (bad input arguments).

Thus, I suggest the opposite approach of capturing known user errors but allowing all others to throw.

-- 
Ben Christensen - Netflix Edge Engineering
+1.310.782.5511  @benjchristensen

--
You received this message because you are subscribed to the Google Groups "HystrixOSS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hystrixoss+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.