Dynamically Setting Hystrix configuration for Unit Tests

[JD Rosensweig]

Hey All,

   I'm trying to test some of my Hystrix commands with some unit tests.  I would like to test the behavior of each when they encounter response rejected, time out, circuit breaker open, and other failure scenarios.  I can't modify the config with the command itself because my constructor does not accept the hystrix config class.  So in lieu of this I thought that maybe setting a system property would do the trick.  This has worked well in my main application when passed in with the -D option.  However when I try something like:

System.setProperty("hystrix.threadpool.default.coreSize", "1");

or

System.setProperty("hystrix.threadpool.HystrixThreadPoolKey.coreSize", "1");

   These don't seem to affect the properties at all.  Is there a better way to override the defaults so I can do these specialized tests?

Thanks,

JD

[Matt Jacobs]

Good question, JD.  We did something similar to check plugin initialization in these unit tests: https://github.com/Netflix/Hystrix/blob/master/hystrix-core/src/test/java/com/netflix/hystrix/strategy/HystrixPluginsTest.java.  These were flaky enough that I ended up commenting them out (for now).  Are your testing flaky or 100% failing?

A different option if you can't use constructor args it to register a custom properties strategy via https://github.com/Netflix/Hystrix/wiki/Plugins#properties-strategy.  This gets consulted before the defaults are used, so you've got a chance to insert whatever behavior/thresholds you'd like in this properties strategy.

Let me know if those pointers are helpful.

-Matt

[Senthilkumar Gopal]

I have used ConfigurationManager.getConfigInstance().setProperty() in Junits successfully.

On Tuesday, April 14, 2015 at 9:44:04 PM UTC-7, JD Rosensweig wrote:

[JD Rosensweig]

The ConfigurationManager worked very well for me.  However I couldn't seem to set this option:

hystrix.command.default.circuitBreaker.forceOpen

   The logs always show this as false.  Even when I set it to "true" like so:

final String configName = "hystrix.command.default.circuitBreaker.forceOpen";

ConfigurationManager.getConfigInstance().setProperty(configName, "true");

   Am I doing something wrong here?

Thanks,

JD 

[Matt Jacobs]

This worked as expected when I tried it out.  See my branch here with the code: https://github.com/mattrjacobs/Hystrix/commits/demonstrate-forcing-circuit-open-via-default-property

[JD Rosensweig]

Hey matt,

   Thanks for the example.  I investigated some more on my side and I think the reason why I thought it wasn't open was because the command was indeed failing, but I have a fallback which does some things on failure and then I use a function like this to figure out the failure reason for my logs:

    private String getFallbackReason() {

        String reason = "Unknown";

        if (isCircuitBreakerOpen()) {

            reason = "Circuit Breaker Open";

        } else if (isResponseRejected()) {

            reason = "Response rejected";

        } else if (isResponseTimedOut()) {

            reason = "Response timed-out";

        }

        return reason;

    }

   When I run my test, isCircuitBreakerOpen ends up being false.  So I get Unknown for the failure reason.  I attached my debugger and I found that isCircuiteBreakerOpen is indeed returning false, but isResponseShortCircuited() is returning true.

   Should I expected isCircuitBreakerOpen to return false in this scenario?  Does your test case show the same thing?

    Or should I be relying on isResponseShortCircuited() to get the "failure type"?

Appreciate our help.

Thanks,

JD

[Matt Jacobs]

This is indeed a bug.  I just demonstrated it in the unit test added by: https://github.com/mattrjacobs/Hystrix/tree/circuit-forced-open-by-properties-queried-properly.  I'll get that fixed up shortly.

Also, that unit test contains an example of how to get the result of a Hystrix command execution, as long as you've got a handle on the command instance: command.getExecutionEvents().  This returns a list of HystrixEventType enums, which in this case is [SHORT_CIRCUITED, FALLBACK_SUCCESS]

[Matt Jacobs]

Now fixed in https://github.com/Netflix/Hystrix/pull/770

[JD Rosensweig]

Great.  Thanks Matt!

By the way.  On another note.  Do you have any recommendations for testing response rejected?  I want to unit test this code path in my various commands, but I'm not sure of the ideal way.

I've been setting the coreSize to 1 and queuing up two commands.  But this is not always consistent.

Thanks,

JD

[Matt Jacobs]

This is done in a few places in the unit tests.  Here's one such example: https://github.com/Netflix/Hystrix/blob/master/hystrix-core/src/test/java/com/netflix/hystrix/HystrixCommandTest.java#L1534

The gist is:

* Configure a semaphore/threadpool to have a low size (call it n)
* Put n commands in the semaphore/pool in a nonblocking way (queue()/observe()), and they should all have fairly high latency

* The next command should get rejected, so you can add your assertions around that

[JD Rosensweig]

Thanks!

[ay...@kramphub.com]

I am facing similar issues while unit testing my code for Hystrix timeout . In one of my tests at run time i am changing the timeout property using System.getProperty () . I have a series of other hystrix tests in same command . 

Problem : When I run my timeout test independently it works fine and dynamic property is picked . But if i run all the tests together then my dynamic timeout property is not picked instead the default one read from properties file is used . 

Is it  because the first test execution initializes HystrixCommand and corresponding plugins which are not changed at run time . 

System.setProperty("hystrix.command.publishMessage.execution.isolation.thread.timeoutInMilliseconds", "1");