FallbackIsolationSemaphoreMaxConcurrentRequests when using ExecutionIsolationStrategy.THREAD

[Chris Mathews]

Scenario:

I want to limit the number of concurrent requests for a particular HystrixCommand however the traffic pattern is sporadic and bursty. Volume will often go above the thread pool size momentarily but I don't want to increase the pool size and I also don't want to start rejecting immediately once we hit the limit.  To accommodate this, I am setting a MaxQueueSize and QueueSizeRejectionThreshold.  Any beyond this requests beyond this queue limit I want to go to fallback.

Settings:

• ExecutionIsolationStrategy = THREAD
• CoreSize = 5
• MaxQueueSize = 15
• QueueSizeRejectionThreshold = 15
• ExecutionIsolationThreadTimeoutInMilliseconds = 5000

When I put this scenario under failure load I started seeing:

• com.netflix.hystrix.exception.HystrixRuntimeException: XXX could not be queued for execution and failed retrieving fallback.
  
• Caused by: java.util.concurrent.RejectedExecutionException
• With the underlying FallbackException: com.netflix.hystrix.exception.HystrixRuntimeException: XXX fallback execution rejected.

Looking at the code I see that HystrixCommand.getFallbackWithProtection() is using Semaphore to limit the number of concurrent Fallback executions.  I know that I can control this by setting FallbackIsolationSemaphoreMaxConcurrentRequests equals to sufficiently high number (in my case probably Integer.MAX_VALUE since the fallback response is static).  This behavior ended up surprising me a little bit because I thought that FallbackIsolationSemaphoreMaxConcurrentRequests was only relevant to ExecutionIsolationStrategy.SEMAPHORE.  I just wanted to confirm that this was intended behavior because neither the javadoc nor the wiki make mention of this scenario.

[Ben Christensen]

The FallbackIsolationSemaphore is completely separate from the normal ExecutionIsolationStrategy THREAD/SEMAPHORE selection.

The fallback isolation is a safety-net for network calls being done inside the getFallback() method. Like most Hystrix defaults it is set fairly aggressively. In your case since you have consciously asserted your fallback is static then yes, it could be Integer.MAX_VALUE.

Sorry if it isn't clear in the docs, the "fallback.isolation" and "execution.isolation" are separate things: https://github.com/Netflix/Hystrix/wiki/Configuration#fallbackisolationsemaphoremaxconcurrentrequests

Where in the docs would make sense to clarify this?

-- 
Ben Christensen - Edge Service Platform Team
+1.310.782.5511 @benjchristensen

--
You received this message because you are subscribed to the Google Groups "HystrixOSS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hystrixoss+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Chris Mathews]

This will probably sound stupid but I think it is just that the Configuration documentation has the fallback settings listed in the Execution section.  Up until that point the documentation focuses on "*isolation.thread*" vs "*isolation.semaphore*" which is only applied if the matching ExecutionIsolationStrategy is set. So I fell into the habit of associating anything "*isolation.semaphore*" with ExecutionIsolationStrategy SEMAPHORE.

If a section was broken out explicitly for Fallback and minor verbiage was added that said it applied to both ExecutionIsolationStrategy THREAD and SEMAPSHORE then I might not have been confused... then again who knows? :)

Unless you disagree or get to it first (and I know how much everyone loves to document), I will create a pull request with my suggested updates.

[Ben Christensen]

I think the changes make sense, I would appreciate your assistance improving the documentation.

It's a wiki so go ahead and edit directly.

Thank you again for being willing to help improve it.

-- 
Ben Christensen - Edge Service Platform Team
+1.310.782.5511 @benjchristensen

[Chris Mathews]

Updates made to https://github.com/Netflix/Hystrix/wiki/Configuration and the sidebar.

[Ben Christensen]

Thank you.

-- 
Ben Christensen - Edge Service Platform Team
+1.310.782.5511 @benjchristensen