HystrixCommand for http calls

[kumar ankit]

First of all the use case: My application is making http calls to a remote server. So, on for each request, I am creating a new HystrixCommand Object and wrapping the network call in it.

Some guidelines I want to seek are:

1. Is creating a new HystrixCommand object, the correct practice? Is there a way to share the same HystrixCommand object ?

2. On the Hystrix dashboard, I can see the threadpool size associated with my command group, but the number of 'active' thread is always a way less than than the threadpool size even though the number of requests in the rolling window are sufficiently high. Is my threadpool being under under-utilized ?

3. What is the cost associated with wrapping the http call inside HystixCommand, as in my case I am seeing the latency of the calls going up ?

Thanks.

[Chris Mathews]

1. Yes creating a new HystrixCommand is the correct practice, in fact the library enforces this by throwing an error if you try to "reuse" a command that has already been executed.  I would not be concerned about the cost of creating a new object on each call... the JVM is pretty good at creating objects. :)

2. Personally, I haven't noticed anything that I find concerning with the active thread count being reported by the dashboard. Do you have specific details?  

3. I haven't done any in depth micro-benchmarking for my project but I have not noticed much overhead.  Anecdotally, I would say a couple milliseconds at most... well under the threshold for most people to care. Can you explain what you mean when you say that you are seeing the latency go up? Again specifics would be most helpful. How are you testing? Are you allowing warm-up time in your perf tests? Does the performance degrade over the course of the test or remain constant? Can you eliminate the network call to isolate the test?

[kumar ankit]

Hi Chris,

Thanks for your insights.

1. Fully resolved. :)

2. I created a pool with core size 15, so on the dash-board, the pool size is correctly being displayed as 15, but the number of active threads is being displayed as 1 most of the time. So,if the pool size is 15 and the qps is around 400, isn't it expected for more threads to be active to enhance the performance?

3. I am providing the warm up time for perf- tests. Prior to wrapping the call in Hystrix command, the latency was around 1.5 milliseconds, post wrapping it went up to 3-3.2 millisecond. On our scale it is getting doubled, so I wanted to checkout if some inefficiency in using Hystrix is there ? That is what made me think about the less number of active threads than the allocated pool size.

4. On a further note, I am executing the HystrixCommand synchronously. Will using it asynchronously give some gain in the performance considering the use as:
      Future<Object> futue = hystrixCommand.queue();
      future.get().

Thanks.

[Chris Mathews]

2. What is generating your load?  Is there think time between requests?  Given that each operation is only taking 3ms (based on the other details you provided) then it seems reasonable you could manage close to 400 qps with a thread or two. My guess is your client just isn't moving fast enough to keep your thread pool busy... otherwise I would expect your qps to be much higher.

3. Correct me if I am wrong but I get the feeling that you are testing essentially a no-op operation?  I doubt you are getting 1.5 milliseconds response time on call that is leaving your JVM... if you are then fantastic for you!  Going from 1.5 milliseconds to 3 milliseconds, while double the time, certainly doesn't imply that Hystrix adds 100% overhead to an operation.  Most remote calls in my experience take longer enough where an additional 2 milliseconds isn't going to hurt anything especially if you factor in the increased stability that implementing a solution like Hystrix is going to bring.

4. Calling hystrixCommand.execute() and hystrixCommand.queue().get() are nearly equivalent. 

HystrixCommand.execute() implementation:

public R execute() {
  try {
    return queue().get();
  } catch (Exception e) {
    throw decomposeException(e);
  }
}

[kumar ankit]

2. Fair enough. I will check if it is really the client that is not moving fast enough.

3. 1.5 millisecond and 3 millisecond - the data I provided was when I hosted the service on localhost.And yes, the the additional latency is acceptable for me considering the   stability Hystrix provides, I just wanted to check out if it can be brought down.

4. Ok. That means essentially the same.

Thanks a lot for clarifying things.

[kumar ankit]

Hi Chris,

Everything is working fine for the application. But, I have this problem in the dash-board that is troubling me. I have created a CommandGroup in which there are two HystrixCommands. Each command is associated with its own Threadpool specified by threadpoolkey in the constructor, but on the dash board the thread associated with one thread-pool is being displayed as 0. Can you suggest any reason for that ? Also, for once I was able to see the actual size of the threadpool but as soon as the command got short-circuited, its size again started being displayed as 0 and so after. Can you think of any particular reason for this ?

How actually the threadpool behave in hystrix ?

Thanks,

[Ben Christensen]

Also, for once I was able to see the actual size of the threadpool but as soon as the command got short-circuited, its size again started being displayed as 0 and so after. Can you think of any particular reason for this ?

When it is short-circuited there is no work being done on the thread-pool. It is all being short-circuited.

[Ben Christensen]

2. What is generating your load?  Is there think time between requests?  Given that each operation is only taking 3ms (based on the other details you provided) then it seems reasonable you could manage close to 400 qps with a thread or two. My guess is your client just isn't moving fast enough to keep your thread pool busy... otherwise I would expect your qps to be much higher.

Correct. The "active" count is a point-in-time snapshot, so with very low latency calls such as your use-case had, you'll rarely see high thread counts.

The "Max Active" count is more interesting as it shows the max number achieved during the past 10 seconds. 

In both of these though, if you are doing work that only takes 1.5ms then there will be very little concurrent execution at 400qps since even distribution of those 400 queries requires 0.6threads (400 queries * 0.0015 seconds = 0.6 threads … or 400 queries * 1.5ms = 600ms). If you have bursts then you will see that concurrency shown in the "Max Active", but the snapshot "Active" counter will very rarely show the bursts since it is capturing a point in time.

[kumar ankit]

Hi Ben, 

I just wanted to know how exactly the thread-pool works ? I mean if I have a super() call in the constructor like: 

super(HystrixCommand.Setter

                .withGroupKey(HystrixCommandGroupKey.Factory.asKey("abc"))

                .andCommandKey(HystrixCommandKey.Factory.asKey("Client"))

                .andThreadPoolKey(HystrixThreadPoolKey.Factory.asKey("Client-Thread-Pool"))

                 .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties.Setter().withCoreSize(DEFAULT_THREAD_POOL_SIZE).withmaxQueueSize(15))  );

and for every request I am creating a new HystrixCommand(). So how does the thread pool actually work ? I assume with the creation of 1st HystrixCommand object, the threadpool is instantiated and the same is used for the subsequent command objects. Also, what happens when the circuit becomes OPEN and then afterwards getting CLOSED ? I mean when in my case the circuit became open, I was seeing the Pool size as 0 in the dash-board, and even after the circuit got CLOSED the pool size remained at 0. It did not come back to original value. Am I using something in a non-recommended way ?

[Ben Christensen]

The first time a command is invoked for a given THREAD_POOL_KEY it creates the underlying pool and it lives for the life of the app (or until Hystrix.reset() is called).

If a HystrixCommand times out then the underlying thread is interrupted (unless that config is disabled), but if the thread ignores interruption it will stay blocked until it clears naturally.

High level information is available at:

https://github.com/Netflix/Hystrix/wiki/How-To-Use#wiki-CommandThreadPool

https://github.com/Netflix/Hystrix/wiki/How-it-Works#threads--thread-pools

The code for the pool is at:

https://github.com/Netflix/Hystrix/blob/master/hystrix-core/src/main/java/com/netflix/hystrix/HystrixThreadPool.java

The factory for creating the pool is at: 

https://github.com/Netflix/Hystrix/blob/master/hystrix-core/src/main/java/com/netflix/hystrix/HystrixThreadPool.java#L92

[Ben Christensen]

That could be a bug … the thread pool visualization is not one we spend a lot of time looking at so it wouldn't shock me if pool size was wrong.

-- 
Ben Christensen - Edge Service Platform Team
+1.310.782.5511 @benjchristensen

On October 7, 2013 at 11:23:30 AM, kumar ankit (k.an...@gmail.com) wrote:

Thanks a lot Ben. 

I was also thinking the same. But that still left me unclear on why when a circuit becomes OPEN , the PoolSize on the Dashboard becomes 0. I mean the underlying pool is still there. So, ideally this should not happen. Is this an issue with the dash board ? And even when the circuit gets CLOSED after sometimes, the PoolSize still gets displayed as 0 on the dash board.
 

--

Kumar Ankit
Ph: +91 9663877355

[Payal Maheshwari]

I have a very basic question -

If the Circuit Breakers are Force Closed using - withCircuitBreakerForceClosed(true). and the thread pool ( of default 10 threads)  exhausts , would the command fail and fallback would be called? 

Thanks.

[Ben Christensen]

It would start being rejected (thread pool and queue full) and go through fallback. 

Ben Christensen

310.782.5511  @benjchristensen

--
You received this message because you are subscribed to the Google Groups "HystrixOSS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hystrixoss+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Payal Maheshwari]

Thanks Ben! Is there any way, I can not use Hystrix thread pool by default?  or set it in a way that the further requests are not rejected.

we have introduced hystrix commands to connect to some of our external vendors and the only feature we want to utilize with hystrix is "Time Out" . No Circuit breakers and no request rejections.

[Ben Christensen]

Yes you can. All of the configuration options are shown here: https://github.com/Netflix/Hystrix/wiki/Configuration

Note however that timeout is not sufficient alone to prevent resource exhaustion if your system is high throughput. All threads in a system can be saturated before timeouts have a chance to do anything. You won’t have many of the benefits of Hystrix if just using timeouts, as you can achieve that just at the network level. Your application server will fall over and start rejecting/queueing incoming requests anyways, so you’ll end up rejecting, just not in a controlled, recoverable manner.

[Payal Maheshwari]

Thanks! We are planning to introduce Circuit breakers in next release since Hystrix was new to us we planned to just go with time outs.

I will check the configuration to not use the thread pools by default.

[Payal Maheshwari]

The thread pool of default 10 threads is per Hystrix Command. Do we identify a command with the Command key specified?

If I have created several instances of a Hystrix command, would they share the same thread pool? or would they have 10 threads each allocated for the task?

[Ben Christensen]

It is not by command, but defaults to a pool per Group. You can specify Group, CommandKey and ThreadPoolKey per HystrixCommand. 

If you have several instances of a single HystrixCommand they will all share the same thread pool. 

The wiki contains details on config options and defaults. Config can be set via code or properties. 

Ben Christensen

310.782.5511  @benjchristensen