Hystrix With Spring MVC using AOP (Am I doing it correctly?)

1,337 views
Skip to first unread message

pranav tiwary

unread,
May 22, 2015, 7:02:45 AM5/22/15
to hystr...@googlegroups.com
I have an existing Spring MVC web app. My application communicate to NoSql Db and In case of Fallback I have MYSQL. Using Hystrix I have to manage these and also keep the concept of circuite breaker and concurrency that Hystrix provides.

Here is What, that I am doing

1. My Controller, Call my service layer : This service layer is basically my HystrixCommandService , which will call the actual Nosql or DB service.
2. Also I have to change some property on runtime, may be due to load, failure etc. Here is the piece of code that i am writing,


My Question : Is my code is production ready, Or Is there another best way to integrate hystrix to an existing spring MVC. Note I dont have spring booot or cloud..

Code : the below class will be called from controller layer.

My HystrixService, which i will put before the actual existing services

@Component
public class AccountCommandService implements IAccountCommandService{
   
    @Autowired
    @Qualifier("accountNoSQLervice")
    private IAccountService nosqlService;
   
    @Autowired
    @Qualifier("accountDBService")
    private IAccountService dbService;
   
    @Autowired
    @Qualifier("bankService")
    private IAccountService bankService;
   
    public Account getAccountDetails(String id){
        return new AccountHystrixCommand(id, nosqlService, dbService, bankService).execute();  // THIS IS MY COMMAND
       
    }
}


My Hystrix Command Class



public class AccountHystrixCommand extends HystrixCommand<Account> {
   
    private IAccountService nosqlService;
    private IAccountService dbService;
    private IAccountService bankService;
   
    private String accountId;
   
  // Archaius to change property on runtime
    static DynamicIntProperty noOfThread =
              DynamicPropertyFactory.getInstance().getIntProperty("NO_OF_THREAD", 1000);
         
   
    public AccountHystrixCommand(String accountId, IAccountService nosqlService,
            IAccountService dbService, IAccountService bankService) {
        super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("AccountService"))
                .andCommandKey(HystrixCommandKey.Factory.asKey("GetAccountDetails"))
                .andCommandPropertiesDefaults(HystrixCommandProperties.Setter()
                        .withExecutionTimeoutInMilliseconds(500)
                        .withCircuitBreakerSleepWindowInMilliseconds(2)
                        .withRequestLogEnabled(false)
                        .withCircuitBreakerRequestVolumeThreshold(2)// after 2 failure circuit will be tripped
                        )
                .andThreadPoolPropertiesDefaults(HystrixThreadPoolProperties.Setter()
                        .withCoreSize(noOfThread.get()) // increase thread pool size What should be optimum size, should it depends upon no of hits
                        )
                );
        // uncomment below line to open the circuit
        ConfigurationManager.getConfigInstance().setProperty("hystrix.command.GetAccountDetails.circuitBreaker.forceOpen", true);
        // uncomment below to default every circuit to open
        //ConfigurationManager.getConfigInstance().setProperty("hystrix.command.default.circuitBreaker.forceOpen", true);
       
        System.out.println("NO OF THREAD="+noOfThread.get());
        this.accountId = accountId;
        this.nosqlService=nosqlService;
        this.dbService=dbService;
        this.bankService=bankService;
    }

    @Override
    protected Account run() throws Exception {
        System.out.println("@Run AccountHystrixCommand thread="+Thread.currentThread().getName());
        return nosqlService.getAccountDetails(accountId);  // CALL MY ACTUAL FIRST SERVICE
    }
   
    @Override
    protected Account getFallback() {
        System.out.println("@getFallback AccountHystrixCommand thread="+Thread.currentThread().getName());
        return new AccountHystrixFallBackCommand(accountId, dbService, bankService).execute(); // CALL MY FALLBACK SERVICE which is another hystrix command

    }
}

Also I am trying another way using Annotation. I am not able to @HystrixCommand of javanica, as it nees spring boot and cloud I think. I am creating my own annotation and using AOP tryign to achieve that. I am taking help from javanica existign code. I am still working on it.

But I prefer this may that I pretty stright and feels like more control.

Matt Jacobs

unread,
May 22, 2015, 2:25:03 PM5/22/15
to hystr...@googlegroups.com
I can't speak to wiring with Spring, as I've never attempted that.  I can mention a few points about your config, though.

* The default for circuitBreakerSleepWindowInMilliseconds is 5000.  By reducing it to 2, you're very close to turning off circuit-breaking entirely.  Using a value of 2 means that only 2ms of load are shed for this circuit every time it trips.  What's your reasoning for this?
* The default for circuitBreakerVolumeThreshold is 20.  If you set it down to 2, that means that seeing at least 2 requests in the rolling window (by default 60 seconds) makes this eligible for the circuit to open.  This means that random failures on a low-volume circuit will be impactful.  I'm not sure if that applies to your scenario, but it's worth mentioning.  Your comment next to that config is incorrect.  Please see: https://github.com/Netflix/Hystrix/wiki/Configuration#circuitbreakerrequestvolumethreshold
* That number of threads seems quite high.  Setting this value is part art/part science, but it's generally much lower than 1000 for any reasonable system.  Setting it that high gives you no benefits of bulkheading.  You can think of this as the maximum number of concurrent requests that should be served.  If your average latency is 100ms, and you see 100 req/sec, then by Little's Law (http://en.wikipedia.org/wiki/Little%27s_law), the expected value of concurrent requests is 10.  Of course, this is a value with a distribution, and so it may burst far higher than that.  It's then up to you to determine how tolerant you are of going to fallback and how precious of a resource your threads are.  Those are questions that are specific to any system.
* Putting the call to the fallback service in a different Hystrix command is absolutely recommended, and described here:  https://github.com/Netflix/Hystrix/wiki/How-To-Use#fallback-cache-via-network.  You should be careful to make sure the main/fallback commands run on separate threadpools if you want their failures to be independent.  An alternate approach is described here: https://github.com/Netflix/Hystrix/wiki/How-To-Use#primary--secondary-with-fallback, if you ever intend to use both systems simultaneously.

Hope this helps,
Matt

pranav tiwary

unread,
May 25, 2015, 6:44:17 AM5/25/15
to hystr...@googlegroups.com
Thanks a lot Matt !! Was a great help.

Ahh, These were my test data to demo the tech team, about how and where hystrix are helping me.
  • circuitBreakerSleepWindowInMil : I will use default value only .
  • circuitBreakerVolumeThreshold : Again default or some value that may be greater than 20. (I misunderstood it earlier)
  • number of threads : 1000 was my test data, yes I would keep it somthing around 20, as we are expectign huge traffic for ecommerce, and these thread would be indulged in reading and writing to Aerospike nosql.
  • The failover command, again would be of same configurations : and is basically doing a RDBMS read and write. Also I am adding below line for fallback
    • .andThreadPoolKey(HystrixThreadPoolKey.Factory.asKey("RemoteServiceXFallback"))) 
  • Does rolling window cycle is of  default 60 seconds.

My another concern was if there is another better way of doing Hystrix with Spring if I am not using spring boot. I created a  new annotations kind of same as of @HystrixCommand of Javanica, but javanica need spring boot, and mine doesnt.

MY ANNOTATIONS

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface ExecuteByHystrix {
    String groupKey() default "";
    String commandKey();
    String threadPoolKey() default "";
    String fallbackMethod() default "";
    int threadPoolSize() default 10;
}

MY ASPECT : still is has some test config data.

@Around("@annotation(com.sd.annotation.ExecuteByHystrix)")
    public Object circuitBreakerAround(final ProceedingJoinPoint aJoinPoint) throws Throwable {
        final Method method = getMethodFromTarget(aJoinPoint);
        Assert.notNull(method, "failed to get method from joinPoint:"+ aJoinPoint);

        ExecuteByHystrix annotation = method.getAnnotation(ExecuteByHystrix.class);
        final String fallbackMethod = annotation.fallbackMethod().trim();

        String commandKey=annotation.commandKey().trim();
        if(commandKey.length()==0){
            throw new RuntimeException("Please provide Command attributes for @ExecuteByHystrix at "+ method.getClass());
        }
        String commandGroupKey=annotation.groupKey();
        if(null==commandGroupKey|| commandGroupKey.trim().length()==0){
            commandGroupKey= aJoinPoint.getSignature().toShortString();
        }

        HystrixCommand.Setter theSetter =
                HystrixCommand.Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey(commandGroupKey));
        theSetter = theSetter.andCommandKey(HystrixCommandKey.Factory.asKey(commandKey));

        String hystrixPropertyName="hystrix.command."+commandKey+".circuitBreaker.forceOpen";
        boolean circuitClose =
                DynamicPropertyFactory.getInstance().getBooleanProperty(hystrixPropertyName, false).get();


        ConfigurationManager.getConfigInstance().setProperty(hystrixPropertyName, circuitClose);


        HystrixCommand theCommand = new HystrixCommand(theSetter) {
            @Override
            protected Object run() throws Exception {
                try {
                    return aJoinPoint.proceed();
                } catch (Exception e) {
                    throw e;
                } catch (Throwable e) {
                    throw new Exception(e);
                }
            }
            @Override
            protected Object getFallback() {
                try {
                    if(null==fallbackMethod || fallbackMethod.length()==0){
                        System.out.println("There is no fallback method provided, Unable to get Data from "+method.getName());
                        return null;
                    }
                    //String accountId=(String) aJoinPoint.getArgs()[0];
                    Method method = aJoinPoint.getTarget().getClass().getMethod(fallbackMethod, String.class);
                    return method.invoke(aJoinPoint.getThis(), aJoinPoint.getArgs());
                } catch (Exception e) {
                    e.printStackTrace();
                }

                return null;
            }
        };
        return theCommand.execute();
    }

Pranav

Matt Jacobs

unread,
May 26, 2015, 10:45:09 PM5/26/15
to hystr...@googlegroups.com
I don't use Spring, so I can't really speak intelligently about your options.  You could open up a Github issue and cc @dmgcodevil - he's responsible for the hystrix-javanica integration and would know far better than me.

-Matt
Reply all
Reply to author
Forward
0 new messages