Capture exception thrown by @HystrixCommand fallback?

[Eric Deandrea]

I posted this out to StackOverflow as well (http://stackoverflow.com/questions/40495259/capture-exception-thrown-by-hystrix-fallback) but thought I would ask it here.

I'm designing a service facade and I have a method signature that looks like this:

public Policy getPolicy(long policyId) throws PolicyNotFoundException

If nothing bad happens then a Policy object (simple POJO) is returned. If the requested policy is not found then a checked exception PolicyNotFoundException is thrown (just as a reference - we follow this article when it comes to best practices on exception handling within an application).

The layer above the service facade layer (in this case a Spring MVC RestController) knows how to handle such a PolicyNotFoundException and return an appropriate payload.

I'm trying to incorporate this into a HystrixCommand by doing something like this:

@HystrixCommand(groupKey = "PolicyService", fallbackMethod = "getPolicySafe", ignoreExceptions = { PolicyNotFoundException.class })
public Policy getPolicy(long policyId) throws PolicyNotFoundException {
    LOGGER.info("Getting policy {}", policyId);

    // Simulate some error condition for testing purposes
    throw new RuntimeException("Something happened!");
}

private Policy getPolicySafe(long policyId, Throwable t) throws PolicyNotFoundException {
    LOGGER.warn("Falling back to circuit-breaker for getting policy {}", policyId, t);
    throw new PolicyNotFoundException(policyId);
}

Basically I want my circuit breaker to simply behave as if the policy wasn't found by the original lookup. The problem I'm having with this though is the exception I throw from the fallback method is getting lost in translation somewhere. The exception I end up seeing in the layer above is the RuntimeException thrown by the command method and not the exception thrown by the fallback method. Is there a way around this? I don't want to change the contract of my original method either nor do I want the layer above this to know anything other than to have to catch PolicyNotFoundException in the case a policy isn't found. Whatever is needed here should be captured within this service facade layer.

Any and all help would be greatly appreciated. Thanks!

[Eric Deandrea]

I did find this (https://github.com/Netflix/Hystrix/issues/1344) and this (https://github.com/Netflix/Hystrix/pull/1397). While this is a step in the right direction it does not completely solve the issue. Think about a developer who is using the MVC pattern - the service facade layer of the application is where all of this hystrix "stuff" comes into play. The API of that layer is the contract between this layer & the layer above it (for a web application - some controller/restful API layer). This solution "pollutes" that contract.

This addition makes the caller of the service facade layer have to know specific details that Hystrix is involved by having to catch HystrixRuntimeException, look for a fallback exception (which is generally a FallbackInvocationException) and then have to read the cause from that.

Why can't we simply throw whatever exception is thrown from the fallback?

[Eric Deandrea]

Actually I was able to solve my use case using this approach. I posted my solution back on my stackoverflow post: http://stackoverflow.com/questions/40495259/capture-exception-thrown-by-hystrix-fallback

[Eric Deandrea]

Actually it seems that this only works while the circuit is closed. Once the circuit is tripped open the HystrixRuntimeException gets propagated back up.

com.netflix.hystrix.exception.HystrixRuntimeException: getPolicy short-circuited and fallback failed.

at com.company.common.api.FaultBarrier.defaultErrorHandler(FaultBarrier.java:84)

at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)

at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)

at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:375)

at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:59)

at org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:136)

at org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:74)

at org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1193)

at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1030)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:980)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:745)

Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: getPolicy short-circuited and fallback failed.

at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:818)

at com.netflix.hystrix.AbstractCommand$22.call(AbstractCommand.java:793)

at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)

at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)

at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)

at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1454)

at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1379)

at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)

at rx.observers.Subscribers$5.onError(Subscribers.java:230)

at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44)

at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)

at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)

at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)

at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)

at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)

at rx.Observable.unsafeSubscribe(Observable.java:10151)

at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)

at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)

at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)

at rx.Observable.subscribe(Observable.java:10247)

at rx.Observable.subscribe(Observable.java:10214)

at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)

at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)

at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:377)

at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:343)

at com.netflix.hystrix.contrib.javanica.command.CommandExecutor.execute(CommandExecutor.java:52)

at com.netflix.hystrix.contrib.javanica.aop.aspectj.HystrixCommandAspect.methodsAnnotatedWithHystrixCommand(HystrixCommandAspect.java:100)

at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)

at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)

at com.aig.entitlementsrs.service.PolicyServiceImpl$$EnhancerBySpringCGLIB$$a8f67a15.getPolicy(<generated>)

at com.aig.entitlementsrs.api.PoliciesApi.getPolicy(PoliciesApi.java:73)

at sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)

at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)

at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)

... 55 common frames omitted

Caused by: java.lang.RuntimeException: Hystrix circuit short-circuited and is OPEN

at com.netflix.hystrix.AbstractCommand.handleShortCircuitViaFallback(AbstractCommand.java:965)

at com.netflix.hystrix.AbstractCommand.applyHystrixSemantics(AbstractCommand.java:546)

at com.netflix.hystrix.AbstractCommand.access$100(AbstractCommand.java:59)

at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:408)

at com.netflix.hystrix.AbstractCommand$4.call(AbstractCommand.java:402)

at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)

... 102 common frames omitted