Calling Remotable service issue

[Stanislav Fujdiar]

Hi Jim,

we found two problems with calling remotable service:

1) methods with return type void

on service side I got exception:

[SEVERE 04.09.2015 08:26:14.575] The following exception was raised

org.oasisopen.sca.ServiceRuntimeException: Return value not serialized

        at org.fabric3.binding.zeromq.runtime.message.NonReliableRequestReplyReceiver$1.run(NonReliableRequestReplyReceiver.java:114)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

2) methods with more than one parameters

if I try to call method I got exception

org.oasisopen.sca.ServiceUnavailableException: java.lang.ClassCastException: [B cannot be cast to [Ljava.lang.Object;

        at org.fabric3.implementation.proxy.jdk.wire.JDKInvocationHandler.invoke(JDKInvocationHandler.java:103)

        at com.sun.proxy.$Proxy77.twoParamWithVoid(Unknown Source)

        at org.librade.Comla.onEvent(Comla.java:46)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.fabric3.implementation.reflection.jdk.ConsumerMethodInvoker.invoke(ConsumerMethodInvoker.java:44)

        at org.fabric3.implementation.pojo.component.InvokerEventStreamHandler.invoke(InvokerEventStreamHandler.java:95)

        at org.fabric3.implementation.pojo.component.InvokerEventStreamHandler.handle(InvokerEventStreamHandler.java:71)

        at org.fabric3.fabric.container.channel.FilterHandler.handle(FilterHandler.java:35)

        at org.fabric3.spi.container.channel.PassThroughHandler.handle(PassThroughHandler.java:26)

        at org.fabric3.channel.impl.SyncFanOutHandler.handle(SyncFanOutHandler.java:27)

        at org.fabric3.spi.container.channel.PassThroughHandler.handle(PassThroughHandler.java:26)

        at org.fabric3.fabric.container.channel.FilterHandler.handle(FilterHandler.java:35)

        at org.fabric3.spi.container.channel.PassThroughHandler.handle(PassThroughHandler.java:26)

        at org.fabric3.fabric.container.handler.TransformerHandler.handle(TransformerHandler.java:47)

        at org.fabric3.binding.zeromq.runtime.message.NonReliableSubscriber$SocketReceiver.run(NonReliableSubscriber.java:201)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassCastException: [B cannot be cast to [Ljava.lang.Object;

        at org.fabric3.binding.zeromq.runtime.interceptor.UnwrappingInterceptor.invoke(UnwrappingInterceptor.java:19)

        at org.fabric3.fabric.container.interceptor.TransformerInterceptor.invoke(TransformerInterceptor.java:57)

        at org.fabric3.implementation.proxy.jdk.wire.JDKInvocationHandler.invoke(JDKInvocationHandler.java:97)

        ... 22 more

I've created bugs in jira(Link to demo is in description)

https://fabric3.atlassian.net/browse/FABRIC-61

https://fabric3.atlassian.net/browse/FABRIC-60

best regards

Stanley

[Jim Marino]

Hi Stanislav,

Thanks for reporting these. I'll start investigating.

Jim

--
You received this message because you are subscribed to the Google Groups "fabric3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric3+u...@googlegroups.com.
To post to this group, send email to fab...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fabric3/f8feb874-90a2-4ca6-ac45-a3ddc0eae3c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jim Marino]

This definitely looks like a problem. I'll need to look further into the best way around this. One of the reasons this probably hasn't surfaced previously is that most of the remotable calls I have seen are modeled with one parameter. I think it is definitely OK to have multiple parameters but with ZMQ most of the apps I have dealt with take byte[] and deserialize directly using something like protobuffs or off-heap mechanisms for performance. 

I'm going to look further into this but as a workaround for the time being, can you try converting your multiple params to a single, complex type and let me know if that works?

Jim

[Tomáš Fecko]

with one input and one output parameter it's working...

Using byte[] for zeromq channels is a common use case. 

In distributed domain, remote sca calls doesn't look very good with byte[] as single parameter (having zeromq as default sca binding). Especialy, when you are planning moving composites from local to remote or opposite - which is one of the best features of sca.

[Tomáš Fecko]

there is also problem with exceptions, if exception is thrown remotely, it's not propagated...

so to sum that up, in version 2.5.3, with zeromq as sca binding, 

you can have remote methods with exactly one input parameter and one output parameter, without any throws clauses...

[Stanislav Fujdiar]

same problem as with return type void is if you have defined return type e.g. String and you want to return null (catched on reciever)

org.oasisopen.sca.ServiceRuntimeException: Return value not serialized

        at org.fabric3.binding.zeromq.runtime.message.NonReliableRequestReplyReceiver$1.run(NonReliableRequestReplyReceiver.java:114)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

problem with exceptions is catched on sender side

java.lang.ClassCastException: java.lang.RuntimeException cannot be cast to java.lang.String

        at com.sun.proxy.$Proxy77.oneParamWithReturn(Unknown Source)

        at org.librade.Comla.onEvent(Comla.java:30)

[Jim Marino]

Hi Stanislav,

You and Tomas bring up some really good points I'd like to explore further. To answer the issue directly, the ZMQ binding doesn't support exception propagation. Here's some background why.

My view is that remote calls are fundamentally different than local calls. This is because of the network (you have to account for latency and failure) and the more decoupled nature of clients and services. Consequently, remote calls should be modeled differently than local ones. 

Exceptions highlight these differences between remote and local calls. RMI propagates exceptions but this can lead to difficulties because clients do not necessarily share the same context as the service they are interacting with. Specifically, if a service propagates an exception that contains a cause whose type is not on the client classpath (e.g. MyServiceException is thrown by a service and that type is not on the client classpath), the client will not be able to load the exception. 

This is further complicated by interoperable protocols such as ZeroMQ's. Fault propagation would need to be designed into an application-level protocol. If a service threw an exception, we would need the binding to somehow propagate it using ZeroMQ message frames by (probably) creating some envelope and fault structure.

WS-* tried to model faults ten years ago and IMO it wound up really complicated and messy (CORBA did too before WS-*). My recommendation for an easier solution is to return errors as part of the response message with a code (the code can be put in an envelope header or be part of the payload). REST does this (the code goes into the HTTP header and messages are part of the HTTP return content) and it works really well since it keeps things simple. 

So that's some of the background why the ZeroMQ binding doesn't support raising exceptions. Could you return exceptions as a part of the response?

Jim

P.S. I'll address some of the broader issues with modeling remote calls in a response I'm going to write to Tomas.

--
You received this message because you are subscribed to the Google Groups "fabric3" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric3+u...@googlegroups.com.
To post to this group, send email to fab...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fabric3/314116db-89a2-4674-9171-3570d1d4aec9%40googlegroups.com.

[Tomas Fecko]

On Tue, Sep 15, 2015 at 10:39 AM, Tomas Fecko <tomas...@gmail.com> wrote:

Hi Jim,

You are right about the services, which you know for sure, that they will be remote and services where you know, you want them as performant as possible.
Using custom serialization is also a good point, I wanted to mention. We are using custom serialization which is something between Externalizable and Simple Binary Encoding.
It would be great, if there is a possibility to write directly and read directly from the stream (something like externalizable with fabric3 interface as parameter).
Creation of byte[] is just one step too many.

Back to the remote services. Usualy in projects I used fabric3, we know exactly which services should be remote and which should be performant.
We usualy used channels for remote fast data distribution and we used local services for services, we need fast access to. That's about 20% of services. 80% of them we don't realy care how fast they are, we just need them to work reliably.

But than we get to the phase, where more factors comes to the scene. e.g. in the project I'm working on right now. At the beginning the project was a prototype, with some basic security in mind.
Than when we tryed opening the project to the public, we wanted to finish the security layer to the point, we feel that it would suffice (you can't realy secure everything to 100%, you just want to make best effort for the money and time you have). At the project we are executing customer code in our system. We are executing jar-s, R code, python code, etc. and we wanted to simply relocate some of the services from the jvm of the code executor to different jvm in the same server instance (we are using ec2 and gce clouds interconnected with VPN). But there are instances we didn't wanted to separate JVMs, as they are running our own code. Creating a sandbox environment is a pain, and having that completely separated feeled more secure to us. So the services which touch databases (rds, kvs, ...) or anything that's not public, are not in the same JVM as the third-party-code we execute. We even wanted to have that in different docker containers, but we didn't get to that, maybe in the future.

Having the ability to relocate services is realy one of the main feature of fabric3, we are counting on. You can't realy plan all the services ahead, so you can't tell if they will be remote or local - as I wrote, some of them are pretty obvious, but some are not. Well at least not in agile project, where you can't realy see the whole picture at the beginning. In startup you usually start building a car and ending up with submarine, but that's how it should be, if customer at the end pay for a submarine. The possibily to simply change service from local to remote and vice versa is therefor realy usefull. We developed some best practices on building sca services, so we can use that feature to it's full potential. e.g. don't use whiteboard pattern, as it can't span local space, make sure every input/output parameters of services are serializable, ...

ZeroMQ binding is presented as the main remote sca-binding. To be honest, it's the first time we realy take it for a serious test drive. And I must say, it's far from being mature even with services with 1 input/output parameter and without exceptions, if there is a problem with it, wire starts behave unpredictable. I don't remember that before zeroMQ there were problems with remote bindings. I can't remember, but we used maybe jms? In that matter, I think the default setting for fabric3 should be for the environments without multicasts. Multicasts are hard to manage, and can cause a mess in infrastructure if not routed properly. They are rarely supported in cloud environments, and in private datacenters, it's many times considered as present and working, but it realy depends on admins, if they can handle them - and in many situations they cause more problems than gain...

I think we discussed that in the past, but there is a cluster layer, which interconnects all the nodes. Why not leverage that for remote bindings? I think fabric3 don't need to have profiles like zeromq or jms present, to be able to call remote services - standalone package should suffice, it sure have. I'm not speaking about jgroups, but you mentioned hazelcast...

what do you think?

Tomas


Dňa 08.09.2015 o 23:35 Jim Marino napísal(a):

Hi Tomas,

This touches on modeling local and remote services. For simplicity, I think they need to be modeled differently. Otherwise, an application will miss important cases (treating everything as local and ignoring network latency and failures) or become overly complex (treating everything as remote).

 

What I have found effective is to layer systems such that remote calls are grouped at the system boundaries, if possible. If performance is not paramount, JSON over a messaging layer or REST works nice. If performance is critical, then I usually handle de/serialization directly with byte[] since that is a typical bottleneck. It's a bit ugly but the byte[] passing is only done in a few places and the arrays are either converted directly into POJOs (e.g. protobufs) or fronted by a flyweight which points back to the original bytes (or off-heap buffer). That doesn't pollute the codebase since most of the services are passed complex Java types as opposed to bytes.

I wanted to focus in particular on something you said, which is that one of the best parts of SCA is the ability to collocate or remotely host services and have the runtime pick a binding using the "SCA binding".

It is really valuable to have multiple options for deployment topologies, e.g. for testing everything can be run locally and in production services can be distributed by simply changing a configuration setting. Is this what you see as the main value in the "SCA binding" - the ability to switch how things are deployed?

I have been thinking about this lately and I am beginning to believe the "SCA binding" where a remote transport is automatically selected (not SCA itself ;-) ) is not the best approach to solving this problem. What is leading me down this path is that nearly every time I have used automatic binding selection I also wanted to tune communication parameters. What I think I want is the ability to selectively apply bindings. For example, in testing use use local wires but in production use ZeroMQ configured in a certain way.

Is this what you are trying to do or am I missing your use case?

Jim