GRFC L4 Add JRuby Support

[Jason Lunn]

There is a GRFC to add support for the JRuby interpreter, which runs on the JVM.

Please provide all feedback on this thread.

[apo...@google.com]

As has been noted before, I think this would be possible by doing a pure implementation, wrapping the java client, or with a JNI wrapper over the C-core (these implementations probably end up relatively different from C-wrapping grpc-ruby).

Actually one thing hasn't been noted yet: If only a client-side library is needed, then perhaps the jruby-platform target could only need to emulate the client-half of the c-wrapping grpc-ruby (would probably simplify things).

As for where the project could live though, one option is it could go into the same grpc repo as current grpc-ruby. Another is it could be done entirely as a third party project in its own repo, for which we could give some guidance.

[Jason Lunn]

It has probably been a decade since I had a reason to try to use JNI. Does anyone have any experience building Gems using that approach? Would it be the case that there would be one JAR per supported platform (x64-mingw32, x86_64-linux, universal-darwin, x86-mingw32), or would it be a fat jar that include multiple JNI wrappers?

I am not clear on what capabilities of the existing gem are needed to satisfy the dependency of the Google Cloud gem (which is my personal motivation to see this move forward). Maybe Mike Moore knows better if the client side would be enough?

I think I prefer to see the JRuby support sit alongside the existing Ruby code in the same repository, and tested in a consistent way. This will help avoid breaking behavior from slipping into the C implementation that isn't observed until someone thinks to update the other repo. It should also make it less likely that a gem update is released for the C derived platforms without a corresponding update for the JRuby variant.

[Mike Moore]

On Thu, Feb 16, 2017 at 9:08 AM, Jason Lunn <jason...@gmail.com> wrote:

It has probably been a decade since I had a reason to try to use JNI. Does anyone have any experience building Gems using that approach? Would it be the case that there would be one JAR per supported platform (x64-mingw32, x86_64-linux, universal-darwin, x86-mingw32), or would it be a fat jar that include multiple JNI wrappers?

I am not clear on what capabilities of the existing gem are needed to satisfy the dependency of the Google Cloud gem (which is my personal motivation to see this move forward). Maybe Mike Moore knows better if the client side would be enough?

The Google Cloud gems are clients only, and so supporting the client-half of GRPC would be sufficient for our use of GRPC.

[kail...@google.com]

Hi Jason,

Thanks for sending this proposal, I took a look and had some comments:

1) As per the proposal, I think that the recommended approach is indeed go down the path of a java library wrapper. JNI wrapping is another possibility, but we should first investigate whether the Java wrapper is not possible before considering it.

2) IMO the jruby implementation can live in a separate repository and I would recommend this to start with. In the future, when the wrapped implementation is stable and passes interop tests we can consider merging it into the main repo. I would recommend starting this work outside of the main gRPC repo. The grpc-ecosystem org is a good candidate. This will allow for maximum flexibility.

3) It would be useful to define some shallow tests that cover the API. Such tests should be added to the existing Ruby implementation first, and can then be used to ensure that the jruby implementation conforms. See https://github.com/grpc/grpc/blob/master/src/python/grpcio_tests/tests/unit/_api_test.py for a starting point. I must reiterate that this type of testing is very shallow and is not sufficient in itself, but a good addition to the interop tests.

4) For any gRPC implementation, we need to ensure that the interop tests pass. See https://github.com/grpc/grpc-java/tree/master/interop-testing, https://github.com/grpc/grpc/tree/master/src/ruby/pb/test and https://github.com/grpc/grpc/blob/master/doc/interop-test-descriptions.md. The jruby implementation needs to pass these in order to be considered conforming. We recommend re-using the ruby interop test suite with the jruby wrapper to the extent possible, independent of the java suite (of course, these can be leveraged as needed). The nice thing is that if the APIs fully conform, the ruby interop tests should just pass.

5) Given the difference between Java and C stacks, there may be API incompatibilities that might not be surmountable, especially in situations where the C core implementation specifics is directly exposed upward. Such situations need to be quantified and further down the line, we can decide how to tackle them and weigh the cost of doing just that vs a full JNI implementation.

Some minor comments on the proposal doc itself:

1) Please add details on implementation ownership, so that it can be discussed. Its unclear if this is something that the proposer is signing up to implement, is that the plan?

2) Could you please update the proposal to cover the implementation approach in more detail based on some of these points. Ie: Location of the prototype repo, some discussion on the API surfaces that need to be conformed to and potential areas you have identified as problematic. This can go hand in hand with a prototype of the java wrapped approach so that we can tackle issues as they happen.

Thanks!

Kailash

[Nicolas Noble]

Personally, I wouldn't see Kailash's last comment as a minor comment as he says; this is a major problem with that gRFC: deciding between JNI or Java, and outlining the details of how to actually implement an API that would be compatible with the current one is a design decision that needs to be investigated, and fully described by the gRFC.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/042830fb-0dd9-4ddb-b982-28a24ca23720%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nicolas Noble]

Ping - any updates on our concerns? 

[Kailash Sethuraman]

Hi Jason,

Are you still keen on moving this forward? 

Thanks

[Jason Lunn]

1) I agree that a Java wrapper is preferred, and I'd contemplate a JNI approach only the wrapper approach hit some unsurmountable roadblock

2) A separate repository requires overhead and introduces opportunities for breakage. As I said before, a single repo approach helps avoid breaking behavior from slipping into the C implementation. It should also make it less likely that a gem update is released for the C derived platforms without a corresponding update for the JRuby variant.

3-5) Per Mike's feedback about the Google Cloud gem's requirements, I'd want to get all the existing tests to pass that are needed to make a viable client, and I'd be willing to settle for raising an exception that the operation is not supported on the rest of the API as a starting point. The eventual goal would be to get complete API compatibility, but a client-only approach would unblock the use of the Google Cloud by JRuby applications the fastest, and that is my core motivation.

I'm willing to contribute a proof of concept PR into the existing GRPC repository. I haven't started such a branch yet, as I didn't want to start such an effort before I got consensus that such a contribution would be welcome in principle, so I have yet to look at the API surfaces in general.

Lastly, I'm at a conference this week, so please accept my apologies if I am not responsive until next week.

 - Jason

[Nicolas Noble]

Ping ? Do we have any update on the status of https://github.com/grpc/proposal/pull/13 and the JRuby effort in general ?

[Jason Lunn]

TL;dr: 123 of 145 unit tests pass; need help to move forward

The proposal PR has been updated with notes detailing the current work in progress and a link to a proof-of-concept branch. A project member with experience with both the Java and Ruby implementations that could help describe how to canonically invoke the right combination of Java APIs to meet the expectations of the Ruby library would be greatly appreciated.

 - Jason

[kail...@google.com]

Hi Jason,

 

Thanks for working on this! Apologies for the delay in getting back to you. We took a closer look at the prototype and at the moment it looks like most of the work has been around the jbuilder integration and stubbing out of C core implementation to get most of the tests passing (do let me know if I missed something!). The bulk of work will be in mapping the ruby api on top of the java api. We can help consult on areas that are not straightforward once you do this.

 

General recommendations on the approach:

Use Java's advanced API instead of the stub API. That means things in io.grpc instead of io.grpc.stub. Specifically, ManagedChannel and ClientCall. You can refer to io.grpc.stub.ClientCalls and the generated code for inspiration/hints. Getting a working MethodDescriptor may be a bit annoying, but after that point most things hopefully start coming together since you could do Channel.newCall() which is very fundamental.

 

As far as the shims go, it may be doable but some items may be infeasible. At that point, it would be good to discuss the specific points on this thread and we can definitely help figure out.  Some areas such as Call/Channel credentials APIs might be candidates for that. There are a number of APIs for ruby channel objects that are specific to the C-core (see options passable to channel constructors in https://github.com/grpc/grpc/blob/c4f46018096a85f239ace283563ad5c192a6c433/include/grpc/impl/codegen/grpc_types.h#L152), along with call credentials, channel credentials, etc. Possibly not everything needs to be emulated, but a minimum of this surface would need to be designed for us to review by the proposer of the gRFC (you) and later implemented.

 

As a lower priority, consider decoupling this implementation from grpc/grpc, it would be nice to land this in grpc-ecosystem eventually.

 

 

Thanks

 

[apo...@google.com]

Also, one more important note on tests:

While the grpc/ruby unit tests are helpful for API conformance, a minimal implementation would still need to be passing with the grpc-wide interop test suite.

(see script in https://github.com/grpc/grpc/blob/master/tools/run_tests/run_interop_tests.py, fyi these can be ran locally with "run_interop_tests.py -l ruby --use_docker")

On Wednesday, February 15, 2017 at 8:59:31 AM UTC-8, Jason Lunn wrote:

[Jason Lunn]

Thanks, I'll look at io.grpc when I get a chance.

[Jason Lunn]

Understood.