Java/Scala Cap'n Proto implementation coming

[Kenton Varda]

Hi all,

Alek Storm, who is working with me on an upcoming project that will be using Cap'n Proto, is going to be writing a Java implementation with additional Scala bindings.

Moses, I know you expressed interest in doing a Scala implementation as well, but I don't know how much you've worked on this.  I don't want to discourage you from contributing, but practically speaking Alek is actually physically working with me (full-time), and is doing this work in order to support a project we're working on together, so it's probably easier for us to coordinate.  What are your thoughts?  Would you like to be involved in reviewing designs?  Work with Alek on the code?  Or just wait and use it when it's ready?  :)

-Kenton

[Nakamura]

I'll try to contribute a little, but obviously I won't be able to work as much as Alek.  I'll ping Alek when I have some free time to figure out something useful I can work on.

[Jorge Ortiz]

I've spent the last few months writing a custom Thrift code generator for Scala, used internally at Foursquare. I would love to help out with design or code reviews for the Scala/Java implementation.

[Thierry Bourrillon]

Hi All,

any news on the scala/java implementation ?

Best Regards,

Thierry

[Kenton Varda]

Hi Thierry,

Unfortunately, I'm pretty sure that Alek is no longer working on this.  On the other hand, Charles Strahan has talked about possibly doing a Java implementation, though he's currently focusing on Ruby.  Are you interested in taking a shot at it?  :)

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

Visit this group at http://groups.google.com/group/capnproto.

[Charles Strahan]

Thierry,

As Kenton mentioned, Ruby will probably be my focus for the next week or so. I was thinking about hacking on Java/JNI bindings next (I'll need that for proper JRuby support), but you can take that on if you'd like (I'd be glad to help out in any I can) :).

-Charles

[kkli...@gmail.com]

Let me start with a big caveat: I AM NOT VOLUNTEERING.

That said, what would the road map look like for doing a java only port of Cap'n Proto?  Not having a native dependency would be huge for many of the use cases I see, and if there isn't a huge amount of effort already dumped into a JNI version, all the better.  Am I totally off base here?  

Kasey

[Kenton Varda]

On Wed, Sep 4, 2013 at 7:37 PM, <kkli...@gmail.com> wrote:

That said, what would the road map look like for doing a java only port of Cap'n Proto?  Not having a native dependency would be huge for many of the use cases I see, and if there isn't a huge amount of effort already dumped into a JNI version, all the better.  Am I totally off base here?

Clearly we will want a pure-Java implementation at some point.

The benefit of the JNI implementation is that it could get done really quickly, because all the complicated logic is already implemented in C++.  Consider that the current C++ implementation is about 10x the lines of code of the Python implementation.  A pure-Java implementation would have to reproduce much of the C++ logic, whereas a JNI implementation would be more like the Python implementation.  Moreover, the JNI implementation would probably be faster.

But, yes, clearly many use cases would prefer to avoid any native code dependency.  A pure-Java implementation would probably be designed similarly to the C++ one, with Reader/Builder classes that wrap a (ByteBuffer, offset) pair containing the actual message content.  This way the wrappers for sub-objects can be instantiated on-the-fly when the getters are called, and you get all the same no-parse benefits as in C++.

-Kenton

[chad...@gmail.com]

I am contemplating writing a pure Scala impl, but just for fun ass a side project. I'll update when I have something...

[Daniel Harrison]

So go ahead and add me to the Scala Implementation Club. I started writing one shortly before Chad announced his, and it's been interesting enough that I've kept going.

I started by trying to JNI Dynamic{Struct,Value,...} as a quick demo, but hit some roadblocks. Since a real JNI implementation presumably wouldn't be based off the dynamic stuff, I decided to take a step back. So I understand how the C++ one works. And I have a good idea of how I'd do a pure Scala one. But I don't have any JNI experience previous to this, so I don't have a good intuition on how that bridge should be designed. Kenton/Alek, have either of you written up anywhere how you were planning on doing that?

[Kenton Varda]

On Mon, Oct 7, 2013 at 9:07 AM, Daniel Harrison <daniel....@gmail.com> wrote:

I started by trying to JNI Dynamic{Struct,Value,...} as a quick demo, but hit some roadblocks. Since a real JNI implementation presumably wouldn't be based off the dynamic stuff, I decided to take a step back. So I understand how the C++ one works. And I have a good idea of how I'd do a pure Scala one. But I don't have any JNI experience previous to this, so I don't have a good intuition on how that bridge should be designed. Kenton/Alek, have either of you written up anywhere how you were planning on doing that?

For Java, I don't think you'd necessarily wrap the dynamic API, because unlike Python, Java doesn't make method calls by name at runtime (I hope).  Probably what you want to do is implement simple primitive field accessors using sun.misc.Unsafe, without using JNI at all, so that they can be inlined.  Fall back to JNI for more expensive calls, such as allocating space for a message, serializing, etc.  It's unclear whether pointer field accessors should use JNI -- these are too complex to inline anyway, but I don't know how much overhead JNI adds to the call.

In any case, this approach probably means that you'd have to use the internal interfaces in capnp/layout.h, and extend them to give you some way to access the underlying pointers for use by Unsafe.  Since these interfaces are currently internal and subject to change, we'll have to coordinate carefully on this.

-Kenton

[adam.ros...@gmail.com]

Hi,

It seems like all the previous movement on this has died off.

I've written a fair amount of Unsafe / off heap type stuff during my days in prop trading. My day job is no longer in trading but I am still stuck with using Thrift / ProtoBuf or a framework like Kryo which has its own problems in a pure Scala project, none of which I like.

I might spike for a day or two on this to see if I can get a better gauge at what it would take to build a full implementation that is pure Java (but Unsafe backed so not technically portable everywhere). Is the main website with the HowTo for other languages the best resource, or can someone on here help bootstrap me with some additional information so I can take a look at this?

Thanks,

Adam

[Kenton Varda]

Hi Adam,

It would be awesome if you took this on.  Be sure to sync up with Charles who has done some work on JNI-based bindings:

https://groups.google.com/d/topic/capnproto/FnX8kWkoxvs/discussion

Were you thinking of combining Unsafe with JNI bindings to the C++ implementation?  I think this makes sense, since the two are likely to be usable in similar contexts.  Long-term I'd like to see a pure-Java implementation based on ByteBuffer (not Unsafe), which should be usable anywhere, but that will take a lot more work.

The web site is the only written resource I know of (other than the code).  Of course, I'm happy to answer questions directly.  I'm currently travelling overseas but I'll be more responsive when I get back next week.

-Kenton

--

[adam.ros...@gmail.com]

Hi Kenton,

This is my ignorance of the underlying implementation, but I was thinking of an Unsafe only type solution, i.e., anything I could do with a ByteBuffer, I can do with Unsafe. 

While ByteBuffers are officially supported, they carry the overhead of repeated bounds checks for each access, forced memory zeroing during allocation, etc. It is quite possible that when I look a little further I will end up needing a mixed Unsafe / JNI solution, but I would think in that case then any ByteBuffer based solution would need JNI as well. Hopefully will get a chance to take a look this weekend.

Thanks,

Adam

[charlie robert]

As an option, I have my Java project that could be used: http://github.com/pauwau/pauwau.  Please see the wiki and the design directory for a little documentation.

All the traffic and the protocol would need to be changed to Cap’n Proto ByteBuffers.  SessionMessages including PauwauMessages, ObjectRefDescs, and any PassByCopy objects.   Scope internalize/externalize are the hooks called from the streams.

If I could help, I’d like that,

charlie

- charlie

[Kenton Varda]

To clarify, I was suggesting using JNI to wrap the C++ library, to avoid the need to re-implement everything.  You'd then use Unsafe just to implement fast primitive accessors, but things like memory allocation, pointer manipulation, etc. would delegate to the C++ implementation.  I think this would be a good way to get things working quickly.

Of course, if you'd rather do more work in order to implement the whole thing in Java, by all means, do it.  :)

I agree that Unsafe is preferable to ByteBuffer where available, but we will eventually need a ByteBuffer implementation as well.

-Kenton

[adam.ros...@gmail.com]

Hey Charlie and Kenton,

Thanks for the info and link to your project. I just finished up a first cut of another open source project, so I should hopefully have some time later in the week to take a look.

Once I get a feel for the the effort of an Unsafe / JNI solution, I might try to scope out a plan to get that going quickly, then see what pure Unsafe and ByteBuffer solutions would look like.

Thanks,

Adam