proposal: using Flatbuffers as the binding protocol in GoMobile

[Daniele Baroncelli]

The type system in GoMobile is still very limited, as it's currently not possibly to work with slices of struct:
https://godoc.org/golang.org/x/mobile/cmd/gobind#hdr-Type_restrictions

In the meantime, developers are starting to release mobile apps with GoMobile and they all have to deal with the same problem: the lack of slices of struct. Each developer has been coming with its own solution. Someone tried with wrappers (https://github.com/scisci/go-mobile-collection), but the most convincing solution so far seems to be Flatbuffers. A few developers have already used it successfully.

Flatbuffers have official implementations in Go and Java. And there is already an open source package for Swift: https://github.com/mzaks/FlatBuffersSwift

I have talked to Robert Winslow, one of the guys behind Flatbuffers, if it made sense to implement Flatbuffers for GoMobile and he seems to agree and he is willing to help.

I am opening this thread, so that discussions can start.

[Joshua Foster]

I would recommend considering Capn Proto. The solution is much more robust than Flatbuffers and has a better API.

Joshua

[Matthew Sackman]

On Tue, Jan 26, 2016 at 05:53:29AM -0800, Joshua Foster wrote:
> I would recommend considering Capn Proto. The solution is much more robust
> than Flatbuffers and has a better API.

Interesting. At risk of this rapidly turning into bikeshedding, I've
been using capnproto in Go for the last year. The Go bindings are fine,
but the Javascript ones are currently broken, there seem to be some
question marks about the Java ones etc. The only "official" one, I
believe, is C++. A quick scan of the flatbuffers repo at
https://github.com/google/flatbuffers/ suggests there are far more
"official" language bindings for flatbuffers, and I'm much encouraged by
the appearance of fuzz testing on
https://google.github.io/flatbuffers/flatbuffers_support.html
As I need good multiple language bindings, I'm currently considering
moving from capnproto to flatbuffers, though I'm yet to use flatbuffers.

I'm curious as to why you feel flatbuffers is less robust. The only
comparison blogs I can find seem to be from a few years ago and don't
tend to reflect using these things in anger for substantial projects.

Matthew

[Joshua Foster]

When I tried getting a Flatbuffers version working, the unmarshall would never work. Flatbuffers also seemed to have more of a "create this, set this, create this, set this..." Capn Proto just worked.

Joshua

[Hyang-Ah Hana Kim]

Thanks for bringing up the issue.

If the goal was to exchange data between two languages, I think use of flatbuffers or other equivalent technologies makes sense.

Byte slices are currently supported, so it's possible to build your idea on top of gobind's byte slice support.

But Gobind is more than just data passing; it offers cross-language object references, callbacks, etc.

For example, Gobind allows Java-side to implement a Go interface, and pass the Java object to Go while holding the actual logic in Java. 

Passing a Go struct in Gobind also means that struct's method are also available to the other language. 

Maybe some methods assume the Go object maintain a certain state during its life time.

I don't see how the use of Flatbuffers (or the equivalents) can help achieving these goal yet.

Crawshaw and I have been going back and forth on how to support struct, slice of structs, etc.

One option we considered is to promote the struct type to its pointer type. It seems doable but some may be unhappy about performance.

Another proposal is to serialize the entire struct object and instantiate an object in Java or Objective-C based on the serialized data. It eliminates cross-language reference counting. 

But we are not sure about how to handle struct with non-trivial methods which may involve unexported fields to work. 

Running out of ideas, I am open to others' ideas...

I believe, once this is settled, we can start looking into suitable encoding/decoding technologies wherever necessary.

- Hana

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

__

[Daniele Baroncelli]

Hi Hana,

thanks for taking part in this discussion

in my opinion, we need to take a step back and answer some basic questions:

- what is the benefit of a GoMobile library?
- what would developers use it for?
- which kind of code would it make sense to write in Go and which one in Java/Swift?

In this diagram I try to describe the typical use for GoMobile:
goo.gl/UkVA9b

In general I would say the Go part should be treated as a black box (only dealing with data and logic), and the threading and callbacks should be managed exclusively from the mobile language.

For this reason, I think the Go library functions should just be called syncronously, providing input parameters and expecting just 1 output value (to be compatible with Java functions), without complex cross-language references. I think the binding should be kept as simple as possible.

I would treat the Java/Swift -> Go interface as an API interface. The data is served by the Go library, and the Java/Swift just queries it, without too many interrelations.


Daniele

[Daniele Baroncelli]

Just one clarification:

the "just 1 output value" should be in form of structured data
hence a Java object matching the Flatbuffer schema, autogenerated following by the definition of the struct which is returned by the Go function

Daniele

[sri...@laddoo.net]

As Hana pointed out, I don't know if mandating flatbuffers, etc works with how reference counting is implemented in gomobile currently.

With my current work, the following approach is proving to be useful:

1. Copy everything unless specified. Every object does not need to be reference counted. Both iOS and Android have a defined application lifecycle. Given this, it is possible to turn most calls across the boundary into copies with reference counting being restricted to just an explicit subset of objects.

2. Following from the above, treat the runtime boundary as an bi-directional RPC layer. There are RPC services where the service objects are reference counted, but all method parameters and return values are copied. Service object references cannot be passed in methods. Services can be exposed by Java/ObjC/Swift/Go.

My understanding is that the above approach doesn't really fit with gomobile bind's current approach since it aims to be a proper binding layer and not merely an rpc layer. However, in practice I have found the rpc approach easier to reason about.

Sridhar

[Daniel Skinner]

> what would developers use it for

As Hana mentioned, gobind allows java to implement a go interface. This is something I found useful early on by declaring a Broadcaster interface in go and in java I implemented the Broadcaster.Stub to facilitate use of http://developer.android.com/reference/android/support/v4/content/LocalBroadcastManager.html as by company convention.

It's not clear simply using flatbuffers accomplishes that type of goal. Right now, I'm still interested in the use of flatbuffers but this is because, as Hana mentioned, there's still a lot to work out with gobind.

[j.e....@gmail.com]

I've had quite a bit of experience in this area.

Briefly: for language interop, the easiest thing to do is to use the excellent support
for JSON and msgpack that Go libraries already provides.

For writing to disk, write one of these in parallel alongside your binary data. For transmitting over the wire, use
the builtin JSON or 3rd party libraries (e.g. ugorji/go/codec or tinylib/msgp) to translate from your Go structs on the fly.

More in depth thoughts:

From investigation of flatbuffers last month:

The current state is that the Go bindings for FlatBuffers are buggy and incomplete;
it wasn't possible to produce equivalent binary output to the C++ reference implementation
with the Go bindings.

The design of FlatBuffers is also, to my mind, poorly thought out: it is such that
there are severe limitations on the amount of data you can express;
you are limited to 64KB + 1 longer array. The
documentation is very incomplete; there is no precise spec, rather just a few
paragraphs of vague casual description that do not match hex dumps
of the reference implementation output. As a result the format is currently so
under-documented that I cannot recommend it.

Summary of flatbuffers investigation: I would like to see the Go bindings improve,
but the originators need to put in better documentation of the format
before that is even possible.

disclaimer: I maintain the first version of the Go bindings for Capnproto.

For capnroto, on the plus side:

+ The serialization design/IDL for Capnproto is generally excellent. Very
expressive and well designed type system and solid evolve-able story.

- But since you talk about having arrays of structs, be aware the in capnproto,
you will have no ability to version such structs; such structs are always
stored in-line in the array. Hence the advantages
of using an evolve-able data format are lost. This begs the question
of why go to the trouble in the first place.
 
- Support for languages beyond C++, python, and Go is unlikely to be completed in the next year.

More meta:

For most new projects I've moved to the simple Msgpack
and using either https://github.com/tinylib/msgp  or  https://github.com/ugorji/go,
with the need for easy json interop being forefront in my mind.

Between those two libraries you have alot of speed tuning options (either
fully dynamic (ugorgi), or fully pre-compiled deserialization (tinylib).

The caveats with Msgpack are poor handling of signed vs unsigned
numbers; the lack standard ways to express time.Time and other
language specific types, and the fact that although they tried to
fix the flaws of version1 with version2, nobody really implemented
version2 fully. Also the evolution of data story is not as bullet-proof;
actual interop with other languages may be particularly
painful because of that. That said, for an all-Go project,
using tinylib/msgp in particular makes serialization so fast and
easy (add a "go generate" one-line directive to your file) that it is my
default solution.  You get quite a bit of ability to evolve your
data, as long as you never re-use an old field name and give
it a different type.  It is somewhat more risky, but as long
as you only care about version N to version N+1 evolve-abilty,
you should be fine.

Stepping back, big picture: Protobufs with GoGoProtobuf
certainly has the most mature tooling and support of any of these.
If you don't want to pay bleeding edge "early adopter" taxes,
and you require a binary format, this is your choice.

[Seb Binet]

a naive question: wouldn't CBOR be a valid alternative?
it's JSON but binary and standardized, isn't it?
and there are implementations in various languages.

-s

[Daniele Baroncelli]

Hi Daniel,

This asynchronous usage sounds interesting, but I think most developers would just be happy with the Go library functions working synchronously and returning fully structured data. Not being able to return structured data, it's the biggest limitation for GoMobile at the moment.

If allowing asynchronous usage creates such complex dependencies, which prevents using slices of struct, maybe we should consider a stripped down version of GoMobile that at least can provide a fully operational usage with synchronous calls.

Go-compiled libraries providing fully structured data to Android and iOS application is already a great achievement. This alone which would attract many new developers to GoMobile.

In summary, there could be two alternative binding tools:
1) only allowing synchronous calls, but supporting all types
2) allowing both synchronous and asynchronous calls, but with a limited set of types, until a better solution is found


Daniele

[Daniele Baroncelli]

In summary, there could be two alternative binding tools:
1) only allowing synchronous calls, but supporting all types
2) allowing both synchronous and asynchronous calls, but with a limited set of types, until a better solution is found

otherwise, just keeping one binding tool, but supporting slices of struct only for synchronous calls?

[Daniel Skinner]

I think the point is not that gobind can't move forward with more types due to complexity, but rather a determination for how to move forward hasn't been made.

As it stands, any synchronization format that's supported on both the go and the java side can be used today with a very light shim. If you're willing to compromise your go structs, you can even avoid the shim and provide structured data today.

As for what makes developers happy, one might even argue that an interface that provides a single struct value at a given position with fields currently supported by gobind already makes them happy when implementing long lists, as will be the case when implementing a recycler view adapter that responds well to changes of the underlying data within go.

There's a number of use-cases and serializing large lists of structs is but one of them.

Out of all the android projects I've done, I can't even remember the last time I manually managed a list of items in memory to be displayed (anecdotal, i know) but for maybe the simplest of cases.

I'd sooner like to see a determination be made for moving forward with gobind that is fast/efficient than a shim I could write on my own in a few hours. Sorry if the opinion comes off hard, it's just my stance on the matter.

As an example, and I don't know what the numbers are here, but if go code is grabbing an xml feed and deserializes to go structs and then reserializes to [insert-serialization-format] to have java deserialize that format, why bother with all the step-arounds? Grab the xml and send it verbatim to Java and let it deserialize once. The details of this are sketchy at best because again it comes down to the use-case and what needs to be done, but simply having a serialization format only solves a small part of the overall problem.

[Daniele Baroncelli]

As an example, and I don't know what the numbers are here, but if go code is grabbing an xml feed and deserializes to go structs and then reserializes to [insert-serialization-format] to have java deserialize that format, why bother with all the step-arounds? Grab the xml and send it verbatim to Java and let it deserialize once. The details of this are sketchy at best because again it comes down to the use-case and what needs to be done, but simply having a serialization format only solves a small part of the overall problem.

 

The advantage of a serialization protocol like FlatBuffer is that it makes no allocation. So it's much faster and more memory efficient.

[Daniel Skinner]

this is comparing apples to oranges though, no? This would be comparing an xml deserializer in java to a flatbuffer. The underlying mechanism of transferring the []byte is still going to cause an allocation (unless that's changed recently).

On Wed, Jan 27, 2016 at 1:14 PM Daniele Baroncelli <daniele.b...@gmail.com> wrote:

As an example, and I don't know what the numbers are here, but if go code is grabbing an xml feed and deserializes to go structs and then reserializes to [insert-serialization-format] to have java deserialize that format, why bother with all the step-arounds? Grab the xml and send it verbatim to Java and let it deserialize once. The details of this are sketchy at best because again it comes down to the use-case and what needs to be done, but simply having a serialization format only solves a small part of the overall problem.

 

The advantage of a serialization protocol like FlatBuffer is that it makes no allocation. So it's much faster and more memory efficient.

--

[Jason E. Aten]

If I understand right, no, []byte won't necessarily cause allocation. Note that []byte is effectively a pointer, albeit a fat pointer that knows its bound and is 24 bytes (3 words) on amd64.

So the pointer that is []byte can point to off-heap memory/memory mapped data, and []byte itself can reside itself on the stack and incur no allocation.

[Daniel Skinner]

https://github.com/golang/go/issues/12113 i believe currently it copies the byte slice, also noted in the gobind docs that []byte doesn't support mutation b/c of this.

[Daniel Skinner]

email sent prematurely... and the docs

Byte slice types. Note the current implementation does not

  support data mutation of slices passed in as function arguments.

but is this only for java -> go and not the other way?

[Daniele Baroncelli]

Jason said it right.

Here are some interesting Go serialization benchmarks:
https://github.com/alecthomas/go_serialization_benchmarks/blob/2f5b86e/README.md

As you can see FlatBuffers is the only serialization with zero allocation.


Daniele

[Jason E. Aten]

Well, those benchmarks are misleading. I pointed out the author how to avoid allocations using capnproto, but he declined to follow said advice.

Daniel pointed out that, although Go in general might not copy for a []byte, gomobile is doing copying nonetheless; per https://github.com/golang/go/issues/12113

[Daniel Skinner]

Yeah, it looks like its copying only for lang -> go

reads: https://github.com/golang/mobile/blob/master/bind/seq/buffer.go#L131

writes: https://github.com/golang/mobile/blob/master/bind/seq/buffer.go#L227

[nic...@gmail.com]

Hopefully this isn't too off topic, but for those interested strictly in a cross-platform protocol binding, you should really look at what keybase is using for their client (https://keybase.io/docs/client/client_architecture#rpc-protocol). You basically define your protocol in AVDL syntax (https://github.com/keybase/client/tree/master/protocol/avdl)  and then are able to build it for ios, android, javascript, and go. I look at gomobile as being more than just a protocol binding obviously, but thought this might interest some.

Dan

On Tuesday, January 26, 2016 at 8:29:41 AM UTC-5, Daniele Baroncelli wrote:

[codep...@gmail.com]

Hey, everyone.

Current gobind implementation is not ideal, but I think stripping down to the synchronous call's only or splitting into two separate types of calling would be a bad idea. I can only speak for Android development, tho.

This implementation allows us to implement entire app runtime inside go, which is tremendously useful because android java threading model doesn't scale well. Async operations support is lacking, and because there is no lambda support - mobile developer should be extremely careful or resources start leaking. Several libraries (including the port's from Rx's) attempted to fix it, but so far every attempt has some bad sides.

Right now it's quite easy to pass Java "callbacks" inside go lib, which allows Java to have the minimal amount of "non-UI" libraries, which in turn, allows focusing on drawing things. instead of figuring out how to get them either from source or from go side of an app.  Stripping down to synchronous only would require implementing some kind of poller in the separate thread and a whole lot of new glue code, which is currently handled by gobind. 

Another good thing is that current implementation allows us to implement state inside go. Not many of mobile developers use it, but go allowed me to implement an entire stateful application, with several long-running (I am talking about from minutes to hours) background tasks. Some would say, that android already provides utilities for that, but the ease of communication between tasks and subsequent cancellation is just too good for me to give up.

As for handling calls in two different manners - it would confuse new developers and support glue code inside of gobind would be undoubtedly harder. Instead of having one problem we will have two.

My thoughts on that matter are that if we need additional speed improvements, perhaps something can be done for cross-language calling speed with primitive arguments when the result is a reference to another struct? Most of the time, the end UI requires primitive types (ints, floats and so on) and strings, so it takes some method chaining (like val.GetThings().GetLast().GetVal() for example)to get there. Handling complex structures like lists\arrays require some additional go code but it comes with almost no additional memory costs. Slices can be quite big, and copying their insides for each result will require additional memory, which is bad for some of us.

Dmitry.

среда, 27 января 2016 г., 22:21:46 UTC+3 пользователь Daniel Skinner написал:

[codep...@gmail.com]

Would it require for serialization mechanism on another side to know go memory layout? If no, than I don't really understand how that suppose to work.

среда, 27 января 2016 г., 22:28:36 UTC+3 пользователь Jason E. Aten написал:

[Joe Blue]

Don't know about anyone else but it would enable lots of developers like me to move forward if some resolution of the binding problem be decided.

At the moment i am using byte[] for exchange of data works and so it would be unable for now. A android / ios would then act as a middleman for exchanging data between the golang layer and the UI layer. Because it's a byte array what it truly represents ( prototocol buffers, flat buffers or even json ) is independent.

I personally am more interested in using polymer for the GUI. I am happy to loose performance but get cross platform ease.

On another approach:

Hanna has a a nice gomobile project on within for serving godoc on her report that I found useful. Extending that to ios but allow developers like myself to get useful project work done near term.
I personally would be very productive if the IOS code for the above project was done.
It would give me one clear option that works now, despite low performance.