Using FlatBuffers as a JSON encoder/decoder for network server

[Jun Zhang]

Hi

I have a C++ network server that uses JSON to communicate with clients. As I find flatbuffers, I am thinking of taking the best of both world, i.e., 

1. the plain text and interoperability of JSON

2. the speed and strongly-type of flatbuffers

I read the document and find that FlatBuffers can handle JSON. However, it has to read the .fbs files first.

In my environment where the binary has to be distributed for use, that might mean that the .fbs files should be distributed as well, which seems a bit troublesome.

Or I can use "xxd -i " the .fbs files and embed them into the source code.

Just like to know are there any other people thinking of similar thing? What are your solutions?

Thanks a lot

Ralph

[Wouter van Oortmerssen]

If you embed them like that in your code, the schema will still appear in plain text in your executable. I'd at least transform the source first into something unreadable using say a simple xor, that can trivially be undone inside your program.

We're of course talking obfuscation here, since if someone really wants to reverse engineer your network traffic, not having the schema is only a minor speedbump.

We are actually also working on binary schemas, but it will take a while for that to be available with JSON.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sushil Rangarajan]

I have a related question here. 

I have a similar use case where the client communicates with the server in JSON. I would like to use FlatBuffers to store the data in binary and deserialize it when I populate the views of my application. How can I leverage both? I'm doing this for a simple android app. 

Steps:

1. Define a schema (using the fields of the JSON response). Do I need to have a sample response loaded on disk? Can the fields of the response change? What if new fields get added to the response?

2. Convert to a binary file using flatc. 

3. When data comes back from server, I parse the response (in JSON) to model objects. 

4. I create the FBB object. Assume that Model is top level class I have deserialized the JSON to.

     FlatBufferBuilder fbb = new FlatBufferBuilder(size);

     fbb.createString(response);

     int model = Model.endModel(fbb);

     fbb.finish(repo);

     

     // Persist

     byte[] result = fbb.sizedByteArray();

     // Read from storage

     ByteBuffer byteBuffer = ByteBuffer.wrap(result);

     Model model = Model.getRootAsRepo(byteBuffer);

     

5. I can now update the views with the Model object

Does this look right? I apologize if this was answered in an earlier post. Can you answer the questions/review inline? 

[Wouter van Oortmerssen]

Sushil:

I have a related question here. 

I have a similar use case where the client communicates with the server in JSON. I would like to use FlatBuffers to store the data in binary and deserialize it when I populate the views of my application. How can I leverage both? I'm doing this for a simple android app. 

If you want to have full Benefits of FlatBuffers, you should have your server generate a binary FlatBuffer, and send that over the wire instead. Sending JSON over the wire is less efficient.

Steps:

1. Define a schema (using the fields of the JSON response). Do I need to have a sample response loaded on disk? Can the fields of the response change? What if new fields get added to the response?

You don't need a sample response.

Yes, fields can be added for example. Read the documentation on schemas on how you can change them in a forwards & backwards compatible way.

 

2. Convert to a binary file using flatc. 

On the server? This does not match up with the rest of your description.

 

3. When data comes back from server, I parse the response (in JSON) to model objects. 

JSON parsing is only available in C++ currently.

 

4. I create the FBB object. Assume that Model is top level class I have deserialized the JSON to.

     FlatBufferBuilder fbb = new FlatBufferBuilder(size);

     fbb.createString(response);

     int model = Model.endModel(fbb);

     fbb.finish(repo);

This just stores a single string in a FlatBuffer, and does no parsing. There's no benefit to doing this.

 

     // Persist

     byte[] result = fbb.sizedByteArray();

     // Read from storage

     ByteBuffer byteBuffer = ByteBuffer.wrap(result);

     Model model = Model.getRootAsRepo(byteBuffer);

     

5. I can now update the views with the Model object

Does this look right? I apologize if this was answered in an earlier post. Can you answer the questions/review inline? 

As I said, what you'd want to do is make the server generate a FlatBuffer binary directly, and no JSON. this is then easy to send, store, and render views from once it arrives in your client. 

[Ged Wed]

thanks for the explanation 

i am wondering if JSON parsing using golang is a priority for this project ?

I use flatbuffers at the moment in golang, but in certain places need to interoperate with JSON.

so for me at least it would be really useful.

[Wouter van Oortmerssen]

Things are a priority if someone wants to work on it :)

Porting all of the JSON parsing code (which is shared with the schema parsing code and the .proto parsing code) to Go seems like a lot of work. Since this code has a very simple interface (json string -> FlatBuffer buffer), it would probably be easier to compile the parser into a library that Go is able to link.

--

[mikkelfj]

Alternatively it should not that difficult to generate a JSON parser by reading a binary schema file and writing out a json parser with actions specific to the schema. It is something I consider doing for C at some point - because there is no reason to tie it in with a much larger compiler project.

[Wouter van Oortmerssen]

Not sure if generating a JSON parser would be that much shorter than a JSON parser that uses a binary schema generically. Also better if you want to be able to read many different kinds of files.