guidance for extending GenerateCPP

[agallego]

I'm trying to extend the CPP generator for the "rpc_service" parser. 

So i browsed the source and found that this is defined in 

idl_gen_cpp.cpp

class CppGenerator : public BaseGenerator {}

*however* 

the parser for service is : 

CheckedError Parser::ParseService() {

So the tl;dr is that you call Parser.Parse()

and then you gen code from here? 

  SymbolTable<ServiceDef> services_;

A few tricky things I envision (still testing) - rpc tags and file names. 

First: if the user outputs things to a file name for the types how do I know to include the name? command line? 

flatc --cpp ... idl.fbs  usually generates idl_generated.h 

so how would my parser know this name? Seems impossible without passing command line options - **however,** seems possible if I just default to the <idl>_generated.h  name 

so the question is how to find out the name of the file. 

Second: I noticed that you can embed tags on the rpc schema i.e.: `idempotent` and `stream`

if I was to pass any other tags for the rpc_service methods , how can i read it. 

I'm interested because I want to tag things with either metrics or not at code generation time. i.e.: 

rpc_servie Foo {

  Method(Bar):Bazz (tag1, tag2, tag3)

}

Where these tags aren't pre defined by the Parser

so maybe there are vectors:

  std::map<std::string, bool> known_attributes_;

and   

std::map<std::string, bool> unknown_attributes_; //  how do you suggest i get a list of unknown tags for the services? 

Anyways, thank you in advance for any tips 

[Wouter van Oortmerssen]

What exactly are you trying to do?

I have a commit under way that is meant to add GRPC code generation support, once that is in you should have that as an example to work from.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Gallego]

I want to add code generation support for seastar. I have a partch which integrates with it. So just want to code gen the stubs now. 

Is the pr on a branch i can check out? 

Sent from mobile, please forgive my handwriting.

[Wouter van Oortmerssen]

This is all internal at the moment, going to make an effort to push it asap. If not I may put it on a branch for you.

[Alexander Gallego]

Awesome. Thanks. 

Sent from mobile, please forgive my handwriting.

[Gonzalo Serrano]

Hi there,

I'm also looking for the way to generate the rpc_service.

I've seen that the idl_parser already handles it and there is a rpc_service in the monster example https://github.com/google/flatbuffers/blob/57ba8a4d46169439d68fec000271fb0059278d44/tests/monster_test.fbs#L72-L75 but i don't think there is any codegen yet.

Did you put that branch you talked about somewhere?

Thanks!

[Wouter van Oortmerssen]

The commit that brings initial GRPC support is in internal code review, and could be made public as early as today. That will be a good one to look at for any RPC you may want to do.

[Wouter van Oortmerssen]

Ok, first commit with GRPC support has landed in FlatBuffers: https://github.com/google/flatbuffers/commit/48f37f9e0a04f2b60046dda7fef20a8b0ebc1a70

I wouldn't be surprised if this is still rough around the edges, and it certainly needs some improvements still (like making the GRPC path zero-copy).

I'd be grateful for anyone having a look and giving feedback, to see if anything else is needed for people to start using this.

[Alexander Gallego]

Awesome, will dig through tonight! thanks

Sincerely,
Alexander Gallego

---*---
------*
*  *  *

[agallego]

Hi Wouter, 

Thanks for posting this diff. I'm on midst of generating my own services thanks to this diff. 

GenerateGRPC()

Is really clean and easy to read! 

Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.

[Gonzalo Serrano]

Hi guys,

I'm trying to integrate gRPC + flatbuffers in Go.

So far the somewhat manual approach i've tried (and it has worked) is the following:

- create the example.proto file with a rpc_service, request and response messages etc

- generate the grpc go code: $ protoc example.proto --go_out=plugins=grpc:.

- generate the example.fb file from the example.proto one with $ flatc --proto example.proto

- generate the fb go code: $ flatc --go example.fb

- remove the go types from the proto generation but leave the service code, so it will use ones found in the flatbuffers (i'm using the same go package)

Then, from the grpc side i've created a flatbuffers codec that implements the grpc.Codec interface found in the Go implementation (https://github.com/grpc/grpc-go/blob/a0ff1e78a98cd140c2d18a0210ab0a2a183a2b4c/rpc_util.go#L54-L62).

Something like this:

// FlatbuffersCodec is a Codec implementation with flatbuffers.
type FlatbuffersCodec struct {
}

func (FlatbuffersCodec) Marshal(v interface{}) ([]byte, error) {
    fb := v.(flatbuffers.Message)
    return fb.Table().Bytes, nil
}

func (FlatbuffersCodec) Unmarshal(data []byte, v interface{}) error {
    fb := v.(flatbuffers.Message)
    flatbuffers.GetRootAs(data, 0, fb)
    fb.SetTable(fb.Table())
    return nil
}

func (FlatbuffersCodec) String() string {
    return "flatbuffers"
}

For this code to work, i've done some things:

- created in the flatbuffers project a Message interface:

type Message interface {
    Table() Table
    SetTable(Table)
    Init(buf []byte, i UOffsetT)
}

- also i've modified the flabuffers idl_gen_go.cpp to generate those 2 extra functions Table() and SetTable(), that along with Init(), will make the generated fbs implement the Message interface:

// Implement the table accessors
static void GenTableAccessors(const StructDef &struct_def,
                               std::string *code_ptr) {
  std::string &code = *code_ptr;


  GenReceiver(struct_def, code_ptr);
  code += " Table() flatbuffers.Table ";
  code += "{\n";
  code += "\treturn rcv._tab\n";
  code += "}\n\n";


  GenReceiver(struct_def, code_ptr);
  code += " SetTable(table flatbuffers.Table) ";
  code += "{\n";
  code += "\trcv._tab = table\n";
  code += "}\n\n";
}

- finally i've implemented this helper function to create the fb from the codec unmarshalling:

func GetRootAs(buf []byte, offset UOffsetT, v Message) {
     n := GetUOffsetT(buf[offset:])
     v.Init(buf, n+offset)
}

Then, to make the client and server code work, i just need to specify the codec option:

- client: 

opts = append(opts, grpc.WithCodec(grpc.FlatbuffersCodec{}))

- server:

opts = append(opts, grpc.CustomCodec(grpc.FlatbuffersCodec{}))

So, what do you guys think of these abstractions? Any feedback is appreciated.

Regards,

Gonzalo Serrano

[Wouter van Oortmerssen]

Alexander: Glad to hear! Let us know if you find any problems.

[Wouter van Oortmerssen]

Gonzalo: that seems a problematic path to me. Proto -> fbs is only an approximate conversion, and having to modify generated code is not workable.

The better way to support Go (or any other language) with GRPC would be to do what the C++ implementation did, which is two steps:

1. Refactor the GRPC code generator for Go to not depend on protobuf analogous to what I did for C++: https://github.com/grpc/grpc/pull/6130
2. Write a Go code generator similar to: https://github.com/google/flatbuffers/commit/48f37f9e0a04f2b60046dda7fef20a8b0ebc1a70

[agallego]

Thanks, I finally have a fully working -  compiling, running, etc. 

code generator that uses seastar as the serialization format and seastar (http://seastar-project.org) for the networking part. 

Now i have 0 cost serialization at the server + 0 copy from nic to application through intel's dpdk. 

Thanks for the PR - pretty helpful. 

Here it is, in case anyone wants to use it! https://github.com/senior7515/smurf/tree/master/src/rpc

[Wouter van Oortmerssen]

Can you explain how it interacts with Seastar and FlatBuffers?

It's not obvious to me from that code how you would use FlatBuffers with it.

[agallego]

Sure. 

Effectively, I have a header and payload format.  Header is 12 bytes, so I can read that first and then read the payload, nothing fancy here. 

The payload itself is a flatbuffers that wraps other flatbuffers in general, but right now it's opaque. That is to say, just a 

body: [uint8]

here is how you would use it with a flatbuffers::FlatbuffersBuilder: 

// 1. make the flatbuffers builder
	flatbuffers::FlatBufferBuilder builder;
	// 2. create the payload
	// ignore the return type for now
	auto req = smf_gen::fbs::rpc::CreateRequest(
	builder, builder.CreateString("hello from client"));
	builder.Finish(req);
	smf::LOG_INFO("About to send rpc request to server of size: {}",
	builder.GetSize());
	smf::rpc_envelope e(builder.GetBufferPointer(), builder.GetSize());
	client.GetSend(std::move(e))
	.then([](auto reply) {
	if(reply) {
	smf::LOG_INFO("Got reply from server with status: ",
	reply.ctx->status());
	smf::LOG_INFO("Got reply from server with response name: ",
	*reply.get()->mutable_name()->c_str());
	} else {
	smf::LOG_INFO("Well.... the server had an OoOps!");
	}

For more info, here is the server: 

https://github.com/senior7515/smurf/blob/feature/typesfae/src/rpc/templates/server.cc

and here is the client code: 

https://github.com/senior7515/smurf/blob/feature/typesfae/src/rpc/templates/client.cc

Here is the flatbuffers schema definition that wraps my RPC.

https://github.com/senior7515/smurf/blob/feature/typesfae/src/flatbuffers/rpc.fbs

The TL;DR: is that seastar uses dpdk to effectively write the network packets to the application memory. From there since I know that the payload 

are formatted in a flatbuffers, i can simply do a pointer cast to the type. 

So this way, I get 0 cost serialization (or very little) on the server and as close to 0 copy from NIC to my application code.

I'm still working through some zero copy outgoing buffer integration w/ seastar, but it's mostly there. 

 

Thoughts would be hugely appreciated, specially on the flatbuffers schema for defining an RPC. 

Here is what the full code generated looks like: 

https://github.com/senior7515/smurf/blob/feature/typesfae/src/rpc/smf_gen/demo_service.smf.fb.h

Thanks. 

[Gonzalo Serrano]

Gonzalo: that seems a problematic path to me. Proto -> fbs is only an approximate conversion, and having to modify generated code is not workable.

Totally agree with you. I just wanted to have a quick proof of concept to be sure that using flatbuffers can be possible.

 

The better way to support Go (or any other language) with GRPC would be to do what the C++ implementation did, which is two steps:

1. Refactor the GRPC code generator for Go to not depend on protobuf analogous to what I did for C++: https://github.com/grpc/grpc/pull/6130
2. Write a Go code generator similar to: https://github.com/google/flatbuffers/commit/48f37f9e0a04f2b60046dda7fef20a8b0ebc1a70

Agree, I'll try to do something like that to make everything automatically generated.

Nevertheless I still think that it will require a for sure a proto.Codec implementation, which implies:

- a Message interface for flatbuffers

- the methods to expose the _tab data, which will end beeing part of that interface

Do you think these abstractions and the implementation i provided make sense?

Thanks.

[Wouter van Oortmerssen]

Thanks, that looks pretty cool.

As for the schema, I think more of the definition of this should be left to FlatBuffers itself.

With `dynamic_headers: [DynamicHeader];` you are emulating a HTTP style header. But FlatBuffers itself already has optional fields and such, so those are best of being defined directly in a schema. Besides, strings for keys and values take up a LOT of space, whereas a scalar FlatBuffer field can be tiny.

Also don't see the point of defining Header, since apparently it is outside the FlatBuffer data.

[Wouter van Oortmerssen]

Again, please look at the FlatBuffers <-> GRPC integration in C++, it works entirely without any use of Protobuf or its classes.

--

You received this message because you are subscribed to the Google Groups "FlatBuffers" group.

[Alexander Gallego]

Sincerely,
Alexander Gallego

---*---
------*
*  *  *

On Mon, Jul 11, 2016 at 1:27 PM, Wouter van Oortmerssen <w...@google.com> wrote:

Thanks, that looks pretty cool.

As for the schema, I think more of the definition of this should be left to FlatBuffers itself.

With `dynamic_headers: [DynamicHeader];` you are emulating a HTTP style header. But FlatBuffers itself already has optional fields and such, so those are best of being defined directly in a schema. Besides, strings for keys and values take up a LOT of space, whereas a scalar FlatBuffer field can be tiny.

Agreed but wouldn't you aggree this this in the future allows me to ship different serializations formats outside of flatbuffers? - right now not the case and in fact DynamicHeaders is not yet in used by my application. 

Love the feedback tho :) 

 

Also don't see the point of defining Header, since apparently it is outside the FlatBuffer data.

So as it stands there are 2 flatbuffes. 

1) Is for the RPC itself - Header + Payload. 

2) The actual data being shipped - however, this can change to something else. 

[Alexander Gallego]

Sincerely,
Alexander Gallego

---*---
------*
*  *  *

On Mon, Jul 11, 2016 at 1:28 PM, 'Wouter van Oortmerssen' via FlatBuffers <flatb...@googlegroups.com> wrote:

Again, please look at the FlatBuffers <-> GRPC integration in C++, it works entirely without any use of Protobuf or its classes.

Awesome, will look at these classes, maybe i missed something. 

Would you mind pointing me at an example. Not sure I know what you mean, will investigate nevertheless. 

Thanks!!!!

 

--
You received this message because you are subscribed to a topic in the Google Groups "FlatBuffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/flatbuffers/sxuOVG0nnJ4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to flatbuffers...@googlegroups.com.

[Wouter van Oortmerssen]

Well, your DynamicHeaders are FlatBuffer tables, so that doesn't sound too easy to switch to another serialization format either.

Ideally, your header (just size, maybe crc if you want) is something every payload has, but then the payload itself is just a binary blob, and its definition should be entirely under the control of the specific use case, so can use FlatBuffers or anything else.