Proposal for Publish Subscribe

[Alex Barlow]

Hi all,

I'm a big user of protobuf in several roles now and a common theme is recreating the publish and subscribe pattern as well as RPC with protobuffers. In particular our use case at the moment is publish protobuf messages to Google Pub Sub, but have also used/experimented with Kafka/Nats/SQS.

The benefits are of course the speed of parsing, typings and forward compatibility, but one large downside is the ability to accidentally subscribe to the wrong type on a topic or publish the wrong type, at the moment, the consumer of a topic provides the type and we reflect that type when consuming, for example.

This can look like the following (in Go)...

c.On(pubsub.HandlerOptions{

Topic: "name_of_topic",

Name:  "update_order_in_bigquery",

Handler: func(ctx context.Context, o *order.Order, _ *pubsub.Msg) error {

return publishToBQ(o)

},

})

As stated before, if the topic is incorrectly named or the type doesn't match the topic, subscribing fails. Worse so in publishing you can publish one type to a channel of other types. This message will never be processed.

Ideally we'd like to type both the publisher and subscribers and generate the code, much like gRPC and the service definition in protobuf.

I propose something like the following..

message Orders {

  string id = 1;

}

service Orders {

  rpc Get(GetRequest) returns (GetResponse);

  rpc Create(CreateRequest) returns (CreateResponse);

  rpc Update(UpdateRequest) returns (UpdateResponse);

  rpc List(ListRequest) returns (ListResponse);

  rpc Subscribe(SubscribeRequest) returns (stream SubscribeResponse) {};

}

publisher Orders {

  topic Created(Order);

  topic Updated(Order);

}

Any thoughts of stories of how other are handling this would be great!

Happy to help contribute too, especially in Go, JS and Typescript domains

[Alex Van Boxel]

If I understand you correctly you want to generate Go, Js and Typescript code out of your Protobuf DSL. That sounds like a valid approach, but you don't need to extend the proto DSL from that, you can do that through message options.

We use options extensively, although not for generating code. You can create a protoc plugin for your languages that act on the annotations and generate code that is specific for the message.

If you have conventions for your topic, even better you can bake that logic into the plugin. Example having:

message Orders {

  option (your.option.package.pubsub) = ["Created", "Updated"]

  string id = 1;
}

could generate a client with the verbs "Created" and "Updated" accepting the Protobuf Order. I don't see the need to extend the protobuf DSL for this, once you understand options the usages as limitless.

We don't use options for this, but we use it a-lot in our Big Data pipelines.

 _/
_/ Alex Van Boxel

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/9d25655b-0f1e-4e04-81f9-1925486d93e2%40googlegroups.com.

[Alex Barlow]

Yes I thought about this and prototyped it out a little bit last night.

But that’s true of RPC too, it could be done with options, but it has special status as its own top level semantic which is really nice.

Hence my proposal. I am considering what you’ve done with options as our current statedgy going forward for sure, one issue is having to include that definition everywhere but we can manage that.

Thanks for the interest

[Derek Perez]

My opinion: I think the option-based annotations are the better way to go for this sort of definition of behavior. Additionally, I agree RPCs could have been just as easily expressed this way and potentially they would have been if they were introduced today.

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CABatEtQMm6gabX9TT0Wgu1NeL4e1Mqo4eJkbVVpmKO-WDUrxyg%40mail.gmail.com.