Multiple producers writing to same Topic

[Mick Jermsurawong]

Hi,

I would like to have multiple producers - from multiple services - writing to the same kafka topic, given that these messages follow the same schema. 

(I see people warning on stackover-flow not to mix different messages to the same topic. But my situation is that those messages have the same semantics)

The use case is I have a service worker that is responsible for doing a task, and it listens (subscribe) to the topic containing events of how to execute that task.

Now a few services would like to execute this kind of task, and would like to publish message to that same topic.

With one producer, I can probably use `TopicProducer` and get the events from entity `using persistentEntity.eventStream` from the service that wish to have task executed,

and my service worker would subscribe to that. The approach is the same example in online-auction, where bidding service publish events and search subscribes to it

With multiple producer, however, I'm not sure if I can find the api support. Otherwise what I have to do is having a new topic for each producer, and the worker has to explicitly subscribe to each of them? What I would like to avoid is the worker has to maintain all the knowledge of who wishes to execute this task, where in the implementation, it is calling ServiceA.taskTopic, ServiceB.taskTopic, ..., ServiceC.taskTopic,

Also, if I would like to set up ask/reply across service in a pub-sub fashion, in the case where master asks the worker to do long-running task, and the worker notifies master when the task is done.

I'm considering setting up Topic for each side of sender/recipient, and each subscribe to one another's Topic? Is that the right approach?

Best,

Mick

[tim....@lightbend.com]

Hi Mick,

You are correct that Lagom does not currently support this directly.

One option would be to do what you say: have each controlling service publish its own topic, and subscribe to them all in your consumer. You could also have a separate aggregator that subscribes to the topics for each producer and merges them into a single topic that the worker service can subscribe to.

Alternatively, you could use Akka Streams Kafka directly to implement this. This is the underlying low-level library that Lagom uses. http://doc.akka.io/docs/akka-stream-kafka/current/home.html

Cheers,

Tim

[tim....@lightbend.com]

Sorry, I forgot to respond to your second question:

On Monday, January 30, 2017 at 12:21:35 PM UTC+11, Mick Jermsurawong wrote:

Also, if I would like to set up ask/reply across service in a pub-sub fashion, in the case where master asks the worker to do long-running task, and the worker notifies master when the task is done.

I'm considering setting up Topic for each side of sender/recipient, and each subscribe to one another's Topic? Is that the right approach?

Yes, this sounds like a good approach.

Cheers,

Tim

[Tim Pigden]

Tim - is this considered a limitation that is likely to be removed at some point? After a quick glance it looks like dropping down to akka-stream-kafka gets us into the realm of byte arrays and no message typing support. Is that correct?

I can think of lots of use cases where entities might want to stream events onto a common stream - any IoT modelling that needed to aggregate data from multiple similar sensors for example.

Or is there another way of achieving a similar effect using alternative tools from the lagom toolbox ? Having a stream per sensor sounds like a lot of overhead.

[Mick Jermsurawong]

Thank you Tim@lightbend! I decided to implement two topics on both side of sender/receiver (master/worker) service instead, and conceptually it is like a channel of communication for them.

After all, i think my use case is not really a pub-sub, since the master will want to know the result from the worker.

Hi Tim, i think the case for streaming events and aggregating them has a nice example from pub-sub here http://www.lagomframework.com/documentation/1.3.x/scala/PubSub.html

My use-case is more problematic since it is cross-service pubsub

[Tim Moore]

On Wed, Feb 8, 2017 at 9:23 PM, Tim Pigden <tim.p...@optrak.com> wrote:

Tim - is this considered a limitation that is likely to be removed at some point?

It isn't in our near-term road map at the moment, but of course we listen to our users and also encourage community contributions. The best next step would be someone interested in this feature to raise an enhancement request at https://github.com/lagom/lagom/issues and describe their use cases.

 

After a quick glance it looks like dropping down to akka-stream-kafka gets us into the realm of byte arrays and no message typing support. Is that correct?

The underlying Kafka client does support serializers. All of Lagom's Kafka support is built on top of this, so it might be worth exploring the Lagom source code to see how it's wired together.

In particular JavadslRegisterTopicProducers.scala or ScaladslRegisterTopicProducers.scala would be a good place to start..

I can think of lots of use cases where entities might want to stream events onto a common stream - any IoT modelling that needed to aggregate data from multiple similar sensors for example.

Or is there another way of achieving a similar effect using alternative tools from the lagom toolbox ? Having a stream per sensor sounds like a lot of overhead.

I think that typically those wouldn't be streaming directly into Kafka, but into some kind of aggregator service that is exposed to the Internet. In that case, the device → service communication would be using Lagom's streaming service calls, and the implementation of that would forward along to a single Kafka topic to be consumed by other services.

Cheers,

Tim

--

Tim Moore

Senior Engineer, Lagom, Lightbend, Inc.

tim....@lightbend.com
+61 420 981 589
Skype: timothy.m.moore

[Tim Pigden]

Hi Tim

So just to bring my much more specific example into the frame and ask whether you think it's a valid case:

I intend to implement a service with persistent entities representing trucks. Each truck has within that entity the current schedule of what it's got to do. It is receiving events in from an android device  on the truck and serving as the "web backend" for the android app.

Each of these truck persistent entities needs to publish events that are picked up by another service that is responsible for mediating inter-truck concerns (impending queuing at facilities, for example)  - i.e. it needs to know the state of all trucks.

I have variable number of vehicles but could easily number 100 or more.

So my architecture is likely to be split up between different services, because each has a distinct and quite complex set of functionality to provide.

So am I right in assuming that pubsub does not work in this case and I'd want to go for Kafka? And consequently I'm in the territory of asking for an enhancement request and dropping down to akka-kafka to do the work in the meantime (assuming the request ever gets on the development radar)

Thanks

Tim

[Michal Borowiecki]

Further to the discussion about multiple producers to the same topic:

We are designing an interaction between 2 services, where one of them sends requests and receives responses from the other and we would like to implement the integration using kafka topics on both input and output.

However, we don't want the latter to be a subscriber to a topic declared by the former. We want the latter service to define the data model for both requests and replies. I DDD lingo, we want the former context to be in the conformist relation to the latter context.

In this use case it would make sense for the latter service to declare both its input and output topic and the service client in the former service to allow publishing to the other service's input topic and subscribe to its output topic (this second bit is what's possible already).

This way a service could advertise, via its descriptor, what topic to send messages to and what the format of the messages should be for it to consume them.

As a consequence of this, as requested in this email thread, one could have multiple producers to the same topic, as it would make sense for multiple services to chose to be in the conformist relation to a service that describes its input topic data model.

The producer message streams would be available on the service client side to publish to.

What are your thoughts on this?

I think this makes for a good feature request and more empowering than simply the ability for multiple services to publish to the same topic.

If you agree, I'll raise a request on github.

Thanks,

Michal

[Ignasi Marimon-Clos i Sunyol]

Hi Tim,

On Fri, Feb 10, 2017 at 2:57 PM, Tim Pigden <tim.p...@optrak.com> wrote:

So am I right in assuming that pubsub does not work in this case and I'd want to go for Kafka? And consequently I'm in the territory of asking for an enhancement request and dropping down to akka-kafka to do the work in the meantime (assuming the request ever gets on the development radar)

Keeping in mind that PubSub will only forward messages intraservice _and_ messages will only be delivered to subscribers that are active. I suspect PubSub is not what you'd want. It sounds like you have a service (Fleet Service)  consuming events from an upstream (Trucks Service). I am not sure if you must consume all events from upstream or getting a latest event from each truck could be enough, but since you want inter-service communication you can't use PubSub.

Cheers,

--

Ignasi Marimon-Clos

Software Developer @ Lagom

[Tim Moore]

Hi Tim,

That sounds like a perfectly good use case, but if I understand you correctly, I think it is already supported by the existing message broker API.

Assuming that all of your truck entities live in a truck service, you would publish a stream of truck events from that service to Kafka, and subscribe to that topic from your inter-truck service. I'm not sure where the multi-producer scenario comes in.

Just to clarify: many entity instances can absolutely publish to the same topic—that is the intended usage of the existing functionality. The original question was about multiple independent services publishing to the same topic.

Cheers,

Tim

--
You received this message because you are subscribed to the Google Groups "Lagom Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framework+unsubscribe@googlegroups.com.
To post to this group, send email to lagom-framework@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/af576eb2-df96-450e-8b1c-0e20be6bc016%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tim Moore]

Hi Michael,

What you're describing sounds similar to what I was thinking, though I have to admit I haven't thought it through in great detail.

Some of the questions that come to mind:

What are the semantics of the topic that you get from the service client? Is it a at-most-once ("fire-and-forget") stream that I just throw messages into? Or is it something where I would hand it some kind of resumable stream of entity events like what you can build with the existing TopicProducer helper?

It would be nice to see a sketch of what the code might look like with the proposed API, including end-to-end examples from the source of the messages to the processing side. It would give us something more concrete to discuss.

Cheers,

Tim

--

You received this message because you are subscribed to the Google Groups "Lagom Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framework+unsubscribe@googlegroups.com.
To post to this group, send email to lagom-framework@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/4257d5dc-a496-496e-baec-3cda349c2ed3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mick Jermsak Jermsurawong]

Hi Michael,

I would like your input on my solution please. I have a more general problem because i have multiple services wishing to get result from one worker service, instead of 1-1 interaction in your problem. With this general problem, I arrived at the solution of a pair of topic (input/output) to represent each async bidirectional communication between worker and client, instead of multiple producers to the same topic.

Following the idea proposed of "worker" service declaring input/output topic, and multiple clients can publish to that same input topic, the downside i thought is that all the clients will have to filter for their results; one additional client input will cause the rest of clients to do more filtering work. And also perhaps multiple result of one client shouldn't be visible to the other.

Downside of my approach is the explicit declaration of each input/output topic that both worker/client has to maintain; client impl depdends on worker api (client subscribes to output topic of worker api) and worker impl depends on client api (worker subscribes to input topic of client api). 

And perhaps a more concerning downside at implementation-level: because someone has distribute the results to the right client topic, my output topic each has an entity event stream and filtering results relevant to this client output topic. More polling from event store proportional of number of output topics. I suspect there's more efficient implementation, by having one entity event stream and put them to different topics. For now i'm constrained by the topic always starting from entity event stream. I would appreciate any one' suggestion here.

Overall I thought that there's no free lunch of having benefit of request/response of sync communication, but without overhead maintenance if we want to do it in async.

Best,

Mick

On Tue, Feb 14, 2017 at 8:23 AM, Tim Moore <tim....@lightbend.com> wrote:

Hi Michael,

What you're describing sounds similar to what I was thinking, though I have to admit I haven't thought it through in great detail.

Some of the questions that come to mind:

What are the semantics of the topic that you get from the service client? Is it a at-most-once ("fire-and-forget") stream that I just throw messages into? Or is it something where I would hand it some kind of resumable stream of entity events like what you can build with the existing TopicProducer helper?

It would be nice to see a sketch of what the code might look like with the proposed API, including end-to-end examples from the source of the messages to the processing side. It would give us something more concrete to discuss.

Cheers,

Tim

--
You received this message because you are subscribed to a topic in the Google Groups "Lagom Framework Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lagom-framework/DPM-juGKydI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lagom-framework+unsubscribe@googlegroups.com.

To post to this group, send email to lagom-framework@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/CAApsmOSDO5SQTtPZ0yvXH1a2M%2Bj37pynoXrt00gyKpb3h8qkWA%40mail.gmail.com.

[Tim Pigden]

Ah, thanks for the clarification

--

Tim Pigden
Optrak Distribution Software Limited
+44 (0)1992 517100
http://www.linkedin.com/in/timpigden
http://optrak.com
Optrak Distribution Software Ltd is a limited company registered in England and Wales.
Company Registration No. 2327613 Registered Offices: Suite 6,The Maltings, Hoe Lane, Ware, SG12 9LR England 
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Optrak Distribution Software Ltd. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.

[Michal Borowiecki]

Hi Mick,

Even though it's 1-1 for us for the purpose of our PoC, we are indeed thinking what we'd have to do once we have more services and the problem you mentioned regarding filtering the results from the output topic is concerning for us too.

I think we can accept the overhead as long as the number of services that work this way is small. Once it becomes a problem, the first solution I would consider is adding a minimalistic kafka-streams app that fans out the results to multiple topics while doing the filtering in one place. The downside is that we may need to drop down to the reactive-kafka library instead of using the lagom topic.subscribe() functionality. I think this is a problem we'd think through in more detail once (and if) we have enough such services to warrant it.

However, I the above only makes sense to me if I think about the topics as input:commands, output:events. If instead I think about it more how you describe it, as input:requests, output:responses, then it feels more natural to design it with separate topics for replies to each client. If you keep one input topic, you can have the name of the output topic passed in the request messages. But it might make more sense to have separate input topics for each client too, for better isolation of load. This though would best be done using reactive-kafka or kafka client apis directly IMO.

Thanks,

Michal

--

	Michal Borowiecki
	Senior Software Engineer L4
	T: +44 208 742 1600 +44 203 249 8448   E: michal.b...@openbet.com W: www.openbet.com	OpenBet Ltd Chiswick Park Building 9 566 Chiswick High Rd London W4 5XT UK

This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postm...@openbet.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by OpenBet for employment and security purposes. To protect the environment please do not print this e-mail unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company registered in England and Wales. Registered no. 3134634. VAT no. GB927523612