Design question about messages and multiple subscribers

[Darko Luketic]

I'm designing an ecommerce platform and am unsure about creating a monolith or using nats/stream as the message bus with microservices.

It's more of a theoretical question.

In the example of a customer creating an order,

I understand that the "web client" sends a "CustomerOrderCreated" to nats stream.

Now let's assume I have a OrderProcessor microservice and it has spawned 3 instances on 3 different machines.

The OrderProcessor's job is to listen for CustomerOrderCreated messages, process that attached data and send a CustomerOrderProcessed message.

However I now have 3 different instances listening for CustomerOrderCreated messages.

I don't want all 3 to process the same order.

How can I make sure only 1 OrderProcessor does the processing?

How do you manage this issue of client or listener synchronization?

I mean you could send a CustomerOrderProcessingStarted message but this still wouldn't stop all 3 instances of initially processing the order. They would all start processing and worst case even all 3 send the CustomerOrderProcessingStarted message deadlocking all instances.

[Colin Sullivan]

Hi Darko - thank you for using NATS!  What you're looking for a queue subscriber, which is supported in core NATS, NATS streaming, and JetStream.  When you specify a queue group name in subscriber creation, messages will only be delivered to one member of the group.  The enables NATS to effectively act as a layer 7 load balancer.  You can scale up or down at anytime with no configuration changes.  When scaling down use the Drain API for a graceful exit.

More on queue groups can be found here:

https://docs.nats.io/nats-concepts/queue

https://docs.nats.io/nats-streaming-concepts/channels/subscriptions/queue-group

[Darko Luketic]

Thank you Colin for the leads.

Now after having played with it, I don't understand,
when your service subscribes to a queuegroup, (nats server)
you have to provide channel and queue

qs, e := conn.QueueSubscribe("scraper.scrapePageRequest", "scrapeWorker", s.NatsMessageHandler)

But when publishing to the channel you only have the channel parameter.

So far I have only played with nats server, not stream. I understand if I'd like to be able to replay messages I'll have to use nats stream?
If not I'm not sure where it fits in.

Jetstream looks very promising.

> --
> You received this message because you are subscribed to a topic in the Google Groups "nats" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/natsio/BhCk97SPU5o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to natsio+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/natsio/9677d58a-0a82-4df7-b7ab-949df7d80380n%40googlegroups.com.


--
Best Regards

Darko Luketic

[iv...@synadia.com]

When you subscribe, you provide the subject you want to subscribe to, and for queue subscribers, the name of the queue group you want this subscription to belong to.

Publishers don't care about subscribers. They simply publish to the subject. The server will match any subscription to the published subject and deliver the message to those subscriptions. For queue groups, the server picks 1 member of each group to deliver the message to. Say you have:

- 3 regular subscriptions on "foo"

- 4 subscriptions on "foo", queue "bar"

- 2 subscriptions on "foo", queue "baz"

If a publisher publishes 1 message on "foo", the server will send one message to each of the regular subscriptions on "foo", 1 message to any of the 4 subscriptions on "foo" queue "bar", and 1 message to any of the 2 subscriptions on "foo" queue "baz".

Hope this helps.