Competing consumers

[Vinicius Carvalho]

Hi guys, I've just started with pubsub, I was wondering if it's possible to have a scenario like in kafka where we can create consumer groups and each group receive one message, but consumers of each group only get one copy of the message. I first thought I could do this by creating one subscription per group, and share the subscription with consumers, but it turns out that each consumer of the subscription gets its own copy of the message. Is it possible to map such scenario?

Regards

[Jason Ganetsky]

Your intuition is correct. Sharing a subscription with consumers should result in messages being distributed across consumers. Can you be more specific about how you are doing this? 

[Vinicius Carvalho]

Hi Jason thanks for the quick response. Let me try to be more specific on my use case.

My scenario is to load balance messages inside a group to several consumers on a queue fashion. Instead of having a fan-out approach where each consumer receives one copy of the message M1, I'd like to have only one consumer on that group (and hence I thought that inside a given subscription messages were only delivered to one consumer at a time) to consume the message. Here's my scenario:

Topic ---> S1, S2 (two subscriptions to the same topic)

S1 --> C1 (one consumer)

S2 --> C2, C3 (two consumers)

I send M1 and I expect to be delivered to S1 and S2 which happens. But I needed it to be consumed by C1 and by C2 OR C3 not C2 AND C3

Does this make sense?

Regards

[Jason Ganetsky]

Yep, that makes sense, and that's how it should already work. Is that not what you are seeing?

What are you doing specifically? Is each consumer doing Pull calls? Are they setting the same value for the subscription string in the PullRequest?

[Vinicius Carvalho]

You are right, I'm sorry. I noticed my consumer was actually creating a new subscription every time.

Sorry about that :)