One topic and multiple subscribers

[Suman Biswas]

Hi,

We have one topic (published new/updated/deleted data) and multiple pull-subscribers (all subscribers will process new/updated/deleted data). Since we have multiple subscribers, we don't want to send ack notification by any of the subscribers but want to send ack notification once messages processes by all subscribers. What is your suggestion?

Thanks,

Suman

[Suman Biswas]

Is there any way to achieve this functionality?

[Kamal Aboul-Hosn]

Hi, Suman,

You should create a separate subscription for each of your subscribers, have each one pull from its subscription, and have every subscriber acknowledge a message when it is done processing it.

Regards,

Kamal

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/805c232d-5679-4d34-84a5-74253cba431f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yannick (Cloud Platform Support)]

Hello Suman, as Kamal pointed out each pull-subscriber manages their own subscription and can acknowledge them individually without affecting the other subscribers at all. Knowing this, do you still need to synchronize the work done by all of your subscribers? If so a more thorough explanation of your use case would help us find the best solution.

On Monday, August 21, 2017 at 7:58:02 AM UTC-4, Kamal Aboul-Hosn wrote:

Hi, Suman,

You should create a separate subscription for each of your subscribers, have each one pull from its subscription, and have every subscriber acknowledge a message when it is done processing it.

Regards,

Kamal

On Sun, Aug 20, 2017 at 9:06 PM Suman Biswas <suman....@gmail.com> wrote:

Is there any way to achieve this functionality?

On Sunday, August 20, 2017 at 5:42:40 PM UTC-7, Suman Biswas wrote:

Hi,

We have one topic (published new/updated/deleted data) and multiple pull-subscribers (all subscribers will process new/updated/deleted data). Since we have multiple subscribers, we don't want to send ack notification by any of the subscribers but want to send ack notification once messages processes by all subscribers. What is your suggestion?

Thanks,

Suman

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

[Suman Biswas]

Hi Yannick,

My understanding about subscription process is I need to create to separate service account for different subscriber. Please correct me if I am wrong.

Here is my configuration:

I have one toipc & 2 subscribers and for each subscriber I have separate subscription process . 

1. Publishing 102 messages

2. I am sending ack notification from each subscription process. 

Issue:

1. Running subscription - 1 process. Sometime it process 50 or 60 or 38 messages. It's random

2. Running subscription - 2 process. There is no message to process

Here is my subscription code snippets 

MessageReceiver receiver = new MessageReceiver() {

@Override

public void receiveMessage(

PubsubMessage message,  AckReplyConsumer consumer) {

System.out.println("MessageID::"+message.getMessageId()+" Message Data::"+message.getData().toStringUtf8());

messages.add(message);            

consumer.ack();

}

};

      try {

  subscriber = Subscriber.defaultBuilder(SubscriptionName.parse(localHashMap.get(ARG_SUBSCRIPTION_NAME)), receiver).setCredentialsProvider(PubSubUtil.createPubsubCredentials(localHashMap.get(ARG_PUBSUB_CRDNTL_FILE))).build();

            

           subscriber = createSubscriberWithErrorListener (subscriber, received);

           subscriber.startAsync().awaitRunning();

           Thread.sleep(6000);

          } catch(Exception e){

         System.out.println("ERROR::"+e.getMessage());

              throw e;

          }

        finally {

         

            if (subscriber != null) {

               subscriber.stopAsync();

            }

          }

Can you please let me know why this issues are?

[Yannick (Cloud Platform Support)]

I think I'm going to need more information about your use case to answer you properly.

First of, you do not need to have a service account for each individual subscriber. If you only have a single subscription shared by several subscriber process, than it is expected that not all of them will receive every message as they are both pulling from and acknowledging the messages from the same subscriptions.

If that's not quite what you intended to be doing, read the subscriber overview so you can better decide which method you should use. Keep in mind that if you want each subscriber to receive every message from a topic, they all need to have their own subscription to the same topic.

As for your code you could try basing yourself on the snippets from the pull subscriber guide.

[Suman Biswas]

Hi Yannick,

Thanks for your update. 

Here is my use case:

We have one upstream and 2 downstream systems. Whatever data upstream will publish both downstream systems should process all data.

Configuration:

one topic and 2 pull subscription processes

As you stated, I have two different pull subscription processes (both are pull subscription processes) for two different subscribers. As you stated, since I have 2 different subscription processes, both subscriber will be able to process all messages. Correct me if I am wrong.

Still there facing following issues 

1. Running subscription - 1 process. Sometime it process 50 or 60 or 38 messages. It's random

2. Running subscription - 2 process. There is no message to process

Thanks,

Suman

[Suman Biswas]

Can you please help me with an example where two different subscribers process all messages from a single topic?

Thanks,

Suman

[Yannick (Cloud Platform Support)]

I believe there might be some confusion so I'll take a moment to clearly establish what the terms are.

Messages are posted from any number of publishers to a topic. That topic distributes every message it receives to any number of subscriptions. The subscriptions are simply lists that hold all of the messages received by that subscription.

Subscribers (which I believe you have been calling subscription processes), whether they are Pull or Push, are the consumers of the subscription. Several subscribers can share a single subscription, meaning they all get a portion of the subscription. This means that if you want to ensure that a subscriber receives every message sent to a topic.

So knowing all this we can look at 2 example configurations:

• 1 topic, 1 subscription, 2 subscribers per subscription, 2 subscribers total: If you publish 10 messages, they will be shared between the subscribers of the single subscription. So each subscriber could get 0-10 messages.
• 1 topic, 2 subscriptions, 1 subscriber per subscription, 2 subscribers total: If you publish 10 messages, each subscription gets all 10 messages. Since each subscription only has 1 subscriber, that subscriber will receive every message. This means that both subscribers will each receive 10 messages.

I hope this made sense and was of use to you.

[Suman Biswas]

Hi Yannick,

I got it and tanks for your explanation. My process is running fine now. 

Thanks,

Suman