After setting Acknowledgment Deadline, msgs still being pushed again frequently

[Eshaan Mehta]

Hi All,

I've a backend service where I am using the Push delivery type of pub/sub to push messages from one server to another. In the Pub sub console, I've created a subscription with an Acknowledgment Deadline of 600 seconds. The issue is that the same message keeps coming back after receiving it once within a span of max 20 seconds. Do I need to do something else to increase the time before a message is pushed again till an acknowledgment is sent ? Or is there any other setting which I've missed ?

In my scenario, I actually want to wait for the processing to be completed once a message is received and on completion of this process, I want to send an acknowledgement back. But the message keeps on coming back within a short span.

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

P please consider the environment before printing this e-mail

[Nick]

Hey Eshaan,

It's possible that messages can be delivered more than once, so the documentation recommends making subscribers idempotent. You could use various methods to store a flag for a given operation while it's being processed to let the subscriber code know, after checking that common storage location of flags, that it can safely discard a push message for an operation already received and in-progress.

If you set the ackDeadlineSeconds parameter, it seems that the receipt of a second message would not be related to the no-acknowledge-retry flow, but simply related to the possibility of receiving messages more than once. An outside possibility to consider is whether the code responsible to sending messages into the topic might be doubling-up the messages as well.

Let me know if this is helpful and whether you have any further questions. I'll be happy to assist. 

Cheers,

Nick
Cloud Platform Community Support 

[Kir Titievsky]

Eshaan,

This does not seems like regular accidental duplication. Let's try to cover the basics first: can you make sure you are both keeping the connection alive, waiting for a response, while you process the message?  Also, are you responding with a 2XX codes, when you do?  

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/bfe2aa2e-aa12-412e-8f4e-fe894b73399a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

[Eshaan Mehta]

Hi Kir,

Thanks for your revert. Your email got into my spam folder somehow, so couldn't reply earlier.

Anyways, after putting my question on the forum, I actually bought a paid support plan, and I raised a ticket for this. The update is that this issue has been confirmed as an issue in Pub/Sub. Now the person responding to the support ticket has sent an update that this issue has been fixed, but he has no confirmation from the Pub/Sub team on when its going to rollout into production. 

So do you have any updates on that ?

Below is the details to the ticket which I raised on Google support :

Created By: Cole Amick (9/9/16, 11:23 PM - 6 days ago)

Hello,

I am able to report that the Pub/Sub engineering team has implemented a fix for this issue, but at this time I am unable to report when the fix will be rolled into production. I sincerely appreciate your patience with this support case, and I am empathetic if you may be upset that I have been unable to provide you a meaningful update in a time you have expected. I would like to let you know that while I have not provided timely updates, I do appreciate that you have reached out to Google Cloud Platform support with this case, and that I do take this issue seriously. Please let me know if you have any additional questions or concerns.

Best regards,
Cole A
Google Cloud Platform Support

Created By: Cole Amick (9/7/16, 11:57 PM - 8 days ago)

Hello,

I do apologize for the length of time you have been waiting for suggestions on how to move forward with this support case. I now have confirmation from the product team that this is a known issue and it is being actively worked on to implement a fix for the behavior you are encountering. At the moment I do not have a timeline of when the behavior will be resolved, but I will be sure to give you an update on the issue tomorrow.

Best regards,
Cole A
Google Cloud Platform Support

Created By: Behzad Yari (9/7/16, 11:41 PM - 8 days ago)

I meant that the Pub/Sub message comes again from Pub/Sub, before the specified Acknowledgment deadline time.

Created By: Behzad Yari (9/7/16, 11:39 PM - 8 days ago)

The simple use case which is causing an issue for us is that the message from Pub/Sub comes before the specified Acknowledgment deadline time applied to the Pub/Sub subscription. We have some video processing to be done on receiving messages from Pub/Sub on our Google Compute Engine servers, but because of this issue, duplicate processing of the same message takes place which is causing unnecessary processing of the same message.

So please get us a definite answer to this issue, we have been waiting on this for quite some time now.

Created By: Cole Amick (9/7/16, 10:47 PM - 8 days ago)

Hello,

Thank you for providing more details on your use case. I am going to provide the additional details to the Pub/Sub product team. I will be sure to get back to you as soon as I have more insight into the behavior you are reporting. I greatly appreciate your patience on this case.

Best regards,
Cole A
Google Cloud Platform Support

Created By: Behzad Yari (9/7/16, 9:24 AM - 8 days ago)

Hi Cole,

My endpoint 'https://ftmpubsub.guguly.com/pubsubtest/' has been configured to process a request to create video from the image URLs it receives in the pub sub message using ffmpeg. So ffmpeg takes around a minute to create the video and once the video is created, we send an acknowledgement to the pub/sub message. Now the the pub/sub subscription created has "Acknowledgment Deadline" defined to 600 seconds. As per what I've read about "Acknowledgment Deadline" is that its "how long Pub/Sub waits for the subscriber to acknowledge receipt before resending the message". So this means if Pub/Sub sends a message to the subscriber, then it will wait for the specified "Acknowledgment Deadline" time till the subscriber has sent an acknowledge receipt. 

Based on the above, my understanding is that if I have specified an "Acknowledgment Deadline" time with the Pub/Sub subscription, then I should keep getting the same message after the "Acknowledgment Deadline" time if I don't send any acknowledge receipt. Is this correct ?

So now if we take my scenario as an example, it means that if my GCE server (subscriber), receives a push message from Pub/Sub and I don't send any acknowledge receipt back to the message, then I should get the same message repeated times, after the specified "Acknowledgment Deadline" time, which in this case is 600 seconds. But that is not what is happening. My GCE server keeps on receiving the same message in 20-30 seconds repeatedly. I have already shown you the console output from my server (see attached image: PubSub Console.png), where it clearly shows the time between each message received. 

Please let me know if you need any further details from me on this or if you're still not able to understand my scenario.

Created By: Cole Amick (9/7/16, 12:47 AM - 8 days ago)

Hello,

I want to check back with you to see if you are still have questions regarding case. From what I can tell it does appear that the Pub/Sub service is working as intended. I understand that in your use case you only want to re-push to your endpoint if no acknowledgement [1] has been received. Can you tell me more about how your endpoint 'https://ftmpubsub.guguly.com/pubsubtest/' has been configured; such as how long it waits until it acknowledges a message? 

Best regards,
Cole A
Google Cloud Platform Support

[1] https://cloud.google.com/pubsub/reference/rest/v1/projects.subscriptions/acknowledge

Created By: Cole Amick (9/3/16, 5:02 AM - 12 days ago)

Hello,

I greatly appreciate your patience with this case. I am going to bring this issue to the attention of the Pub/Sub team. I will be sure to provide you an update monday.

Best regards,
Cole A
Google Cloud Platform Support

Created By: Cole Amick (9/3/16, 12:19 AM - 12 days ago)

Hello,

I want to reach out to you and let you know that I am still actively investigating the behavior you have described in this case. I understand that this issue is important to you. I will be sure to contact you again momentarily.

Best regards,
Cole A
Google Cloud Platform Support

Created By: Cole Amick (9/2/16, 5:17 AM - 13 days ago)

Hello,

I am currently working on reproducing the behavior you have described. Thank you for the details you have provided. I will also be consulting with additional members of the Google Cloud Support team regarding this issue. I will be sure to report back to you tomorrow morning providing you an update on my investigation into this case.

Best regards,
Cole A
Google Cloud Platform Support

Created By: Cole Amick (9/1/16, 11:01 PM)

Hello,

Thank you for your patience with this case. My name is Cole and I will be assisting you with this case moving forward. I understand that are reporting the specified  acknowledgement deadline time you have applied to pub/sub is not being accurately enforced. Please allow me some time to further review this case and I will report back to you as soon as possible. 

I would like to propose to change case priority per the paragraph 5.15 of the Technical Support Services Guidelines [1]:

The case currently does not reflect that your application is generating severe user-facing errors, hence I propose to set case priority to P2. If there is additional impact not mentioned in the case, please feel free to let me know. Keeping case priorities up to date helps us to deliver better service to you.

Best regards,
Cole A
Google Cloud Platform Support
 
[1] http://support.google.com/enterprise/doc/terms/tssg_cloud.html

Created By: Behzad Yari (9/1/16, 9:02 PM)

Hi Lionel,

I am not convinced with your response. So what I tried is to create a test case for you. So I've added a GCE VM instance (facetrfm-pub-sub-multi-zone-9dlq) within the instance group: facetrfm-pub-sub-multi-zone, created in my account, with a test push endpoint defined. 

The details and screenshot for the Topic created in Pub/Sub Google Console is as shown in the screenshot : "PubSub Console.png", with the ack deadline time defined as 600 seconds.

Now in my GCE VM instance, I've defined the subscriber for the push endpoint :https://ftmpubsub.guguly.com/pubsubtest/ . If you see the screenshot ("VM Instance Console.png") from my VM instance, you can see that I've logged the message ("hi google this is pub sub test") which I published to this Google Pub/Sub subscription.   

Now if you see the time difference between each received message, you can see that once a message is pushed to my server from Google Pub/Sub, it again keeps on coming back ignoring the 600 seconds ack deadline time. 

So can you please look into this issue, by using a GCE instance and if you want, you can also check the same on my GCE VM instance.

Created By: Behzad Yari (9/1/16, 9:02 PM)

Uploaded

PubSub Console.png (37124 bytes)

Created By: Behzad Yari (9/1/16, 9:00 PM)

Uploaded

VM Instance Console.png (134762 bytes)

Created By: Lionel Mena (8/31/16, 7:12 PM)

Hi Behzad, 

I can't try with a GCE endpoint but I believe that, regardless of the push endpoint, Pub/sub is the sole responsible of the acknowledgement deadline logic so I don't think than using a custom push endpoint will change pub/sub behavior. 

I understand that, according to you, subscriptions are not following the ack deadline. This is why  I've set up a sample that pushes to a subscriptions with ack deadline of 10min set and an endpoint that will ack in 9 min and I didn't get any further delivery attempt which means that Pub/sub is following the established deadline of 10 min.

The only thing I found out after playing with different deadlines was that Pub/sub takes some time ( > 10s and <1min) to propagate changes to subscriptions.

Best regards,

Lionel M. 
Technical Solutions Representative 
Google Cloud Platform Support

Created By: Behzad Yari (8/31/16, 3:04 PM)

Lionel,

We are actually using GCE as the push endpoint, so can you try with that ? Also the issue is not that the messages are not getting acknowledged, the issue is that they are not following the ack deadline. I've tried creating different Subscriptions and changed ack deadline values as well, but the messages never actually follow the ack deadline time.

Please let me know what troubleshooting I can do for this, since the suggestion which you have given is not something which I am looking for. The thing I am looking for is the messages not following the acknowledgement deadline.

Created By: Lionel Mena (8/31/16, 2:09 PM)

Hi Behzad, 

I've tried to reproduce your issue from a GAE push endpoint with no luck. I set up the subscription from the console (in case the Cloud Console wasn't working properly) and tried with different deadlines and all were working as expected.

My suggestion would be to do more troubleshooting at your side to find out why messages are not getting acknowledge. To do this you can make use of the pertinent Stackdriver monitoring metrics where you can check the oldest unacknowledged and its count (subscription/oldest_unacked_message_age, subscription/num_outstanding_messages ) among other metrics [1]

Best regards,

Lionel M. 
Technical Solutions Representative 
Google Cloud Platform Support

[1] https://cloud.google.com/monitoring/api/metrics#gcp-pubsub

Created By: Behzad Yari (8/30/16, 7:12 PM)

Hi Lionel,

Thanks for your response. Please find answers inline to your questions:

1) Could you go to [1] and share the response you get for the subscription "pubsubpushtest"?
Here's the response:
{
 "name": "projects/show-me-there/subscriptions/ftm",
 "topic": "projects/show-me-there/topics/pubsubpushtest",
 "pushConfig": {
  "pushEndpoint": "https://ftmpubsub.guguly.com/?token=GOOGLE_PUB_SUB_TOKEN"
 },
 "ackDeadlineSeconds": 600
}

2) Have you checked the HTTP response code is being returned by the push endpoint? Any of the following HTTP status codes is interpreted as success by the Pub/Sub system: 200, 201, 204, or 102. If your service returns any other code, Google Cloud Pub/Sub retries indefinitely, using an exponential backoff algorit
Yes, we always send a  HTTP status code of 200 always. But that's not my concern. My problem is that whenever we receive a message pushed via Pub/Sub to our servers, then since we have set an ack deadline of 600 seconds, we expect that the same message should not be pushed again till 600 seconds. But what we found was that the message keeps on coming back after around 10-20 seconds.

3) What this working before in production and suddenly changed its behavior or you are just testing a push endpoint?
No, we haven't gone to production yet with Pub Sub, because we were using Amazon SQS before migrating to Google Pub/Sub, and we never faced such an issue in SQS.

4) How did you set ack deadline, through the API or through the Cloud Console?
We have set the ack deadline via Cloud console and not through API (you can see the attached screenshot for the settings we have made)

Created By: Lionel Mena (8/30/16, 6:57 PM)

Hi Behzad, 

Thank you for contacting Google Cloud Platform Support. My name is Lionel and I’m happy to assist you with this case.

I'd like to know the following

1) Could you go to [1] and share the response you get for the subscription "pubsubpushtest"?

2) Have you checked the HTTP response code is being returned by the push endpoint? Any of the following HTTP status codes is interpreted as success by the Pub/Sub system: 200, 201, 204, or 102. If your service returns any other code, Google Cloud Pub/Sub retries indefinitely, using an exponential backoff algorit

3) What this working before in production and suddenly changed its behavior or you are just testing a push endpoint?

4) How did you set ack deadline, through the API or through the Cloud Console?

Best regards,

Lionel M. 
Technical Solutions Representative 
Google Cloud Platform Support

On 9 September 2016 at 19:21, Kir Titievsky <k...@google.com> wrote:

Eshaan,

This does not seems like regular accidental duplication. Let's try to cover the basics first: can you make sure you are both keeping the connection alive, waiting for a response, while you process the message?  Also, are you responding with a 2XX codes, when you do?  

On Thu, Aug 25, 2016 at 2:33 AM, Eshaan Mehta <eshaan@metadesignsolutions.com> wrote:

Hi All,

I've a backend service where I am using the Push delivery type of pub/sub to push messages from one server to another. In the Pub sub console, I've created a subscription with an Acknowledgment Deadline of 600 seconds. The issue is that the same message keeps coming back after receiving it once within a span of max 20 seconds. Do I need to do something else to increase the time before a message is pushed again till an acknowledgment is sent ? Or is there any other setting which I've missed ?

In my scenario, I actually want to wait for the processing to be completed once a message is received and on completion of this process, I want to send an acknowledgement back. But the message keeps on coming back within a short span.

This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

P please consider the environment before printing this e-mail

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsubscrib...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/bfe2aa2e-aa12-412e-8f4e-fe894b73399a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--

Thanks & Regards

_________________________

Eshaan Mehta, Team Lead

MetaDesign Solutions, Gurgaon
Tel: +91-98-11-896496

Skype: mds_eshaan
eshaan@metadesignsolutions.com | www.metadesignsolutions.com

[Nick]

Hey Eeshan,

Did that issue close yet? The issue thread itself would probably be the best place to get that information. I can't speak authoritatively on the timeline for any roll-outs.

Cheers

Nick
Cloud Platform Community Support

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/bfe2aa2e-aa12-412e-8f4e-fe894b73399a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--

Thanks & Regards

_________________________

Eshaan Mehta, Team Lead

MetaDesign Solutions, Gurgaon
Tel: +91-98-11-896496

Skype: mds_eshaan
eshaan@metadesignsolutions.com | www.metadesignsolutions.com