Deduplication using Google Cloud Queue's with a Pull Model

328 views
Skip to first unread message

Austin

unread,
Oct 12, 2021, 2:42:08 PM10/12/21
to Google App Engine
I am looking at the various cloud queue options to handle webhooks that can be received multiple times and out of order, before being processed by an internal system. It seems the pull task queue is being deprecated in favor of pub/sub, but pub/sub does not offer deduplication, retries, and scheduled delivery. I only need one handler/subscriber per message and would like to pull via the api however it seems these features are exclusive to cloud tasks or pub/sub, which is made more confusing since it is possible to create cloud task pull queues. If I use a google cloud task queue how can I configure the internal system to mark it self as available to process the next message in the queue?

Kaan

unread,
Oct 15, 2021, 10:11:15 AM10/15/21
to Google App Engine
Just use push queues with unique id's? When you get the name already exists exception, you'll know it's a duplicate, otherwise retry

I'm guessing there's a processing/cost restriction too

Reply all
Reply to author
Forward
0 new messages