Infinite Message Retention Policy

[bill...@navimatics.com]

As mentioned in a previous message in this group, I am building a service that uses a finite resource pool. Although there are different ways that one can implement the pool, my favorite is to use a PubSub.

In this scheme resources would be referenced by messages in the PubSub. A pull from the PubSub would be used to implement resource acquisition and a publish into the PubSub would be used to implement resource release. Because the PubSub has at-least-one (and not exactly-once) delivey semantics this is an imprecise pool, but I can deal with getting the same resource twice elsewhere in my service. I can seed the PubSub with a number of resource-referencing messages and get a pool with 0 lines of code and 0 headaches.

The biggest problem with this scheme is the message retention policy. It is conceivable for resources to remain unused for a considerable period of time, for example during prototyping and developing or if the pool is purposefully over-provisioned. This means that the PubSub could eventually (after 7 days) evict some of those messages and the service could be starved out of resources.

So I am wondering if you would consider adding the ability for infinite message retention on a finite queue.

Bill

[Kir Titievsky]

Bill, Extending the retention period for a longer time that seven days is something we have considered.  What duration would have made this a non-issue for you?

Philosophically, we try to think of Pub/Sub as a notification system rather than a storage system.  This means infinite retention is not something we are looking into now.  Designs that treat Pub/Sub as a notification -- or triggering -- mechanism, rather than a persistence mechanism are more likely to be successful.  Meaning, if you anticipate tasks that might live for more than seven days, they should live in a database.  A periodic job should deal with tasks that are older than 7 days. 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/d9825947-c0ac-4813-8358-032e7df27104%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

[Bill Zissimopoulos]

In my scenario there would be a small number of messages (say less than 1000) that would potentially be retained “forever”. If “forever” is not possible, anything over 7 days would be an improvement. I would hope that at least 30 days should be possible.

I would prefer it if there was no need for a cron job that exists solely for the purpose of “touching” the messages and keeping them in the queue. If such a job is required, the PubSub mechanism becomes less attractive for this purpose.

I understand your point of PubSub being a notification system rather than a storage system. However queues can be used to build some very interesting constructs, so from the point of view of the queue user I would prefer as few constraints as possible.

Thank you again for your timely answers.

Bill

[Bill Zissimopoulos]

As an alternative you could also consider adding some form of dead-letter queue functionality to PubSub with undelivered messages being posted to the dead-letter queue after the 7-day retention period expires.

This would allow a subscriber to the dead-letter queue to examine those messages and possibly repost them to the pool queue.

Bill

You received this message because you are subscribed to a topic in the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cloud-pubsub-discuss/8-k6ruzIDzw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/D7501C7E.1283C%25billziss%40navimatics.com.

[Katayoon (Cloud Platform Support)]

I recommend that you create a feature request in the Issue Tracker so that you can get updates on your request there.