Unlimited topics in PubSub

[Joshua Fox]

Upon receiving a command that triggers the sending of a specific group of messages, the server pumps out 10-100  messages. A  client needs be be subscribed to receive exactly this group of messages. Thousands of such commands may be received in an hour, so we need a new "topic" for each  such command.

But there is a limit of 10,000 "topics."

How can this be accomplished in PubSub? Is there some concept other than "topic" to use? 

 Actually, 10K might be enough , if we can create a topic on each such command and destroy it at the end. But I don't think that topics are meant to be used that way. Even so, we might eventually hit the 10K limit.

Or is a technology other than PubSub preferable?

[Volkan Özdemir]

teşekkürler.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/CAD%3DB7cMfaeR94%2Bxn-SDkhMYYHUM9bdEE%2BYztufYAop_ch86txg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>

[George (Cloud Platform Support)]

Hello Joshua, 

How does your use-case look like, exactly, that requires creating and deleting topics at such a high rate? 

You seem to need delivering a large number of messages to each client. This can be accomplished without subscribing to a topic. You may consider a technology such as Firebase Cloud Messaging (FCM), which offers a broad range of messaging options and capabilities. For use cases such as instant messaging, a message can transfer a payload of up to 4KB to a client app.

[Joshua Fox]

On Tue, Jan 16, 2018 at 3:48 AM, 'George (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:

Hello Joshua, 

How does your use-case look like, exactly, that requires creating and deleting topics at such a high rate? 

A browser sends a request to a  frontend webapp which sends a request to generate objects in a time-consuming algorithm. The backend server gradually generates these objects and sends them out. The browser polls the frontend webapp asking if any objects are available. The  frontend server pull objects off the 'queue'/'topic'/'channel' and returns them to the browser.

The whole process takes under 2 minutes. 10-100 objects are generated on each request..  

 

You seem to need delivering a large number of messages to each client. 

This can be accomplished without subscribing to a topic. You may consider a technology such as Firebase Cloud Messaging (FCM), which offers a broad range of messaging options and capabilities. For use cases such as instant messaging, a message can transfer a payload of up to 4KB to a client app.

We need a larger payload; indeed, we'd prefer to have more than the 1MB offered in PubSub.

More generally, FCM  seems oriented to clients (enduser devices) whereas our messaging scenario is entirely server-side.

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/eb913382-f0ff-40a7-9a06-30bc10f2d131%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

 

JOSHUA FOX

Principal Software Architect | Freightos

T (Israel): +972-545691165 | T (US):  +1-3123400953  

Smooth shipping.

[George (Cloud Platform Support)]

Hi Joshua, 

You are encouraged to open a feature request of similar content in the issue tracker, so Developers may get a chance to have a thorough look and decide on eventual implementation. 

[Joshua Fox]

Actually, I believe that PubSub's basic design  is built around long-lived, not-too-numerous topics.  Short-lived, numerous "channels"/"topics"/"queues" are a basically different use case.

I sense that what I am describing requires a different messaging system  -- or perhaps PubSub already supports the use case in some way that I am not quite realizing.

On Sat, Jan 20, 2018 at 6:46 AM, 'George (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

Hi Joshua, 

You are encouraged to open a feature request of similar content in the issue tracker, so Developers may get a chance to have a thorough look and decide on eventual implementation. 

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/692229c0-d73a-4bd5-96ce-f7517e0490ac%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[George (Cloud Platform Support)]

Hi Joshua, 

You are right, a topic is meant to be used for a considerable amount of time, in any case not for just a one-time event with a bunch of messages. Are there any more specific requirements for this feature request? You are encouraged to submit it through appropriate channels, namely the public issue tracker. 

[Joshua Fox]

namely the public issue tracker. 

I did, and they sent me back to this Google Group and to Stack Overflow.

Followup: I asked on StackExchange and  it looks like the correct terminology is "temporary queues." See this how-to article on how to do this in the JMS standard.

So, it looks like if we want this, we'll need to go to a different messaging system.

On Tue, Jan 23, 2018 at 4:37 AM, 'George (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

Hi Joshua, 

You are right, a topic is meant to be used for a considerable amount of time, in any case not for just a one-time event with a bunch of messages. Are there any more specific requirements for this feature request? You are encourage to submit it through appropriate channels, namely the public issue tracker. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/7bf363dd-c9cd-4845-bb74-def8739a6f2d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[George (Cloud Platform Support)]

Hi Joshua, 

Your feature request in the issue tracker is getting attention and will be addressed shortly.