number of topic limit

1,237 views
Skip to first unread message

Alberto Garcia

unread,
Oct 2, 2016, 12:57:34 PM10/2/16
to Google Cloud Pub/Sub Discussions
I am working in design of a project with devices, My plan is use pubsub to transfer messages but the limit is 10.000 topics per projects and in short term we will more than 50.000 devices and each device has his own topic.  Can be increase this limits?

Thanks
Alberto

Kir Titievsky

unread,
Oct 3, 2016, 5:55:06 AM10/3/16
to Alberto Garcia, Google Cloud Pub/Sub Discussions
Alberto,

Hi and thanks for writing.  Sounds like an exciting project.  Going to 50K topics might be technically possible, but I feel it might not be the right choice architecturally because managing and monitoring 50K topics individually is difficult.  

Could you tell me more about why you need a separate topic per device?  Typically, we see users dump everything in a single Topic and sort it all downstream.  

Kir
Product Manager
Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/e864bb60-1119-40e0-8190-133edee885c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

alberto.ags

unread,
Oct 3, 2016, 6:15:03 AM10/3/16
to Kir Titievsky, Google Cloud Pub/Sub Discussions
​Thank for you answer

My plan to create topic per device is we need send message a specific device-id. if my device create subscription to same topic I should create logic to discard message on client when the message is for other devices creating a lot data traffic consuming bandwidth and stressing client system because 99% of messages is for others.

I would like use pubsub as a rpc system to call invoke functions on device undividually. 

Thanks and I am novice so all suggest will be wellcome.

Alberto








 


​

On Mon, Oct 3, 2016 at 11:54 AM, Kir Titievsky <k...@google.com> wrote:
Alberto,

Hi and thanks for writing.  Sounds like an exciting project.  Going to 50K topics might be technically possible, but I feel it might not be the right choice architecturally because managing and monitoring 50K topics individually is difficult.  

Could you tell me more about why you need a separate topic per device?  Typically, we see users dump everything in a single Topic and sort it all downstream.  

Kir
Product Manager
Cloud Pub/Sub

On Sun, Oct 2, 2016, 12:57 PM Alberto Garcia <alber...@gmail.com> wrote:
I am working in design of a project with devices, My plan is use pubsub to transfer messages but the limit is 10.000 topics per projects and in short term we will more than 50.000 devices and each device has his own topic.  Can be increase this limits?

Thanks
Alberto

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

On Mon, Oct 3, 2016 at 11:54 AM, Kir Titievsky <k...@google.com> wrote:
Alberto,

Hi and thanks for writing.  Sounds like an exciting project.  Going to 50K topics might be technically possible, but I feel it might not be the right choice architecturally because managing and monitoring 50K topics individually is difficult.  

Could you tell me more about why you need a separate topic per device?  Typically, we see users dump everything in a single Topic and sort it all downstream.  

Kir
Product Manager
Cloud Pub/Sub

On Sun, Oct 2, 2016, 12:57 PM Alberto Garcia <alber...@gmail.com> wrote:
I am working in design of a project with devices, My plan is use pubsub to transfer messages but the limit is 10.000 topics per projects and in short term we will more than 50.000 devices and each device has his own topic.  Can be increase this limits?

Thanks
Alberto

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

Kir Titievsky

unread,
Oct 3, 2016, 6:59:53 AM10/3/16
to alberto.ags, Google Cloud Pub/Sub Discussions
Thanks for the explanation, Alberto.  That's a good reason if you are going to use Pub/Sub to communicate to the devices. That said, Pub/Sub is deigned for server-server communication so not an ideal tool for this job.  My first thought on this is to use a different mechanism for server->device communication from the rest.

I would keep device state in a persistent backend database and have the devices periodically poll for it with a what "What should my state be now?" This can be a synchronous operation with the front-end hosted on AppEngine (or AppEngine flex) or a VM.  With messaging, you are relying on each individual device to resolve it's target state correctly and perhaps don't event have a server model that knows what each device is doing. 

Depending on how sophisticated your devices are, Firebase, Google Cloud Messaging or webhooks running on the devices themselves might be good alternatives to devices polling your server for updates. 

I'll share this with a team that has more experience with this use case in in case they have better alternatives.


k


On Mon, Oct 3, 2016 at 6:15 AM alberto.ags <alber...@gmail.com> wrote:
​Thank for you answer

My plan to create topic per device is we need send message a specific device-id. if my device create subscription to same topic I should create logic to discard message on client when the message is for other devices creating a lot data traffic consuming bandwidth and stressing client system because 99% of messages is for others.

I would like use pubsub as a rpc system to call invoke functions on device undividually. 

Thanks and I am novice so all suggest will be wellcome.

Alberto








 


​

On Mon, Oct 3, 2016 at 11:54 AM, Kir Titievsky <k...@google.com> wrote:
Alberto,

Hi and thanks for writing.  Sounds like an exciting project.  Going to 50K topics might be technically possible, but I feel it might not be the right choice architecturally because managing and monitoring 50K topics individually is difficult.  

Could you tell me more about why you need a separate topic per device?  Typically, we see users dump everything in a single Topic and sort it all downstream.  

Kir
Product Manager
Cloud Pub/Sub

On Sun, Oct 2, 2016, 12:57 PM Alberto Garcia <alber...@gmail.com> wrote:
I am working in design of a project with devices, My plan is use pubsub to transfer messages but the limit is 10.000 topics per projects and in short term we will more than 50.000 devices and each device has his own topic.  Can be increase this limits?

Thanks
Alberto

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

On Mon, Oct 3, 2016 at 11:54 AM, Kir Titievsky <k...@google.com> wrote:
Alberto,

Hi and thanks for writing.  Sounds like an exciting project.  Going to 50K topics might be technically possible, but I feel it might not be the right choice architecturally because managing and monitoring 50K topics individually is difficult.  

Could you tell me more about why you need a separate topic per device?  Typically, we see users dump everything in a single Topic and sort it all downstream.  

Kir
Product Manager
Cloud Pub/Sub

On Sun, Oct 2, 2016, 12:57 PM Alberto Garcia <alber...@gmail.com> wrote:
I am working in design of a project with devices, My plan is use pubsub to transfer messages but the limit is 10.000 topics per projects and in short term we will more than 50.000 devices and each device has his own topic.  Can be increase this limits?

Thanks
Alberto

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

alberto.ags

unread,
Oct 3, 2016, 10:26:36 AM10/3/16
to Kir Titievsky, Google Cloud Pub/Sub Discussions
Hi Kir

Thanks is really appreciate your explanation. IMO is very important the point of view of products developer and when pub/sub have been designed for server-server is a relevant info. 
I like the idea of use firebase I will try this alternative again thank and I encourage to give this kind of explanation in a lot of cases it is more useful than a piece of code function.

Regards
Alberto 



k











 


​

Cloud Pub/Sub

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

On Mon, Oct 3, 2016 at 11:54 AM, Kir Titievsky <k...@google.com> wrote:
Alberto,

Hi and thanks for writing.  Sounds like an exciting project.  Going to 50K topics might be technically possible, but I feel it might not be the right choice architecturally because managing and monitoring 50K topics individually is difficult.  

Could you tell me more about why you need a separate topic per device?  Typically, we see users dump everything in a single Topic and sort it all downstream.  

Kir
Product Manager
Cloud Pub/Sub

On Sun, Oct 2, 2016, 12:57 PM Alberto Garcia <alber...@gmail.com> wrote:
I am working in design of a project with devices, My plan is use pubsub to transfer messages but the limit is 10.000 topics per projects and in short term we will more than 50.000 devices and each device has his own topic.  Can be increase this limits?

Thanks
Alberto

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

Reply all
Reply to author
Forward
0 new messages