PubSub many-to-many, understanding supportability and limitations

[Wael Al Masri]

Hello everyone,

I'm using PubSub for a custom semi-centralised blockchain network where I have each node in the network publish and subscribe to transactions and blocks added to the chain.

My goal is to have a fanout approach where each node must receive the message even if one acknowledges it first. My understanding is that I will need a different subscription for each of the nodes in order to achieve the above.

When I looked into the limitations, it seems like there's a maximum number of subscriptions per project which is 10,000.

How can I achieve a the above for more than that number? Or is PubSub not a possible option for my implementation?

If not, do you recommend a specific alternative?

I'm aware of p2p methods used in other networks, but the issue there is speed of delivery which makes the network extremely slow with growth. In my case I do not care or even prefer the PubSub system to be centralised, so the "decentralised" argument here is not valid.

I thought of simply having my nodes "not acknowledging" messages, but that can be abused by anyone who touches the distributed code, and I'm not sure what other issues will arise from that.

Should I maybe have multiple projects, each with similar configs, and randomly select some of the nodes that will be listening to one and publishing to another to "relay" messages?

Any feedback, guidance, direction would be greatly appreciated.

[Jose Gutierrez Paliza]

I understand what you want but in Pub/Sub it is not possible to request an increase of subscriptions[1]. These limits of 10,000 subscriptions are per project.

The idea that you shared about having multiple projects seems feasible. Anyway that would have to be on your config side.

[1]https://cloud.google.com/pubsub/quotas#resource_limits