How does PubSub act as event Ingestion system and reliable/durable queue system both

[Balesh Kumar]

Hi All,

I have some experience in working with other cloud vendors like Azure and AWS, and these vendors have different offering for message queue systems and high speed event ingestion systems. For instance, Azure Service bus, AWS SQS, AWS SNS are some examples of messaging systems and EventHub and Kinesis are some examples of high speed event ingestion systems. These two are altogether different services and have different architectures. Event ingestion systems work something similar to Kafka, where subscribers do not have features like acknowledgement of message, dirty queues etc. Subscribers have to take care of check-pointing of messages themselves.
On the other side, messaging system provides all these features and thus assumed to be have low throughput. Also, you need to reserve some capacity (Throughput unit in case of eventhub) in event ingestion systems and system can only scale to that limit.
How does Cloud pubsub plays both of these roles? And how does it make sure that it still provide high throughput?

Thanks
Balesh

[Jordan (Cloud Platform Support)]

You are correct that Google Cloud Pub/Sub is a marriage of both the event ingestion system and messaging system that you have described. 

Pub/Sub is much like the messaging system you defined as it makes use of Publishers that publish messages to Topics which can have many Subscribers who all receive those messages and can acknowledge them after they have been processed. Pub/Sub also automatically scales without having to manually define or reserve capacity units. 

Pub/Sub is also very much like the described event ingestion system as Pub/Sub is built to be used and depended on by event-driven computing systems for event ingestion and delivery in multi-service configurations with reliable throughput and performance.

- You can find the common uses of Pub/Sub directly in our overview documentation, and find the exact throughput limits in our Quota page.  

 

[Kamal Aboul-Hosn]

Hi, Balesh,

You are right that Google Cloud Pub/Sub tends to cover both roles as a message queue and as a high-throughput message ingestion system. We have seen it used for both by internal Google teams and by external Cloud customers. High throughput with high availability has always been Pub/Sub's primary objective. The system can scale horizontally, distributing load among many servers based on the needs of any particular topic. This is why the cost of the service is based on usage and not capacity. This focus on scale comes at the cost of some advanced features you might see in traditional message queues. We'd continue to consider implications for scaling and availability whenever we'd look at adding features of this nature in the future. There is a little more detail in our architecture overview.

I hope that helps!

Kamal