Best Practices

Jimmie Butler

unread,

Mar 6, 2017, 4:27:41 PM3/6/17

to Google Cloud Pub/Sub Discussions

Anyone have some articles on general/high level best practices for using google pubsub?

I'm also more specifically interested in best practices around what a topic should be conceptually. I've come across the idea of "information" instead of "commands", which I think means the topic shouldn't be in any way related to the service that would subscribe to the messages.

Kir Titievsky

unread,

Mar 6, 2017, 4:30:33 PM3/6/17

to Jimmie Butler, Google Cloud Pub/Sub Discussions

Jimmie,

Designing for topics to be independent of what will consume messages is definitely a sane place to start. A good answer, in the end, will probably depend on what you are trying to accomplish. Will you tell us a bit more about the problem you plan to solve?

k

On Mon, Mar 6, 2017 at 4:23 PM, Jimmie Butler <jimmi...@gmail.com> wrote:

Anyone have some articles on general/high level best practices for using google pubsub?

I'm also more specifically interested in best practices around what a topic should be conceptually. I've come across the idea of "information" instead of "commands", which I think means the topic shouldn't be in any way related to the service that would subscribe to the messages.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/eaac748f-3ab6-4eec-9958-1877d199dec2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

Jimmie Butler

unread,

Mar 6, 2017, 4:39:28 PM3/6/17

to Google Cloud Pub/Sub Discussions, Jimmie Butler

We have an events microservice which writes to some table, and several other microservices that need something to go through the events microservice (intents, messages etc). We'll probably have more in future.

It seems like there's 2 options

- Create an events topic, that means we never have to update the events microservice, but seems to couple the sender and the receiver.

- Create an "intent.new" or "message.new" topic that gets picked up by the events microservice. This would require an update to events to add more triggers.

Kir Titievsky

unread,

Mar 6, 2017, 4:52:28 PM3/6/17

to Jimmie Butler, Google Cloud Pub/Sub Discussions

Sounds to me like your events microservice is a consumer of data as it writes something down in response to events. I think what you are after, very basically, is all producers of events -- typically your front ends that respond to user requests -- dump data to a topic (or multiple topics, possibly). You then build your events service as something that understands how to pull and process events of different types. If you expect the events service to process all events, after doing some basic logic selection, it might be simpler to have all your front ends throw all events into the same topic (e.g. clicks and page views all go to the same place). If you expect different, independent downstream processing to happen to different types of events, you may want to create several different topics.

Does that help at all?

On Mon, Mar 6, 2017 at 4:39 PM, Jimmie Butler <jimmi...@gmail.com> wrote:

We have an events microservice which writes to some table, and several other microservices that need something to go through the events microservice (intents, messages etc). We'll probably have more in future.

It seems like there's 2 options
- Create an events topic, that means we never have to update the events microservice, but seems to couple the sender and the receiver.
- Create an "intent.new" or "message.new" topic that gets picked up by the events microservice. This would require an update to events to add more triggers.

On Monday, March 6, 2017 at 4:30:33 PM UTC-5, Kir Titievsky wrote:

Jimmie,

Designing for topics to be independent of what will consume messages is definitely a sane place to start. A good answer, in the end, will probably depend on what you are trying to accomplish. Will you tell us a bit more about the problem you plan to solve?

k

On Mon, Mar 6, 2017 at 4:23 PM, Jimmie Butler <jimmi...@gmail.com> wrote:

Anyone have some articles on general/high level best practices for using google pubsub?

I'm also more specifically interested in best practices around what a topic should be conceptually. I've come across the idea of "information" instead of "commands", which I think means the topic shouldn't be in any way related to the service that would subscribe to the messages.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsubscrib...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/eaac748f-3ab6-4eec-9958-1877d199dec2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/65b66549-a88f-4f2c-8b2f-5684517b6a03%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Jimmie Butler

unread,

Mar 6, 2017, 5:24:45 PM3/6/17

to Google Cloud Pub/Sub Discussions, Jimmie Butler

That helps, part of what I'm looking for is advantages/disadvantages to writing to a single event vs each service writing to their own events.

On Monday, March 6, 2017 at 4:52:28 PM UTC-5, Kir Titievsky wrote:

Sounds to me like your events microservice is a consumer of data as it writes something down in response to events. I think what you are after, very basically, is all producers of events -- typically your front ends that respond to user requests -- dump data to a topic (or multiple topics, possibly). You then build your events service as something that understands how to pull and process events of different types. If you expect the events service to process all events, after doing some basic logic selection, it might be simpler to have all your front ends throw all events into the same topic (e.g. clicks and page views all go to the same place). If you expect different, independent downstream processing to happen to different types of events, you may want to create several different topics.

Does that help at all?

On Mon, Mar 6, 2017 at 4:39 PM, Jimmie Butler <jimmi...@gmail.com> wrote:

We have an events microservice which writes to some table, and several other microservices that need something to go through the events microservice (intents, messages etc). We'll probably have more in future.

It seems like there's 2 options
- Create an events topic, that means we never have to update the events microservice, but seems to couple the sender and the receiver.
- Create an "intent.new" or "message.new" topic that gets picked up by the events microservice. This would require an update to events to add more triggers.

On Monday, March 6, 2017 at 4:30:33 PM UTC-5, Kir Titievsky wrote:

Jimmie,

Designing for topics to be independent of what will consume messages is definitely a sane place to start. A good answer, in the end, will probably depend on what you are trying to accomplish. Will you tell us a bit more about the problem you plan to solve?

k

On Mon, Mar 6, 2017 at 4:23 PM, Jimmie Butler <jimmi...@gmail.com> wrote:

Anyone have some articles on general/high level best practices for using google pubsub?

I'm also more specifically interested in best practices around what a topic should be conceptually. I've come across the idea of "information" instead of "commands", which I think means the topic shouldn't be in any way related to the service that would subscribe to the messages.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/eaac748f-3ab6-4eec-9958-1877d199dec2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/65b66549-a88f-4f2c-8b2f-5684517b6a03%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Jimmie Butler

unread,

Mar 8, 2017, 1:37:23 PM3/8/17

to Google Cloud Pub/Sub Discussions, Jimmie Butler

Is it a common pattern to from an endpoint send out directly to pubsub, rather than saving to DB first?

Guess that makes it easier to handle retries on a db or connection failure, but seems like then showing data that may be out of date.

Kir Titievsky

unread,

Mar 8, 2017, 11:34:25 PM3/8/17

to Jimmie Butler, Google Cloud Pub/Sub Discussions

Jimmie, both patterns are common. If your database is fast and reliable, write there first and to Pub/Sub after. If not, pub/sub will provide the reliability as the first layer.

On Wed, Mar 8, 2017 at 10:37 AM Jimmie Butler <jimmi...@gmail.com> wrote:

Is it a common pattern to from an endpoint send out directly to pubsub, rather than saving to DB first?
Guess that makes it easier to handle retries on a db or connection failure, but seems like then showing data that may be out of date.

On Monday, March 6, 2017 at 5:24:45 PM UTC-5, Jimmie Butler wrote:

That helps, part of what I'm looking for is advantages/disadvantages to writing to a single event vs each service writing to their own events.

On Monday, March 6, 2017 at 4:52:28 PM UTC-5, Kir Titievsky wrote:

Sounds to me like your events microservice is a consumer of data as it writes something down in response to events. I think what you are after, very basically, is all producers of events -- typically your front ends that respond to user requests -- dump data to a topic (or multiple topics, possibly). You then build your events service as something that understands how to pull and process events of different types. If you expect the events service to process all events, after doing some basic logic selection, it might be simpler to have all your front ends throw all events into the same topic (e.g. clicks and page views all go to the same place). If you expect different, independent downstream processing to happen to different types of events, you may want to create several different topics.

Does that help at all?

On Mon, Mar 6, 2017 at 4:39 PM, Jimmie Butler <jimmi...@gmail.com> wrote:

We have an events microservice which writes to some table, and several other microservices that need something to go through the events microservice (intents, messages etc). We'll probably have more in future.

It seems like there's 2 options
- Create an events topic, that means we never have to update the events microservice, but seems to couple the sender and the receiver.
- Create an "intent.new" or "message.new" topic that gets picked up by the events microservice. This would require an update to events to add more triggers.

On Monday, March 6, 2017 at 4:30:33 PM UTC-5, Kir Titievsky wrote:

Jimmie,

Designing for topics to be independent of what will consume messages is definitely a sane place to start. A good answer, in the end, will probably depend on what you are trying to accomplish. Will you tell us a bit more about the problem you plan to solve?

k

On Mon, Mar 6, 2017 at 4:23 PM, Jimmie Butler <jimmi...@gmail.com> wrote:

Anyone have some articles on general/high level best practices for using google pubsub?

I'm also more specifically interested in best practices around what a topic should be conceptually. I've come across the idea of "information" instead of "commands", which I think means the topic shouldn't be in any way related to the service that would subscribe to the messages.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/eaac748f-3ab6-4eec-9958-1877d199dec2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/65b66549-a88f-4f2c-8b2f-5684517b6a03%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Kir Titievsky | Product Manager | Google Cloud Pub/Sub

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/28f8b9cb-15d5-4bbe-884c-98b11357364a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward