Webhooks as Pub/Sub producers?

1,132 views
Skip to first unread message

Avinash Ananth

unread,
Dec 1, 2017, 6:33:30 PM12/1/17
to Google Cloud Pub/Sub Discussions
I use a few services that provide webhooks to receive data about various events of interest.

While coming across the rest API for producers https://cloud.google.com/pubsub/docs/publisher , I was wondering if there are plans to support a non-base 64 encoded message payload (i.e. raw JSON payloads) so I can configure webhooks from aforementioned services to be enqueued into Pubsub without creating and maintaining middleware. 

This would make it a lot easier for me to reliably process these webhooks. 

Jordan (Cloud Platform Support)

unread,
Dec 1, 2017, 7:52:11 PM12/1/17
to Google Cloud Pub/Sub Discussions
Looking into the API documentation it looks as though it supports gRPC (aka non-JSON raw binary). Therefore the message can be your raw data. The new Client Libraries for Pub/Sub also use gRPC by default. 

- Note that Google Groups is for general product discussions and is not for reporting Google-end feature requests. To report a feature request it is recommended to use the Public Issue Tracker. 

Kir Titievsky

unread,
Dec 4, 2017, 11:48:01 AM12/4/17
to Jordan (Cloud Platform Support), Google Cloud Pub/Sub Discussions
Avinash,

There are couple schools of thought on this.  The issue with un-encoded payloads is that it's easy for a malformed json string to slip in. And now you have to handle a request that you can't parse as JSON.  While encoded payload can always be passed around safely.  When you are ready to decode, it's typically a single method call away from a full string.  Our opinion has been that the portability of encoded payloads is more important than sparing developers that decoding step.  But we may have been wrong.  Could you tell us what about encoded payloads makes them much more difficult to process in your case?

k
-- 
Kir Titievsky | Product Manager | Google Cloud Pub/Sub 


On Fri, Dec 1, 2017 at 7:52 PM 'Jordan (Cloud Platform Support)' via Google Cloud Pub/Sub Discussions <cloud-pubs...@googlegroups.com> wrote:
Looking into the API documentation it looks as though it supports gRPC (aka non-JSON raw binary). Therefore the message can be your raw data. The new Client Libraries for Pub/Sub also use gRPC by default. 

- Note that Google Groups is for general product discussions and is not for reporting Google-end feature requests. To report a feature request it is recommended to use the Public Issue Tracker. 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/8b1a0be3-9a69-43ad-8faa-b8704e573169%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Avinash Ananth

unread,
Dec 4, 2017, 3:51:15 PM12/4/17
to Google Cloud Pub/Sub Discussions
Jordan,

I don't have or would not like to have control over the producer in my use case (I'll try to describe this further below in this message)

Kir,

My use case as I mentioned is to be able to reliably handle webhooks from various third-party services. There are two problems here 
  • Reliability in receiving messages from these services 
  • Reliability in my applications handling these webhooks and accounting for failures 
The concern I'm trying to address here is on the "interface" of these two concerns, leaning more towards the latter. 

An ideal world in this use case for me would be having an endpoint from Google corresponding to a topic that can accept arbitrary (JSON) payloads and enqueue them, so my problem space is now reduced to interacting with these queues. The usage would look something like
In the third-party service that offers webhooks:
  • I configure a target endpoint to be the Pubsub queue publish URL 
  • Set the required auth headers
  • I don't care about the shape of the webhook payload as I expect the Pubsub publish endpoint to take in arbitrary payloads
In my application(s):
  • I map my Pubsub topics to handlers for third-party service
  • Now I have a consistent and easier strategy around planning for downtime, retries, error tracking, monitoring, scaling etc. 
  • Can move towards not owning infrastructure to perform these tasks by adopting "serverless" methods (Google cloud function etc.)
Reg. the schools of thought on encoding - Given you folks have public APIs, I'd argue you folks are already validating the shape of your inputs in some form anyway right (format, size etc.)? 
I'm taking the liberty here of assuming server-side validations happens on your side in addition to any checks in the clients you vend. 

Note, I'd have no issues handling encoded payloads of any sort on the consumer side. The flexibility I'd like to have is for producers, with the goal of decouple the receiving and processing webhooks/events without maintaining a producer bus of sorts on my side. 
This use case would make idempotency keys / deterministic message IDs not possible, but that is acceptable for me. 

Does this make sense?


On Monday, December 4, 2017 at 8:48:01 AM UTC-8, Kir Titievsky wrote:
Avinash,

There are couple schools of thought on this.  The issue with un-encoded payloads is that it's easy for a malformed json string to slip in. And now you have to handle a request that you can't parse as JSON.  While encoded payload can always be passed around safely.  When you are ready to decode, it's typically a single method call away from a full string.  Our opinion has been that the portability of encoded payloads is more important than sparing developers that decoding step.  But we may have been wrong.  Could you tell us what about encoded payloads makes them much more difficult to process in your case?

k
-- 
Kir Titievsky | Product Manager | Google Cloud Pub/Sub 


On Fri, Dec 1, 2017 at 7:52 PM 'Jordan (Cloud Platform Support)' via Google Cloud Pub/Sub Discussions <cloud-pubs...@googlegroups.com> wrote:
Looking into the API documentation it looks as though it supports gRPC (aka non-JSON raw binary). Therefore the message can be your raw data. The new Client Libraries for Pub/Sub also use gRPC by default. 

- Note that Google Groups is for general product discussions and is not for reporting Google-end feature requests. To report a feature request it is recommended to use the Public Issue Tracker. 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

Kir Titievsky

unread,
Dec 4, 2017, 4:53:22 PM12/4/17
to a...@avinashak.com, Google Cloud Pub/Sub Discussions
I think I understand.  It sounds like the primary issue is being able to deduplicate, sort or route messages without doing much with the content. If that's so, Pub/Sub allows you to specify up key-value pair "attributes" on messages which are transmitted un-encoded.  Does that help?


To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/6d93b8b8-e943-45fe-b39c-1efdfb94008d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Avinash Ananth

unread,
Dec 4, 2017, 4:58:42 PM12/4/17
to Google Cloud Pub/Sub Discussions
Not quite - the primary issue is to allow a producer to enqueue/publish a message to a topic via a HTTP(s) POST to an endpoint that takes arbitrary payload/JSON (regardless of validation) . To put it another way, goal is to avoid maintaining a HTTP endpoint to receive webhooks.  
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

Kir Titievsky

unread,
Dec 4, 2017, 5:32:56 PM12/4/17
to a...@avinashak.com, Google Cloud Pub/Sub Discussions
Could you tell me a bit more about this third party webhook service you work with? Trying to get a sense of what things are hard for it and which are not.  E.g. a Pub/Sub push payload is valid JSON, even if the data field is encoded.  Presumably, if you got un-encoded content you'd still have to configure some kind of schema to interpret it.  If not, you are not doing any better than just working with a string. 


To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
--
Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/ec9c2ba3-f426-409e-a9ac-384c22af6923%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Dan Cowell

unread,
Dec 5, 2017, 5:10:04 AM12/5/17
to Google Cloud Pub/Sub Discussions
It looks to me like the two of you are talking about two different ends of the PubSub transaction.

Kir, I believe you're talking about push delivery of messages from Pub/Sub (i.e. a push subscription) while Avinash, you're talking about delivery of an arbitrary string (JSON in this case) to Pub/Sub (i.e. entering a Pub/Sub API publisher endpoint into a third party's webhook config, to which they would POST a payload of arbitrary, unencoded JSON that you have no control over, which would be accepted by Pub/Sub).

There are two issues here.

1. I don't believe there is an endpoint in the Pub/Sub API which supports non-authenticated publish operations, and I can almost guarantee whichever vendor is providing this webhook functionality won't support Google OAuth.
2. The Pub/Sub API requires a structured message by design.

Realistically, the best approach here is probably a lightweight HTTP Cloud Function which accepts arbitrary data, handles auth with Pub/Sub (this should be automatic with the node Client Library, I think) and publishes a message in the correct format to the topic of your choice. This would probably be doable in less than 30 LOC.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
--
Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages