How to support more number of webhooks without affecting it's performance

40 views
Skip to first unread message

Sneha S

unread,
Aug 6, 2018, 7:52:03 AM8/6/18
to Webhooks
Hi all,

 I am seeking help to decide on how can I develop a webhook system which can process some large number of request (pushing events to 30 to 50 URLs) without affecting its real-time pushing behaviour.

Now I am using Kafka producer-consumer system.    I have a webhook consumer which consume one event and push to every subscribed webhooks url.

------
step 1--> event received at the webhook consumer
step 2-->fetch all webhook subscribed to this event
step 3--> for each webhook apply the filter, construct the response and post it to webhook url.
------
Here I am using 10second socket timeout and I am doing serial processing cause I need to track the success and failure responses for each individual webhooks.

If I have 10 hooks it will result in the 1.6-minute delay to process the last webhook in the queue when considering the worst case scenario.  {{10 (number of hooks) *10 (socket timeout for one hook) = 100s }}

is there any strategy that I could follow to speed up my webhooks without compromising the webhook count?
Reply all
Reply to author
Forward
0 new messages