How to support more number of webhooks without affecting it's performance

[Sneha S]

Hi all,

 I am seeking help to decide on how can I develop a webhook system which can process some large number of request (pushing events to 30 to 50 URLs) without affecting its real-time pushing behaviour.

Now I am using Kafka producer-consumer system.    I have a webhook consumer which consume one event and push to every subscribed webhooks url.

------

step 1--> event received at the webhook consumer

step 2-->fetch all webhook subscribed to this event

step 3--> for each webhook apply the filter, construct the response and post it to webhook url.

------

Here I am using 10second socket timeout and I am doing serial processing cause I need to track the success and failure responses for each individual webhooks.

If I have 10 hooks it will result in the 1.6-minute delay to process the last webhook in the queue when considering the worst case scenario.  {{10 (number of hooks) *10 (socket timeout for one hook) = 100s }}

is there any strategy that I could follow to speed up my webhooks without compromising the webhook count?