Missed Github webhook events

[Bruce Becker]

Hello!

I would like to ask for some advice in debugging a strange issue we have in our Jenkins : pipeline events are often not triggered.

The setup is as follows:

1. Several (O(100)) multibranch pipeline jobs configured with Github source plugin, with Jenkinsfile in the repository
2. Each repository has a webhook to the usual /github-webhook/ url on our controller
3. Each multibranch pipeline job is configured to listen to events on specific branch (e.g. main) and pull requests to that branch
4. The jobs are configured to scan the github repo once a day for changes -- we kept this rate really low because of the number of jobs and the resources it was taking on the jenkins controller when a scan was triggered.

Desired behaviour:

• New pull requests create new PR jobs in jenkins.
• Merged pull requests remove the PR job and trigger the mainline branch ('main' above)

Actual behaviour

• New pull requests are sometimes created and triggered
• Many merge events do not trigger the main branch pipeline.

I know that we are receiving the events, because i can see them in the jenkins log.

What I cannot explain is that sometimes the event in Github triggers and event in Jenkins and sometimes not. It just seems like there is a sampling or filtering of events going on. I can't find any pattern, only that it seems to increase with the increased number of jobs in our Jenkins.

Has anyone seen similar behaviour?

Please let me know if I can provide any further details...
Thanks,
Bruce