We are also experiencing the same issue. I am trying to setup a job to be triggered by pull requests which uses the following refspec: +refs/pull-requests/:refs/remotes/origin/pull-requests/ What happens is that Jenkins is trying to build all existing pull requests before it even starts reacting to push notifications.
I've not tried this, but you could look at "Additional behaviours" > "Strategy for choosing what to build" > "Maximum age of commit" to limit what is getting built.