I just landed a feature in Treeherder which lets you bulk cancel a group
of jobs (but not necessarily the whole push). I wrote up a blog post
about it here (includes a tutorial video):
This is a relatively new feature so please file bugs (needinfo'ing me)
if you run into problems.
Hope people find this helpful,