I just landed a feature in Treeherder which lets you bulk cancel a group
of jobs (but not necessarily the whole push). I wrote up a blog post
about it here (includes a tutorial video):
http://wlach.github.io/blog/2017/02/cancel-all-the-things/
This is a relatively new feature so please file bugs (needinfo'ing me)
if you run into problems.
Hope people find this helpful,
Will