It depends on how you are using ZAP.
If you are _not_ using the scope then the easiest option is to set the passive scanner to only scan things in scope:
If you are using the scope then you another option would be to just uninstall all of the passive scan rule add-ons.
If thats not an option then you'll need to turn off each individual rule.
I was thinking about adding a "default passive scan rule level" option - if you raise an issue for that then it might be more likely to happen? :)
Cheers,
Simon