Most event types do not support "passive" dispatch in the way scrolling events do. Most event handlers, whether passed the passive flag or not, are still going to be called from the same task as all other event handlers, and processing those handlers will block the next paint regardless (and so will affect responsiveness, and will get measured by INP).
For scrolling, what happens is that the browser is actually able to implement default actions, then delay scheduling those JS events in the first place, so users have a smooth scrolling experience which cannot get blocked on main-thread delays. That is not true for most other (discrete) events, which INP measures.
-Michal