I agree with gevisz: a plugin could have an impact on performance even
if not used. Just imagine that there's a busy wait loop that fires up
when the plugin is loaded -- it would just hog the machine (well, at
least a core).
Nonetheless, mostly what plugins do upon startup is:
define functions
define various mappings, commands
Vim, of course, does have to determine whether or not a given sequence
of symbols invokes a command, function, or mapping. There are a number
of ways to do this; some are log-m (m=qty characters in the string,
n=qty of strings), so adding in extra mappings, commands, and functions
should be negligible cost in time.
So, I would not expect unused plugins to exact a performance cost.
Regards,
Chip Campbell