Yeah, use it without plugins and start adding them as you need. If you use Vundle (my preference) or Pathogen to install plugins, it's all very easy to just install something and remove it if it doesn't seem to be of any benefit.
I went from TextMate to Emacs, which requires *a lot* of screwing around to make it useful. When I left Emacs and moved to Vim, despite the usual muscle-memory issues, I was pleasantly surprised how much Vim already did things I had to jump through hoops to get Emacs to do. You'll probably find you only really want a handful of plugins.
Inviato da iPhone