I don't know about :vimgrep, but in :grep, you can include/exclude files
or folders according to your ":h grepprg". I use "git grep" as my grep
program, so to exclude files or folders I do this.
:grep --no-index pattern -- :!folder :!file.txt
To reiterate, my 'grepprg' is set to this:
:let &gp = 'git grep --recursive-submodules -In $*'
--
Enan