Ansible doesn't know what variables you are going to use in a template up front, so any group or host variable in the configuration will be loaded.
You may wish to move your secured files to something like (toplevel) production.yml and do a
ansible-playbook site.yml -e @production.yml
and reduce the number of vault files you have, and this way it would only load the production one.
Alternatively, keep your inventory in different directories
-i inventory/prod/inventory.ini, group_vars, host_vars
-i inventory/dev/inventory.ini, group_vars, host_vars
To keep them isolated.