So what I did was went in and updated the /var/opt/gitlab/git-data/repositories/reponame.git/config file so it looked like this:
[core]
repositoryformatversion = 0
filemode = true
bare = true
[pack]
windowMemory = 128m
packSizeLimit = 512m
and then repacked the repo: #git repack -a -d
I also increased the swap from 2 to 3GB but after these settings were in effect and the repo repacked, git hardly touched the swap file.
What I would love to know however is if a file I can edit that controls what those initial config files look like so I don't have to do this for every new repo that is created