Awx vendors (copy and paste at a point in time) a copy of ansible's dynamic inventory. We do this because we know it to work with the dependencies that we pin (i.e. boto version for aws or shade version for openstack). Therefore, the feature set of the vendored version may be behind upstream releases of ansible.
In searching the code I don't find any results for groupby_custom_field_excludes.
Now let's try and find this feature in Ansible. After some googling I see that PR 36877 added support for the feature your looking for!
https://github.com/ansible/ansible/pull/36877 From the PR tags we can see that it affects ansible 2.6 so let's go to the 2.6 branch.