I did some hacks to attempt to get different settings per-server, for the case where I was deploying an app multiple times for different tenants on the same host system. I wanted to change the user, deploy_to, and annoyingly, I had to have a different git_wrapper_path, because otherwise they get deployed to /tmp, and conflict with each other.
Many would suggest this is an ideal use case for containers, and maybe I'll back and do that, but at the time containers were just too immature on the target platforms I cared about.
I'm not clear if you are trying to do something similar or not.
While I eventually got it working well enough for my needs, I what I found is that the places where certain variables were set/initialised or defaulted made it rather difficult. I think that I have a pull request lingering if you really want it.
My suggestion is to attempt to localise the things you need to do, and then create a task that runs config/deploy/${HOSTNAME} to do the things you need to do which are server specific.