Using vault to encrypt all credentials for connecting to your hosts and not storing your vault password on disk would help.
You could keep your ansible configuration in source control and build a new ansible controller virtual machine whenever you needed to do a deployment if you wanted to, although that would be a pain if you do lots of production deployments.
I am no expert but well managed secrets, removing weak crypto and keeping systems patched goes a long way towards getting a full nights sleep.
Jon