At one time there was a setting for this, and there may still be but I seem to recall it has suffered from bit rot and does not work as well as it used to.
Search source code in frameworks/base and packages/apps/Settings for all references to the symbol
USB_AUDIO_AUTOMATIC_ROUTING_DISABLED and then follow the bread crumbs from there.
Please update here with anything you find, as I am curious too.