Yes, the settings in default.yml seem to have been the issue.
The problem was not connecting to mediasoup from the client, but as you said, connecting FS<->mediasoup internally.
The current standard settings seem to rely on having the machine's public facing IP address configured on a dummy interface, since the freeswitch configuration is build for NAT translated access. Or maybe not; I don't understand it completely.
Since I've now got two SIP profiles for public and private network addresses, I needed to set mediasoup to use the private address profile and a local address to connect to freeswitch.
Both listen only and microphone now work, on both private and public addresses. Only IPv6 continues to be a problem.