A little tracing with javax.net.debug=ssl has solved the issue.
The issue is our certificates lacked the extended key usage extension for client authentication. They have server authentication, but both appear to be needed (according to the SSL trace).
Renewing certs should sort that out. For reference the usage extensions that are needed are;
1.3.6.1.5.5.7.3.2 The certificate can be used for Client Authentication only
1.3.6.1.5.5.7.3.1 The certificate can be used for Server Authentication only