TLS handshake failed with upgraded server - could it be old certificate?

[Eskil Varenius]

Hello,

I used to connect to a x11vnc server with TigerVNC. Now the server has been replaced (not by me). When I try to connect again, I get "Authentication error: TLS handshake failed". I suspect this may be due to a certificate saved somewhere on my Mac OS X system. Is this a viable suspicion, and if so: could someone tell me where to find and remove the old files for this server so I can start a fresh?

It could of course be another reason, and if so I'm grateful for advice.

Kind regards

Eskil

[Pierre Ossman]

Could you check the log? It should show more details as to why it failed.

Instructions for getting more logging here:

https://github.com/TigerVNC/tigervnc/wiki/Debug-Logs

Regards
--
Pierre Ossman Software Development
Cendio AB https://cendio.com
Teknikringen 8 https://twitter.com/ThinLinc
583 30 Linköping https://facebook.com/ThinLinc
Phone: +46-13-214600

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[Eskil Varenius]

Hi,

Thank you for the suggestions. This is what I get from the client side (masked with HOST, IP, PORT):

(venv) eskil@zappa Desktop % open /Applications/TigerVNC\ Viewer\ 1.11.0.app --args -Log \*:file:100
(venv) eskil@zappa Desktop % cat /tmp/vncviewer.log

Thu Aug 26 18:10:27 2021
 DecodeManager: Detected 4 CPU core(s)
 DecodeManager: Creating 4 decoder thread(s)
 TcpSocket:   Connecting to HOST [IP] port PORT
 CConn:       Connected to host HOST port PORT

Thu Aug 26 18:10:29 2021
 CConnection: reading protocol version
 CConnection: Server supports RFB protocol version 3.8
 CConnection: Using RFB protocol version 3.8
 CConnection: processing security types message
 CConnection: Server offers security type VeNCrypt(19)
 CConnection: Server offers security type [unknown secType](18)
 CConnection: Choosing security type VeNCrypt(19)
 CConnection: processing security message
 CConnection: processing security message
 CConnection: processing security message
 CConnection: processing security message
 CConnection: processing security message
 CVeNCrypt:   Server offers security type TLSVnc (258)
 CConnection: processing security message
 CVeNCrypt:   Server offers security type X509Vnc (261)
 CVeNCrypt:   Choosing security type TLSVnc (258)
 CConnection: processing security message
 TLS:         Anonymous session has been set
 CConnection: processing security message
 TLS:         TLS Handshake failed: A TLS fatal alert has been received.
 TLS:        
 CConn:       Authentication failure: TLS Handshake failed

On the server side, there is a log file (I have not configured it so I don't know the level) which contains this for my attempt:

26/08/2021 16:10:27 SSL: accept_openssl(OPENSSL_VNC)
26/08/2021 16:10:27 SSL: spawning helper process to handle: IP:23616
26/08/2021 16:10:27 SSL: helper for peerport 23616 is pid 2524:
26/08/2021 16:10:27 connect_tcp: trying:   127.0.0.1 20000
26/08/2021 16:10:28 check_vnc_tls_mode: waited: 1.419161 / 1.40 input: (future) RFB Handshake
26/08/2021 16:10:29 check_vnc_tls_mode: version: 3.8
26/08/2021 16:10:29 check_vnc_tls_mode: reply: 19 (VeNCrypt)
26/08/2021 16:10:29 vencrypt: received 0.2 client version.
26/08/2021 16:10:29 vencrypt: client selected sub-type: 258 (rfbVencryptTlsVnc)
26/08/2021 16:10:29 Using Anonymous Diffie-Hellman mode.
26/08/2021 16:10:29 WARNING: Anonymous Diffie-Hellman uses encryption but is
26/08/2021 16:10:29 WARNING: susceptible to a Man-In-The-Middle attack.
26/08/2021 16:10:29 loaded Diffie Hellman 1024 bits, 0.000s
26/08/2021 16:10:29 SSL: ssl_init[2524]: 7/7 initialization timeout: 20 secs.
26/08/2021 16:10:29 SSL: ssl_helper[2524]: SSL_accept() *FATAL: -1 SSL FAILED
26/08/2021 16:10:29 SSL: error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher
26/08/2021 16:10:29 SSL: ssl_helper[2524]: Proto: unknown
26/08/2021 16:10:29 SSL: ssl_helper[2524]: exit case 2 (ssl_init failed)
26/08/2021 16:10:29 SSL: accept_openssl: cookie from ssl_helper[2524] FAILED. 0

Looking at this myself, I wondered what would happen if I turned off the "Anonymous SSL" under "Security" in the client. Then I get a warning that the "hostname does not match server certificate" but if I accept this and proceed, I can actually log in. So, the problem seems to be that the server thinks TLSVnc should work, but it doesn't. If I then only accept X509Vnc on the client, then it works - with the warning. I guess I could fix a proper certificate at the server side, and use the X509Vnc. But I'm still curious why the TLSVnc doesn't work?

Kind regards

Eskil

[Pierre Ossman]

On 26/08/2021 18:16, Eskil Varenius wrote:
>
> Looking at this myself, I wondered what would happen if I turned off the
> "Anonymous SSL" under "Security" in the client. Then I get a warning that
> the "hostname does not match server certificate" but if I accept this and
> proceed, I can actually log in. So, the problem seems to be that the server
> thinks TLSVnc should work, but it doesn't. If I then only accept X509Vnc on
> the client, then it works - with the warning. I guess I could fix a proper
> certificate at the server side, and use the X509Vnc. But I'm still curious
> why the TLSVnc doesn't work?
>

Not entirely clear unfortunately, except for the fact that the client
and server aren't able to find a cipher to agree on.

I know there has been a push to disable anonymous tls recently, so there
might be some new defaults that interfere here. We have code that
explicitly works around this so it's likely that it's the x11vnc side
that needs some changes here.