I got some further debug information, this is what the server output is when the connection happens
24/03/2021 16:59:46 Got connection from client 192.168.206.117
24/03/2021 16:59:46 other clients:
24/03/2021 16:59:46 Got 'ws' WebSockets handshake
24/03/2021 16:59:46 Got protocol: binary
24/03/2021 16:59:46 - webSocketsHandshake: using binary/raw encoding
24/03/2021 16:59:46 - WebSockets client version hybi-13
24/03/2021 16:59:46 cleaned up wsctx
24/03/2021 16:59:46 Disabled X server key autorepeat.
24/03/2021 16:59:46 to force back on run: 'xset r on' (3 times)
24/03/2021 16:59:46 webSocketsDecodeHybi_enter: len=12; CTX: readlen=0 readPos=0x55dd42f86900 writeTo=(nil) state=0 toRead=0 remaining=0 nReadRaw=0 carrylen=0 carryBuf=0x55dd42f9a1b8
24/03/2021 16:59:46 header_read to 0x55dd42f86900 with len=14
24/03/2021 16:59:46 read 14 bytes from socket
24/03/2021 16:59:46 first header bytes received; opcode=2 lenbyte=12
24/03/2021 16:59:46 header complete: state=0 flen=18 writeTo=0x55dd42f8690e
24/03/2021 16:59:46 calling read with buf=0x55dd42f8690e and len=4 (decodebuf=0x55dd42f86900 headerLen=6)
24/03/2021 16:59:46 read 4 bytes from socket; nRead=18
24/03/2021 16:59:46 toDecode=12 from n=4 carrylen=0 headerLen=6
24/03/2021 16:59:46 mask decoding; i=3 toDecode=12
24/03/2021 16:59:46 carrying over 0 bytes from 0x55dd42f8691e to 0x55dd42f9a1b8
24/03/2021 16:59:46 copy to 12 bytes to dst buffer; readPos=0x55dd42f86906, readLen=12
24/03/2021 16:59:46 after copy: readPos=(nil), readLen=0
24/03/2021 16:59:46 frame received successfully, cleaning up: read=0 hlen=6 plen=12
24/03/2021 16:59:46 cleaned up wsctx
24/03/2021 16:59:46 webSocketsDecodeHybi_exit: len=12; CTX: readlen=0 readPos=0x55dd42f86900 writePos=(nil) state=0 toRead=0 remaining=0 nRead=0 carrylen=0 carryBuf=0x55dd42f9a1b8 result=12
24/03/2021 16:59:46 Client Protocol Version 3.8
24/03/2021 16:59:46 Protocol version sent 3.8, using 3.8
24/03/2021 16:59:46 webSocketsDecodeHybi_enter: len=1; CTX: readlen=0 readPos=0x55dd42f86900 writeTo=(nil) state=0 toRead=0 remaining=0 nReadRaw=0 carrylen=0 carryBuf=0x55dd42f9a1b8
24/03/2021 16:59:46 header_read to 0x55dd42f86900 with len=14
24/03/2021 16:59:46 read 7 bytes from socket
24/03/2021 16:59:46 first header bytes received; opcode=2 lenbyte=1
24/03/2021 16:59:46 header complete: state=0 flen=7 writeTo=0x55dd42f86907
24/03/2021 16:59:46 calling read with buf=0x55dd42f86907 and len=0 (decodebuf=0x55dd42f86900 headerLen=6)
24/03/2021 16:59:46 toDecode=1 from n=0 carrylen=0 headerLen=6
24/03/2021 16:59:46 mask decoding; i=0 toDecode=1
24/03/2021 16:59:46 carrying over 1 bytes from 0x55dd42f86907 to 0x55dd42f9a1b8
24/03/2021 16:59:46 cleaned up wsctx
24/03/2021 16:59:46 webSocketsDecodeHybi_exit: len=1; CTX: readlen=0 readPos=0x55dd42f86900 writePos=(nil) state=0 toRead=0 remaining=0 nRead=0 carrylen=0 carryBuf=0x55dd42f9a1b8 result=-1
24/03/2021 17:00:06 ReadExact: select timeout
24/03/2021 17:00:06 rfbProcessClientSecurityType: read: Connection timed out
24/03/2021 17:00:06 client_count: 0
24/03/2021 17:00:06 Restored X server key autorepeat to: 1
24/03/2021 17:00:06 connect_once: invalid password or early disconnect. 1
24/03/2021 17:00:06 connect_once: waiting for next connection.
24/03/2021 17:00:06 Client 192.168.206.117 gone
24/03/2021 17:00:06 Statistics events Transmit/ RawEquiv ( saved)
24/03/2021 17:00:06 TOTALS : 0 | 0/ 0 ( 0.0%)
24/03/2021 17:00:06 Statistics events Received/ RawEquiv ( saved)
24/03/2021 17:00:06 TOTALS : 0 | 0/ 0 ( 0.0%)
I don't have a password configured so an invalid password can't be the reason