The server doesn't support that at the moment, and I see why that's
potentially a problem with '-maxconnections 1' (because, since
-maxconnections is implemented at the socket level, the TCP connection
that is held open while the authentication dialog is open counts toward
the connection limit.) -rfbwait just specifies the maximum amount of
time that the server should wait for a network read or write to
complete. The problem is that, when the viewer's authentication dialog
is open, the server isn't waiting for a read or write to complete. It's
just sitting idle until the viewer sends it the expected authentication
credentials, and the RFB protocol doesn't require that those credentials
be sent in a timely manner.
TigerVNC has a parameter called "IdleTimeout" that drops a connection
after a specified period of inactivity. That effectively prevents an
open VNC viewer authentication dialog from holding open a TCP connection
to the server indefinitely, but the problem with IdleTimeout is that it
doesn't just apply to authentication. If, for instance, you specify
IdleTimeout=60, the VNC connection will be dropped if the user steps out
of the office for a minute. That's not generally desirable behavior. I
think the correct solution would be to implement an authentication
timeout-- to specify that, after the server enters the authentication
state for a particular connection, authentication must complete within a
specified period of time or the connection will be dropped. I'm happy
to implement that feature as a funded development project. Contact me
offline for an estimate if you are interested in pursuing that.