If you create the ssl-listener with reuse?=true instead, like this
(define listener (ssl-listen port 5 #f #t ctx))
does the problem go away? If so, the error might happen because the OS reserves the port number for a while after the listener is closed; see the paragraph about TIME_WAIT in the tcp-listener docs. If that's the issue, I'm a little confused why the TCP version would succeed, though.
I believe SSL_shutdown is completely unrelated to the state of the TCP ports. Think of it like a "Connection: close" HTTP header; it announces your intention to close the TCP connection soon, but it has no actual effect on the transport layer.
Ryan