TLS SNI extension and literal IPv4 and IPv6 addresses in hostname

[Ivan Zhakov]

According to RFC 4366 [1] literal IPv4 and IPv6 addresses are not
permitted in "HostName" used for TLS SNI (server name indication)
extension:
[[[
"HostName" contains the fully qualified DNS hostname of the server,
as understood by the client. The hostname is represented as a byte
string using UTF-8 encoding [UTF8], without a trailing dot.

If the hostname labels contain only US-ASCII characters, then the
client MUST ensure that labels are separated only by the byte 0x2E,
representing the dot character U+002E (requirement 1 in Section 3.1
of [IDNA] notwithstanding). If the server needs to match the
HostName against names that contain non-US-ASCII characters, it MUST
perform the conversion operation described in Section 4 of [IDNA],
treating the HostName as a "query string" (i.e., the AllowUnassigned
flag MUST be set). Note that IDNA allows labels to be separated by
any of the Unicode characters U+002E, U+3002, U+FF0E, and U+FF61;
therefore, servers MUST accept any of these characters as a label
separator. If the server only needs to match the HostName against
names containing exclusively ASCII characters, it MUST compare ASCII
names case-insensitively.

Literal IPv4 and IPv6 addresses are not permitted in "HostName".
]]]

Currently serf just pass argument of serf_ssl_set_hostname() call to
OpenSSL function SSL_set_tlsext_host_name() and this violates spec and
causes server to reject request.

I want to add code to detect whether provided hostname is IP address
to serf. But I'm not sure is it serf library responsibility or caller
should not call serf_ssl_set_hostname() when hostname is numeric IP
address?

[1] http://www.ietf.org/rfc/rfc4366.txt

--

Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com