How to re-use a socket with a new SSL session?

Andrew Sumner

unread,

Sep 10, 2009, 7:57:14 PM9/10/09

to

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01CA3270.38895250
Content-Type: text/plain;
charset="iso-8859-1"

I've got a situation where a listener on the server receives a connection
then the client sends a login packet. The server creates a new "worker"
process as the logged in user and passes it the socket ID. The worker
process takes over communication with the client (which is unaware of what's
happened on the server) and the listener listens for a new connection. Its
old, mature software, running on Windows that can't be changed that much
(large customer base).

Implementing openssl I've hit a wall. I need to initiate SSL on the
connection in the listener, before the client sends the login packet as it
obviously shouldn't be sent in the clear. There's no way to pass the SSL
objects to the new process, so I've been trying to close the SSL session and
initiate a new one on the still-open socket from the worker process.

To do this, the listener sends a packet to the client when it starts the
worker process, telling the client to shutdown its SSL. The listener does
the same (calls SSL_shutdown then SSL_free). The worker process sends a
packet in the clear to the client on the socket (this works), then calls
SSL_new, BIO_new_socket, SSL_set_bio and SSL_accept. When the client gets
the packet in the clear from the worker process, it also calls SSL_new,
BIO_new_socket, SSL_set_bio and then SSL_connect. So far so good, it all
appears to work.

The client can then send messages to the server worker process using
SSL_write, and the worker receives them ok. However, when the worker sends
something to the client, nothing comes through - SSL_read fails.

In summary, is there any way of closing an SSL session on a socket, then
opening a brand new one?

regards,
Andrew

------_=_NextPart_001_01CA3270.38895250
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
5.5.2653.12">
<TITLE>How to re-use a socket with a new SSL session?</TITLE>
</HEAD>
<BODY>

I've got a situation where a listener =
on the server receives a connection then the client sends a login =
packet.  The server creates a new "worker" process as =
the logged in user and passes it the socket ID.  The worker =
process takes over communication with the client (which is unaware of =
what's happened on the server) and the listener listens for a new =
connection.  Its old, mature software, running on Windows that =
can't be changed that much (large customer base).

Implementing openssl I've hit a =
wall.  I need to initiate SSL on the connection in the listener, =
before the client sends the login packet as it obviously shouldn't be =
sent in the clear.  There's no way to pass the SSL objects to the =
new process, so I've been trying to close the SSL session and initiate =
a new one on the still-open socket from the worker process.  =

To do this, the listener sends a =
packet to the client when it starts the worker process, telling the =
client to shutdown its SSL.  The listener does the same (calls =
SSL_shutdown then SSL_free).  The worker process sends a packet in =
the clear to the client on the socket (this works), then calls SSL_new, =
BIO_new_socket, SSL_set_bio and SSL_accept.  When the client gets =
the packet in the clear from the worker process, it also calls SSL_new, =
BIO_new_socket, SSL_set_bio and then SSL_connect.  So far so good, =
it all appears to work.

The client can then send messages to =
the server worker process using SSL_write, and the worker receives them =
ok.  However, when the worker sends something to the client, =
nothing comes through - SSL_read fails.

In summary, is there any way of =
closing an SSL session on a socket, then opening a brand new =
one?

regards,
 Andrew

</BODY>
</HTML>
------_=_NextPart_001_01CA3270.38895250--
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org

David Schwartz

unread,

Sep 10, 2009, 8:55:02 PM9/10/09

to

Andrew Sumner wrote:

> Implementing openssl I've hit a wall.
> I need to initiate SSL on the connection in the
> listener, before the client sends the login packet as it
> obviously shouldn't be sent in the clear.

> There's no way to pass the SSL objects to the new process,
> so I've been trying to close the SSL session and initiate a
> new one on the still-open socket from the worker process.

This has ugly potential security problems.

> To do this, the listener sends a packet to the client when
> it starts the worker process, telling the client to shutdown
> its SSL. The listener does the same (calls SSL_shutdown
> then SSL_free). The worker process sends a packet in the clear
> to the client on the socket (this works), then calls SSL_new,
> BIO_new_socket, SSL_set_bio and SSL_accept. When the client
> gets the packet in the clear from the worker process, it also
> calls SSL_new, BIO_new_socket, SSL_set_bio and then SSL_connect.
> So far so good, it all appears to work.

Key word is appears to. This is exceptionally difficult to get right. The
problem is that both ends must precisely agree on where the SSL connection
ends in both half-duplex streams. There are ugly ways to fix this -- for
example, send a bunch of 0xff bytes followed by an 0x00 byte and have each
end ignore all 0xff bytes until it reads an 0x00 byte. But that's really,
really yucky.

The other possible way is to go with BIO pairs and encapsulate all the SSL
data in a higher-level protocol that can separate the two SSL exchanges.
Also very yucky.

Here's another butt ugly solution -- in the first SSL connection, send the
client a random 128-bit string. Then shutdown the connection. Have the
client make a new connection, send the 128-bit string, and then negotiate
SSL. When your server sees the 128-bit string, hand off the connection as an
"already logged in" one. Yuck.

> The client can then send messages to the server worker process
> using SSL_write, and the worker receives them ok. However,
> when the worker sends something to the client, nothing comes
> through - SSL_read fails.

> In summary, is there any way of closing an SSL session on a socket, then
opening a brand new one?

Yes, but that's usually a really bad solution. Here's a better one:
Implement SSL in a separate process that accepts SSL connections from
clients and makes local direct connections to the server. Continue to proxy
the data. You can use an off-the-shelf proxy (such as sslproxy), modifying
it only as/if needed. Clean, elegant, sensible. Alternatively, have the
server process create a bidirectional socketpair and itself act as an SSL
proxy. Hand the worker the plaintext end of that pair.

DS

Andrew Sumner

unread,

Sep 10, 2009, 10:22:36 PM9/10/09

to

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01CA327D.2F61CD80
Content-Type: text/plain;
charset="iso-8859-1"

Hi David,

Thanks, you've saved me from tons of frustration and wasted time chasing an
unworkable solution. The proxy idea sounds very promising, I'll check it
out.

cheers
Andrew

------_=_NextPart_001_01CA327D.2F61CD80
Content-Type: text/html;
charset="iso-8859-1"

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>

<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2653.12">
<TITLE>RE: How to re-use a socket with a new SSL session?</TITLE>
</HEAD>
<BODY>

Hi David,

Thanks, you've saved me from tons of frustration and wasted time chasing an 
 unworkable solution.  The proxy idea sounds very promising, I'll check it out.

cheers
 Andrew

</BODY>
</HTML>
------_=_NextPart_001_01CA327D.2F61CD80--

Victor Duchovni

unread,

Sep 10, 2009, 10:33:48 PM9/10/09

to

On Fri, Sep 11, 2009 at 11:14:18AM +1000, Andrew Sumner wrote:

> Hi David,
>
> Thanks, you've saved me from tons of frustration and wasted time chasing an
> unworkable solution. The proxy idea sounds very promising, I'll check it
> out.

You should be able to cleanly shut-down SSL on both sides, and
resume in a new process, provided the application protocol has
a clean session termination phase.

For example, implementing an application level "STOPTLS" verb that
the initiator may request and the responder must confirm, after which
both sides tear down TLS. The connection stays open, and the initiator
may follow-up with a "client HELLO" to resume SSL.

--
Viktor.

Andrew Sumner

unread,

Sep 10, 2009, 10:51:47 PM9/10/09

to

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01CA328A.BB06C860
Content-Type: text/plain;
charset="iso-8859-1"

Hi Victor,

> You should be able to cleanly shut-down SSL on both sides, and
> resume in a new process, provided the application protocol has
> a clean session termination phase.

> For example, implementing an application level "STOPTLS" verb that
> the initiator may request and the responder must confirm, after which
> both sides tear down TLS. The connection stays open, and the initiator
> may follow-up with a "client HELLO" to resume SSL.

Victor, you've just described exactly what I've been trying to do. A clean
shutdown on both sides, socket connection left open, then a "client HELLO"
after which both sides initiate SSL again.

I just can't seem to find a method of doing it that actually works.

cheers
Andrew

------_=_NextPart_001_01CA328A.BB06C860
Content-Type: text/html;
charset="iso-8859-1"

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2653.12">
<TITLE>RE: How to re-use a socket with a new SSL session?</TITLE>
</HEAD>
<BODY>

Hi Victor,

> You should be able to cleanly shut-down SSL on both sides, and
 > resume in a new process, provided the application protocol has
 > a clean session termination phase.

> For example, implementing an application level "STOPTLS" verb that
 > the initiator may request and the responder must confirm, after which
 > both sides tear down TLS. The connection stays open, and the initiator
 > may follow-up with a "client HELLO" to resume SSL.

Victor, you've just described exactly what I've been trying to do. A clean
 shutdown on both sides, socket connection left open, then a "client HELLO"
 after which both sides initiate SSL again.

I just can't seem to find a method of doing it that actually works.

cheers
 Andrew

</BODY>
</HTML>
------_=_NextPart_001_01CA328A.BB06C860--

Victor Duchovni

unread,

Sep 11, 2009, 10:09:49 AM9/11/09

to

On Fri, Sep 11, 2009 at 12:51:15PM +1000, Andrew Sumner wrote:

> Victor, you've just described exactly what I've been trying to do. A clean
> shutdown on both sides, socket connection left open, then a "client HELLO"
> after which both sides initiate SSL again.
>
> I just can't seem to find a method of doing it that actually works.

Initially, does your client build an SSL connection over an already
(TCP) established connection passed to it as a file descriptor?

Initially, does your server accept an SSL connection over an already
(TCP) established connection passed to it as a file descriptor?

Do both parties call SSL_shutdown() at least once, and a second time if
the initial return value is zero?

Do you use an external session cache (store serialized SSL_SESSION
objects) in a store accessible to multiple processes via IPC or an
appropriate shared resource with robust locking? If so, the re-connect
will be efficient, if you pre-load the saved session into the client
SSL state.

--
Viktor.

Darryl Miles

unread,

Sep 13, 2009, 10:10:18 PM9/13/09

to

Andrew Sumner wrote:
> The client can then send messages to the server worker process using
> SSL_write, and the worker receives them ok. However, when the worker
> sends something to the client, nothing comes through - SSL_read fails.
>
> In summary, is there any way of closing an SSL session on a socket, then
> opening a brand new one?

If hope I understand the problem correctly (from your original email and
the nature of the responses of others on the issue).

You can keep an underlying TCP connection and reuse it for different SSL
sessions (where each SSL session has exclusive use of the underlying TCP
socket while that SSL session is active/open). So TCP socket stays open
a long time and sequentially an SSL session is created over it, used,
then closed, then the socket becomes idle again and can be used to open
a new SSL session. Rinse, repeat.

You can't take an already open SSL session and hand it to another
process. Technically there is no reason why this can't be done
(theoretically) if OpenSSL could freeze the entire active SSL handle
state to a binary data blob, which was then conveyed to another process
to be thawed. Seems like a lot of work for someone to implement and
maybe not many users in need of it.

You may be able to take an already open TCP socket handle and hand it to
another process. Unix has a file descriptor passing mechanism to
achieve this, Windows I'm not so sure on and would like to know myself
if this is possible maybe tweaking handle ACLs and such ?

The underlying application protocol needs to either be a client/server
relationship (where the client is always in control of what happens
next, such as the STARTTLS/STOPTLS case).

You just need to understand how SSL_shutdown() works, you need to
perform a full shutdown where both ends acknowledge the session is over
in both directions and all further data until further notice is cleartext.

It may also be helpful to call SSL_set_read_ahead(ssl, 0) to disable
readahead optimization just before you issue the SSL_shutdown(ssl). So
that the only bytes read out of the socket are SSL protocol and any
bytes behind are ready to be picked up by your application (and not
accidentally eaten by OpenSSL library trying to bulk read data in to
optimize SSL_read() performance).

Of course if there is any kind of SSL protocol error (which should be a
rare occurrence) the easiest way out is to drop the socket and cleanup
your SSL library handles. Rather than try to have some way of
re-synchronize the application.

The SSL_shutdown() recently had a patch to fix a problem with
non-blocking I/O and some scenarios I had been able to observe where
data was not correctly flushed to the kernel/lower-layer from OpenSSL
during shutdown handling when trying to re-used the underlying data
channel for a future SSL session.

The SSL_shutdown() command once issued effectively closes the write half
of the bi-directional SSL stream, the other end's application should get
notification of this and issue its own SSL_shutdown() which closes the
write half of the bi-direction SSL stream (from the far end), then once
your local end get notification of receiving this (by SSL_shutdown(ssl)
returning 0) you know that it's safe to disconnect the socket/fd from
OpenSSL handle and cleanup SSL structure (while hanging onto the still
open socket/fd).

During your wait after having called SSL_shutdown(ssl) at least once and
you now wait for SSL_shutdown() to return 0. You should also attempt to
SSL_read() and discard any data read in (unless you really do have a use
for it), it maybe possible for the far end send application data and if
you don't do SSL_read() you will never see the "SSL shutdown packet"
that the SSL_shutdown() function is waiting for to be able to return 0.

This may happen with regular SMTP servers where the server end had a
connection idle timeout at exactly the same moment the client end issued
a SSL_shutdown(). Many SMTP server send some kind of "XXX Shutdown due
to idle timeout." message just before closing the socket. For many
client-server protocols however it is not usually possible for the
server end to spontaneously send some application data because the
client is usually in 100% control of what the server does (I suppose
IMAP4 can!).

Hope this helps,

Darryl

Andrew Sumner

unread,

Sep 13, 2009, 11:12:52 PM9/13/09

to

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01CA34E0.6F66B0C0
Content-Type: text/plain;
charset="iso-8859-1"

Hi Viktor,

> Initially, does your client build an SSL connection over an already
> (TCP) established connection passed to it as a file descriptor?

yes.

> Initially, does your server accept an SSL connection over an already
> (TCP) established connection passed to it as a file descriptor?

yes.

> Do both parties call SSL_shutdown() at least once, and a second time if
> the initial return value is zero?

yes.

> Do you use an external session cache (store serialized SSL_SESSION
> objects) in a store accessible to multiple processes via IPC or an
> appropriate shared resource with robust locking? If so, the re-connect
> will be efficient, if you pre-load the saved session into the client
> SSL state.

No, for various reasons shared memory is not an option for this application.
The socket ID is passed from the listener to the worker process on the
server
via a pipe.

Here's some over-simplified snippets of the relevant parts of the code
(assume
undeclared/uninitialized variables have been defined and setup correctly):-

CLIENT:

/* connect socket */
if (-1 == connect( iSock, (struct sockaddr *)&SA, sizeof(SA) )
exit_error("couldn't connect etc");

/* test message to/from the server, no encryption */
send( iSock, szSendBuf1, 127, 0 );
recv( iSock, szRecvBuf1, 127, 0);

if (strncmp(szSendBuf1, szRecvBuf1, 127))
exit_error("problem communicating with server");

/* setup SSL session */
if (!(pBio = BIO_new_socket( iSock, BIO_NOCLOSE )))
exit_error("failed to create BIO");

if (!(pSSL = SSL_new( pCTX )))
exit_error("failed to create SSL");

SSL_set_bio( pSSL, pBIO, pBIO );

if (SSL_connect( pSSL ) <= 0)
exit_error("SSL_connect failed");

/* send test message with SSL encryption */
if (SSL_write( pSSL, szSendBuf2, strlen(szSendBuf2) ) <= 0)
exit_error("SSL_write 1 failed");

if (SSL_read( pSSL, szRecvBuf2, sizeof(sRecvBuf2) ) <= 0)
exit_error("SSL_read 1 failed");

if (strncmp(szSendBuf2, szRecvBuf2, strlen(szSendBuf2))
exit_error("SSL write/read test failed");

/* close down connection */
if (!SSL_shutdown( pSSL ))
SSL_shutdown( pSSL );
SSL_free( pSSL );

/* wait for server to tell us its ok to start ssl again */
recv( iSock, szRecvBuf3, 127, 0 );
if (strncmp( szRecvBuf3, RESTART_SSL_MESSAGE, 127 ))
exit_error("Unexpected message from server");

/* restart SSL */
if (!(pBio = BIO_new_socket( iSock, BIO_NOCLOSE )))
exit_error("failed to create BIO");

if (!(pSSL = SSL_new( pCTX )))
exit_error("failed to create SSL");

SSL_set_bio( pSSL, pBIO, pBIO );

if (SSL_connect( pSSL ) <= 0)
exit_error("SSL_connect failed");

/* send test message with SSL encryption */
if (SSL_write( pSSL, szSendBuf2, strlen(szSendBuf2) ) <= 0)
exit_error("SSL_write 1 failed");

if (SSL_read( pSSL, szRecvBuf2, sizeof(sRecvBuf2) ) <= 0) /* THIS READ IS
FAILING */
exit_error("SSL_read 1 failed");

if (strncmp(szSendBuf2, szRecvBuf2, strlen(szSendBuf2))
exit_error("SSL write/read test failed");

-- end snippet.

SERVER - LISTENER SNIPPET

if (-1 == listen( iListenSock, 5 ))
exit_error("listen failed");

iSock = accept( iListenSock, 0, 0 );

/* wait for unencrypted test message from client */
recv( iSock, szBuf, 127, 0 );
send( iSock, szBuf, strlen(szBuf), 0 );

if (!(pSSL = SSL_new( pCTX )))
exit_error("failed to create SSL");

SSL_set_bio( pSSL, pBIO, pBIO );

if (SSL_accept( pSSL ) <= 0)
exit_error("SSL_connect failed");

/* wait for test message from client */
if (SSL_read( pSSL, szBuf, sizeof(szBuf) ) <= 0)
exit_error("SSL_read failed");

if (SSL_write( pSSL, szBuf, strlen(szBuf) ) <= 0)
exit_error("SSL_write failed");

/* shutdown SSL */
if (!(SSL_shutdown( pSSL ))
SSL_shutdown( pSSL );
SSL_free( pSSL );

/* create new worker process (not shown here) */

/* hand off socket ID to worker (not shown) */

-- end snippet

SERVER - WORKER SNIPPET

/* send message in the clear telling client to restart SSL */
send( iSock, RESTART_SSL_MESSAGE, strlen(RESTART_SSL_MESSAGE), 0 );

/* start SSL */
if (!(pSSL = SSL_new( pCTX )))
exit_error("failed to create SSL");

SSL_set_bio( pSSL, pBIO, pBIO );

if (SSL_accept( pSSL ) <= 0)
exit_error("SSL_connect failed");

/* wait for test message from client */
if (SSL_read( pSSL, szBuf, sizeof(szBuf) ) <= 0)
exit_error("SSL_read failed");

if (SSL_write( pSSL, szBuf, strlen(szBuf) ) <= 0)
exit_error("SSL_write failed");

-- end snippet

I've tried it with a single process on the server end (ie eliminating the
socket
handoff to another process, just starting, stopping, re-starting SSL on the
socket).

The re-started SSL appears to work - the SSL_connect and SSL_accept both
succeed, the
server receives the string sent to it by the client, but when it sends it
back the
client just doesn't receive it - its SSL_read blocks indefinitely until the
server
closes the connection. I can send additional SSL_writes from the server -
they all
report success, but the client will hang on that first SSL_read after
restarting SSL.

It really feels like I'm missing something incredibly simple, but for the
life of me
I can't see what it is. Unless, of course, what I'm trying to do isn't
supported by
openssl.

cheers
Andrew

------_=_NextPart_001_01CA34E0.6F66B0C0
Content-Type: text/html;
charset="iso-8859-1"

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2653.12">
<TITLE>RE: How to re-use a socket with a new SSL session?</TITLE>
</HEAD>
<BODY>

Hi Viktor,

> Initially, does your client build an SSL connection over an already
 > (TCP) established connection passed to it as a file descriptor?

yes.

> Initially, does your server accept an SSL connection over an already
 > (TCP) established connection passed to it as a file descriptor?

yes.

> Do both parties call SSL_shutdown() at least once, and a second time if
 > the initial return value is zero?

yes.

> Do you use an external session cache (store serialized SSL_SESSION
 > objects) in a store accessible to multiple processes via IPC or an
 > appropriate shared resource with robust locking? If so, the re-connect
 > will be efficient, if you pre-load the saved session into the client
 > SSL state.

No, for various reasons shared memory is not an option for this application.
 The socket ID is passed from the listener to the worker process on the server
 via a pipe.

Here's some over-simplified snippets of the relevant parts of the code (assume 
 undeclared/uninitialized variables have been defined and setup correctly):-

CLIENT:

/* connect socket */
   if (-1 == connect( iSock, (struct sockaddr *)&SA, sizeof(SA) )
     exit_error("couldn't connect etc");
   
   /* test message to/from the server, no encryption */
   send( iSock, szSendBuf1, 127, 0 );
   recv( iSock, szRecvBuf1, 127, 0);
   
   if (strncmp(szSendBuf1, szRecvBuf1, 127))
     exit_error("problem communicating with server");
   
   /* setup SSL session */
   if (!(pBio = BIO_new_socket( iSock, BIO_NOCLOSE )))
     exit_error("failed to create BIO");

if (!(pSSL = SSL_new( pCTX )))
     exit_error("failed to create SSL");

SSL_set_bio( pSSL, pBIO, pBIO );

if (SSL_connect( pSSL ) <= 0)
     exit_error("SSL_connect failed");

/* send test message with SSL encryption */
   if (SSL_write( pSSL, szSendBuf2, strlen(szSendBuf2) ) <= 0)
     exit_error("SSL_write 1 failed");

if (SSL_read( pSSL, szRecvBuf2, sizeof(sRecvBuf2) ) <= 0)
     exit_error("SSL_read 1 failed");
       
   if (strncmp(szSendBuf2, szRecvBuf2, strlen(szSendBuf2))
     exit_error("SSL write/read test failed");

/* close down connection */
   if (!SSL_shutdown( pSSL ))
     SSL_shutdown( pSSL );
   SSL_free( pSSL );

/* wait for server to tell us its ok to start ssl again */
   recv( iSock, szRecvBuf3, 127, 0 );
   if (strncmp( szRecvBuf3, RESTART_SSL_MESSAGE, 127 ))
     exit_error("Unexpected message from server");

/* restart SSL */
   if (!(pBio = BIO_new_socket( iSock, BIO_NOCLOSE )))
     exit_error("failed to create BIO");

if (!(pSSL = SSL_new( pCTX )))
     exit_error("failed to create SSL");

SSL_set_bio( pSSL, pBIO, pBIO );

if (SSL_connect( pSSL ) <= 0)
     exit_error("SSL_connect failed");

/* send test message with SSL encryption */
   if (SSL_write( pSSL, szSendBuf2, strlen(szSendBuf2) ) <= 0)
     exit_error("SSL_write 1 failed");

if (SSL_read( pSSL, szRecvBuf2, sizeof(sRecvBuf2) ) <= 0) /* THIS READ IS FAILING */
     exit_error("SSL_read 1 failed");
       
   if (strncmp(szSendBuf2, szRecvBuf2, strlen(szSendBuf2))
     exit_error("SSL write/read test failed");

-- end snippet.

SERVER - LISTENER SNIPPET

if (-1 == listen( iListenSock, 5 ))
    exit_error("listen failed");

iSock = accept( iListenSock, 0, 0 );

/* wait for unencrypted test message from client */
   recv( iSock, szBuf, 127, 0 );
   send( iSock, szBuf, strlen(szBuf), 0 );

if (!(pSSL = SSL_new( pCTX )))
     exit_error("failed to create SSL");

SSL_set_bio( pSSL, pBIO, pBIO );

if (SSL_accept( pSSL ) <= 0)
     exit_error("SSL_connect failed");

/* wait for test message from client */
   if (SSL_read( pSSL, szBuf, sizeof(szBuf) ) <= 0)
     exit_error("SSL_read failed");

if (SSL_write( pSSL, szBuf, strlen(szBuf) ) <= 0)
     exit_error("SSL_write failed");

/* shutdown SSL */
   if (!(SSL_shutdown( pSSL ))
     SSL_shutdown( pSSL );
   SSL_free( pSSL );

/* create new worker process (not shown here) */

/* hand off socket ID to worker (not shown) */
   
 -- end snippet

SERVER - WORKER SNIPPET

/* send message in the clear telling client to restart SSL */
   send( iSock, RESTART_SSL_MESSAGE, strlen(RESTART_SSL_MESSAGE), 0 );

/* start SSL */
   if (!(pSSL = SSL_new( pCTX )))
     exit_error("failed to create SSL");

SSL_set_bio( pSSL, pBIO, pBIO );

if (SSL_accept( pSSL ) <= 0)
     exit_error("SSL_connect failed");

/* wait for test message from client */
   if (SSL_read( pSSL, szBuf, sizeof(szBuf) ) <= 0)
     exit_error("SSL_read failed");

if (SSL_write( pSSL, szBuf, strlen(szBuf) ) <= 0)
     exit_error("SSL_write failed");

-- end snippet

I've tried it with a single process on the server end (ie eliminating the socket
 handoff to another process, just starting, stopping, re-starting SSL on the socket).

The re-started SSL appears to work - the SSL_connect and SSL_accept both succeed, the
 server receives the string sent to it by the client, but when it sends it back the
 client just doesn't receive it - its SSL_read blocks indefinitely until the server
 closes the connection.  I can send additional SSL_writes from the server - they all
 report success, but the client will hang on that first SSL_read after restarting SSL.

It really feels like I'm missing something incredibly simple, but for the life of me
 I can't see what it is.  Unless, of course, what I'm trying to do isn't supported by
 openssl.

cheers
 Andrew

</BODY>
</HTML>
------_=_NextPart_001_01CA34E0.6F66B0C0--

Andrew Sumner

unread,

Sep 14, 2009, 5:12:50 AM9/14/09

to

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01CA34EE.592C0B80
Content-Type: text/plain;
charset="iso-8859-1"

Darryl,

Thanks for your detailed suggestions, especially:

> It may also be helpful to call SSL_set_read_ahead(ssl, 0) to disable
> readahead optimization just before you issue the SSL_shutdown(ssl).

For some reason, adding that line before the shutdowns made all the
difference - it now works perfectly.

Thanks also to Viktor and David.

cheers,
Andrew

------_=_NextPart_001_01CA34EE.592C0B80
Content-Type: text/html;
charset="iso-8859-1"

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2653.12">
<TITLE>RE: How to re-use a socket with a new SSL session?</TITLE>
</HEAD>
<BODY>

Darryl,

Thanks for your detailed suggestions, especially:

> It may also be helpful to call SSL_set_read_ahead(ssl, 0) to disable 
 > readahead optimization just before you issue the SSL_shutdown(ssl).

For some reason, adding that line before the shutdowns made all the 
 difference - it now works perfectly.

Thanks also to Viktor and David.

cheers,

Andrew

</BODY>
</HTML>
------_=_NextPart_001_01CA34EE.592C0B80--

Victor Duchovni

unread,

Sep 14, 2009, 10:28:04 AM9/14/09

to

On Mon, Sep 14, 2009 at 01:49:23PM +1000, Andrew Sumner wrote:

> Darryl,
>
> Thanks for your detailed suggestions, especially:
>
> > It may also be helpful to call SSL_set_read_ahead(ssl, 0) to disable
> > readahead optimization just before you issue the SSL_shutdown(ssl).
>
> For some reason, adding that line before the shutdowns made all the
> difference - it now works perfectly.

With read-ahead, parts of the subsequent traffic may be consumed as part
of the SSL_shutdown(). It may be possible to do this with read-ahead
enabled via a half-duplex shutdown:

Client. Server.
-----------------------------------
STOPTLS message (assuming server can't refuse)
SSL_read() until
(SSL_get_shutdown(s)
& SSL_RECEIVED_SHUTDOWN)
SSL_shutdown() == 0
SSL_shutdown() == 1
SSL_read() until
(SSL_get_shutdown(s)
& SSL_RECEIVED_SHUTDOWN)
(assuming SSL_read() works
after SSL_SENT_SHUTDOWN).

SSL_shutdown() == 1

... at any time later ...

[optional STARTTLS message? possible STARTTLS response]
SSL_connect() SSL_accept()

Above, the server only consumes the client's "close notify", because
the client does not start the re-connect phase until it has consumed
the server's "close notify", and the client is the next one to send.

A plain-text STARTTLS and OK response may be useful to make sure both
sides are ready and willing to resume if the session is not resumed
immediately. An external session cache (need not be "shared memory")
is useful in a mature implementation of multi-session connections.

--
Viktor.

handshak3

unread,

Dec 2, 2011, 5:56:51 AM12/2/11

to

I am facing the same problem for resuing an existing socket for SSL conection between a client and server.

I am using openssl-0.98h

Here is the high level code of the function which i am trying to modify

int SSLeaySocketImpl::sslify(bool client,OpenSSLContext*_ssl_context,bool isRecursed)
{
//some code

if (_ssl_context)
{
if(ssl)
{
SSL_free(ssl);
}
ssl_context = _ssl_context;
ssl = ssl_context->get_connection();
}
if(SSL_set_fd(ssl,fd) == 0) //not okay
{
//To DO: close fd
}// if

// ToDO: Get current time

err=SSL_ERROR_WANT_READ; //something non-zero
do
{
//Check for timeout
if ( timercmp(&current_time_struct, &connect_timeout_struct, >) )
{
this->close();
error=ConnectError::instance();
return -ETIMEDOUT; // Using negative value to differentiate SSL error from "plain" one
}//id

ret = client ? SSL_connect( ssl ) : SSL_accept(ssl);
err=SSL_get_error( ssl, ret );
switch(err)
{
case SSL_ERROR_NONE:
break;
case SSL_ERROR_ZERO_RETURN:
// The SSL session has been terminated, but the socket may still be connected
//close socket fd
return -ECONNRESET; // Using negative value to differentiate SSL error from "plain" one
break;
case SSL_ERROR_WANT_READ:
case SSL_ERROR_WANT_WRITE:
fprintf(stderr,"%s - Debugging note(4)\n",client?"CLIENT":"SERVER");
FD_ZERO(&fdvar); //initialize
FD_SET(fd, &fdvar); //turn on file descriptor
tv_subtract(connect_timeout_struct, current_time_struct, select_timeout_struct);
// check if it is available for reading or writing
if(err==SSL_ERROR_WANT_READ)
{
err_select=select(fd+1,&fdvar,NULL,NULL,&select_timeout_struct);
}
else
{
err_select=select(fd+1,NULL,&fdvar,NULL,&select_timeout_struct);
}//if
//is there a select error?
if(err_select==-1)
{
return -errno; // Using negative value to differentiate SSL error from "plain" one
}
else if(err_select==0) //timeout too quickly
{
continue; //keep spinning...
}//if
if(!FD_ISSET(fd, &fdvar))
{
//the socket wasn't ready....failed
continue; // wrong socket woke me up
} //
break;
case SSL_ERROR_SYSCALL:
fprintf(stderr,"%s - Debugging note(5)\n",client?"CLIENT":"SERVER");
// check if the remote side closed the connection
if(ret == 0 || (ret == -1 && errno == ECONNRESET))
return -ECONNRESET; // Using negative value to differentiate SSL error from "plain" one
// else fall through to default
default:
//this->close();
fprintf(stderr,"%s - Debugging note(6)\n",client?"CLIENT":"SERVER");
if (client) {
if (isRecursed)
{
this->close();
OPENSSL_THROW_STACK("SSL_connect failed");
}
int err;
err = ERR_GET_REASON(ERR_peek_error());
if (err == SSL_R_CERTIFICATE_VERIFY_FAILED)
{
// Reset the SSL_CTX to look for pem file
if (ptr && ptr->reset_CA_certs_if_possible());
{
if (!SSL_shutdown(ssl))
SSL_shutdown(ssl);
fprintf(stderr,"resetting the certs \n");
return sslify(true,ptr,true);
}

}

} else {
if (isRecursed)
{
this->close();
OPENSSL_THROW_STACK("SSL_connect failed");
}
int err;
err = ERR_GET_REASON(ERR_peek_error());
if (err == SSL_R_TLSV1_ALERT_UNKNOWN_CA || err == SSL_R_SSLV3_ALERT_BAD_CERTIFICATE)
{
if (!SSL_shutdown(ssl))
SSL_shutdown(ssl);
fprintf(stderr,"resetting the server\n");
return sslify(false,ssl_context,true);
}
}
}//switch
}while (err!=SSL_ERROR_NONE);
// Some other code for house keeping
}

ssldump proves the handshake is initiated, but client immediately sends an unexpected message alert to the server.

here is the ssldump:

New TCP connection #1: localhost.localdomain(10289) <-> localhost.localdomain(23037)
1 1 0.0001 (0.0001) C>SV3.1(67) Handshake
ClientHello
Version 3.1
random[32]=
4e d8 a4 ee fd 9e be ec 58 1b 3e 16 fd a6 b2 e9
a5 fe 65 34 79 e4 65 03 81 d4 f2 b7 7a a8 ac 55
cipher suites
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_IDEA_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
compression methods
NULL
1 2 0.0005 (0.0003) S>CV3.1(74) Handshake
ServerHello
Version 3.1
random[32]=
4e d8 a4 ee a5 b1 69 b8 9a a1 e4 01 07 96 e4 a1
31 dc 7d b7 e3 01 48 4e d1 f1 21 c6 58 4e 74 ff
session_id[32]=
81 98 54 07 4c 14 b2 37 d6 13 05 26 05 5f bc dd
46 5c d6 aa 9e ee 20 22 3e 13 7b 98 13 d0 30 ee
cipherSuite TLS_RSA_WITH_3DES_EDE_CBC_SHA
compressionMethod NULL
1 3 0.0005 (0.0000) S>CV3.1(1278) Handshake
Certificate
1 4 0.0005 (0.0000) S>CV3.1(160) Handshake
CertificateRequest
certificate_types rsa_sign
certificate_types dss_sign
certificate_types unknown value
certificate_authority
31 1b 30 19 06 03 55 04 0a 13 12 50 61 79 50 61
6c 20 70 76 6f 69 63 75 20 74 65 73 74 31 0e 30
0c 06 03 55 04 0b 13 05 49 6e 66 72 61 31 0f 30
0d 06 03 55 04 03 13 06 70 76 6f 69 63 75 31 20
30 1e 06 09 2a 86 48 86 f7 0d 01 09 01 16 11 70
76 6f 69 63 75 40 70 61 79 70 61 6c 2e 63 6f 6d
ServerHelloDone
1 5 0.0008 (0.0003) C>SV3.1(2) Alert
level fatal
value unknown_ca
1 6 0.0130 (0.0121) C>SV3.1(67) Handshake
ClientHello
Version 3.1
random[32]=
4e d8 a4 ee 61 a7 5b 6c 64 89 61 fb cf 0a a2 85
65 04 40 47 23 0d 62 35 b4 59 b5 bb 2e b7 a8 22
cipher suites
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_IDEA_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
compression methods
NULL
1 7 0.0130 (0.0000) C>SV3.1(2) Alert
level fatal
value unexpected_message
1 0.0130 (0.0000) C>S TCP FIN
1 8 0.0131 (0.0001) S>CV3.1(74) Handshake
ServerHello
Version 3.1
random[32]=
4e d8 a4 ee 42 58 23 7f 51 e4 c1 10 b6 9d f0 7e
25 bd 8a 95 3c e2 dd 0e 4d 3e 61 38 33 2b a9 58
session_id[32]=
5b ca 68 de 2f 18 cb 03 df 8b 23 b9 b0 19 87 7e
02 72 f7 bb e2 a5 21 6a 7e c8 a8 38 12 83 d8 fe
cipherSuite TLS_RSA_WITH_3DES_EDE_CBC_SHA
compressionMethod NULL
1 9 0.0131 (0.0000) S>CV3.1(1278) Handshake
Certificate
1 10 0.0131 (0.0000) S>CV3.1(160) Handshake
CertificateRequest
certificate_types rsa_sign
certificate_types dss_sign
certificate_types unknown value
certificate_authority
30 81 8d 31 0b 30 09 06 03 55 04 06 13 02 55 53
30 1e 06 09 2a 86 48 86 f7 0d 01 09 01 16 11 70
76 6f 69 63 75 40 70 61 79 70 61 6c 2e 63 6f 6d
ServerHelloDone

Please assist me to get rid of this;
1 7 0.0130 (0.0000) C>SV3.1(2) Alert
level fatal
value unexpected_message

Looking forward to a reply

regards