[ovs-discuss][OVN][SSL] replacing ssl key and certs file at runtime has no effect on ovn-controller connection

[Girish Moodalbail]

In ovn-kubernetes K8s CNI project we use SSL connections between ovn-controller and OVN SB DB. Our goal is to rotate the privateKey/signedCert used by ovn-controller very often. When the rotation occurs, we want ovn-controller to redo the TLS handshake without dropping the TCP connection or without requiring ovn-controller restart.

In ovn-controller code, I see that in the main loop we call update_ssl_config(), which through a series of functions checks if SSL files are modified, and if so, calls into OpenSSL library updating the SSL context. At this point, the expectation is for ovn-controller to restart the TLS handshake so that we are using new SSL keys/certs. However, we don't see this happening.

I ran tcpdump on the ovn-controller side to check for TLS handshake packets, but I didn't see any TLS related packets. With `stream_ssl` module logging set to debug, I don't see any SSL control plane messages in ovn-controller.log.

I also created certs with expiry time of 10mins using OVS-PKI and restarted ovn-controller. My expectation was that after 10mins the SSL connection should error out with certificates already expired. I don't see
that is happening as well. When I run the `ovn-sbctl` command using the expired certs it obviously fails.

Looks to me that once the JSON-RPC session is created between ovn-controller and  OVN SB DB process, then nothing seems to re-trigger the TLS handshake.

Are we missing something?

Regards,
~Girish

[Girish Moodalbail]

It looks to me that the function stream_ssl_set_key_and_cert() in lib/stream-ssl.c is incorrect.

void
stream_ssl_set_key_and_cert(const char *private_key_file,
                            const char *certificate_file)
{
    if (update_ssl_config(&private_key, private_key_file)
        && update_ssl_config(&certificate, certificate_file)) {
        stream_ssl_set_certificate_file__(certificate_file);
        stream_ssl_set_private_key_file__(private_key_file);
    }
}

1. Say, the private key and the corresponding certificate file was replaced on the file system at T0 and T2 respectively.

2. At T1, the ovn-controller code calls update_ssl_config(private_key) and update_ssl_config(certificate_file)

2a: The first call to update_ssl_config(private_key) returns true and the file `mtime` is updated. The second call to  update_ssl_config(certificate_file) returns False

3. At T3, the ovn-controller code calls to update_ssl_config(private_key) will return False, and the modified `certifcate file` will never be picked?

Because of 1 - 3 above, the new files will never be picked by the ovn-controller. What we have found is that if I delete both the files and then copy over the private key and certificate files, then it works. This may be because of how we handle the ENOENT case in update_ssl_config() 

Regards,

~Girish