Thanks for the response!
Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
Charle...@allworx.com | 585.421.5565
________________________________________
From: owner-ope...@openssl.org [owner-ope...@openssl.org] on behalf of Viktor Dukhovni [openss...@dukhovni.org]
Sent: Thursday, November 20, 2014 7:26 PM
To: openss...@openssl.org
Subject: Re: Small memory leak on multithreaded server
On Thu, Nov 20, 2014 at 10:19:32PM +0000, Barbe, Charles wrote:
> I can do any combination of steps 2,3 and 4 above (ie. leave some
> of them out) and I always get the same amount of memory left over
> after I shut down my application. I believe this means that this
> is some sort of global information that OpenSSL is hanging on to
> and not something in my SSL connection structure.
>
> Specifically I get 20 blocks totaling 253 bytes. I have stack
> traces of where each block is allocated but I cannot figure out
> how this memory should be cleaned up. Each of the 20 blocks filter
> down to 1 of 5 root stack traces. The stack traces are:
A fixed amount of memory that is not deallocated and is independent
of the number of operations performed, is NOT a memory leak.
Librariers to allocate memory for the lifetime of the process during
one time initialization or first use of a function. This is normal.
Tracking this down is a waste of time IMHO.
--
Viktor.
void OpensslConnection::cleanup()
{
if(ssl != NULL)
{
if(isConnected)
{
while(SSL_shutdown(ssl) == 0)
;
}
SSL_free(ssl);
ERR_remove_state(0);
ssl = NULL;
}
isConnected = false;
}
And here is the code that runs to shut down my SSL library:
static void
openSslShutdown ()
{
CONF_modules_free();
ERR_remove_state(0);
CONF_modules_unload(1);
ERR_free_strings();
EVP_cleanup();
CRYPTO_cleanup_all_ex_data();
if (opensslLocks != NULL)
{
for(int i = 0; i < CRYPTO_num_locks(); i++)
{
PAL_mutexDestroy (opensslLocks[i]);
}
IST_FREE (opensslLocks);
}
}
Also, I have numerous worker threads handling connections and they all do the following before they exit:
ERR_remove_thread_state(0);
Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
Charle...@allworx.com | 585.421.5565
________________________________________
From: owner-ope...@openssl.org [owner-ope...@openssl.org] on behalf of Jeffrey Walton [nolo...@gmail.com]
Sent: Thursday, November 20, 2014 6:03 PM
To: OpenSSL Users List
Subject: Re: Small memory leak on multithreaded server
> Any help would be appreciated.
This could be one of two problems. First, it could be an issue with
your code and the way you handle cleanup. To help diagnose this,
please show us your cleanup code.
Second, it could be the memory leak from the compression methods. This
is a well known problem dating back a few years that has not been
fixed. See, for example,
http://rt.openssl.org/Ticket/Display.html?id=2561&user=guest&pass=guest.
On Thu, Nov 20, 2014 at 5:19 PM, Barbe, Charles
<Charle...@allworx.com> wrote:
> Hello,
>
> I have noticed a small and consistent memory leak in my multithreaded openssl server and am wondering if somebody can help me figure out what I need to do to free it when my application closes. I am on OpenSSL version 1.0.1j. Here's how I reproduce the leak:
>
> 1) Start up my server
> 2) Load my homepage with Chrome
> 3) Load my homepage with IE
> 4) Load my homepage with Firefox
>
> I can do any combination of steps 2,3 and 4 above (ie. leave some of them out) and I always get the same amount of memory left over after I shut down my application. I believe this means that this is some sort of global information that OpenSSL is hanging on to and not something in my SSL connection structure.
>
> Specifically I get 20 blocks totaling 253 bytes. I have stack traces of where each block is allocated but I cannot figure out how this memory should be cleaned up. Each of the 20 blocks filter down to 1 of 5 root stack traces. The stack traces are:
>
What looks suspicious to me is the calls to "x509v3_cache_extensions" that are in the traces below. This implies to me that openssl is caching something. How do I ask it to clear that cache? If i need to augment the library to do so, where are the pointers to the objects it is caching so I can free it? I tried to trace the code but got lost... i'm sure I can trace it given enough time but thought I'd ask the community first.
I also note that I have no indication of any "left over memory" from my allocation of the certs that I am providing to my SSL_CTX... meaning that I believe I am cleaning my SSL_CTX and associated resources appropriately. The memory in question is definitely allocated by OpenSSL as a result of calling SSL_accept... I just need to figure out how to ask OpenSSL to free it.
Thanks so much for the help.
Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
Charle...@allworx.com | 585.421.5565
________________________________________
From: owner-ope...@openssl.org [owner-ope...@openssl.org] on behalf of Dr. Stephen Henson [st...@openssl.org]
Sent: Thursday, November 20, 2014 9:50 PM
To: openss...@openssl.org
Subject: Re: Small memory leak on multithreaded server
On Thu, Nov 20, 2014, Barbe, Charles wrote:
> Hello,
>
> I have noticed a small and consistent memory leak in my multithreaded openssl server and am wondering if somebody can help me figure out what I need to do to free it when my application closes. I am on OpenSSL version 1.0.1j. Here's how I reproduce the leak:
>
> 1) Start up my server
> 2) Load my homepage with Chrome
> 3) Load my homepage with IE
> 4) Load my homepage with Firefox
>
> I can do any combination of steps 2,3 and 4 above (ie. leave some of them out) and I always get the same amount of memory left over after I shut down my application. I believe this means that this is some sort of global information that OpenSSL is hanging on to and not something in my SSL connection structure.
>
> Specifically I get 20 blocks totaling 253 bytes. I have stack traces of where each block is allocated but I cannot figure out how this memory should be cleaned up. Each of the 20 blocks filter down to 1 of 5 root stack traces. The stack traces are:
>
> Repeated 6 times:
> Any help would be appreciated.
>
Not possible to be sure from that trace but at a guess I'd say a STACK of
certificates hasn't been freed up.
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
Here's the extra code that I forgot to include:
STATUS
SSL_shutDown()
{
// Client
if (clientCtxt != NULL)
{
SSL_CTX_free (clientCtxt);
clientCtxt = NULL;
}
// View
clearCertsVector (getCertCachePointer (SSL_TYPE_VIEW));
if (pViewKeyCache != NULL)
{
EVP_PKEY_free (pViewKeyCache);
pViewKeyCache = NULL;
}
if (viewServerCtxt != NULL)
{
SSL_CTX_free (viewServerCtxt);
viewServerCtxt = NULL;
}
// Web - only if server supports HTTPS
if (supportHTTPS)
{
clearCertsVector (getCertCachePointer (SSL_TYPE_WEB));
if (pPendingWebKeyCache != NULL)
{
EVP_PKEY_free (pPendingWebKeyCache);
pPendingWebKeyCache = NULL;
}
if (pWebKeyCache != NULL)
{
EVP_PKEY_free (pWebKeyCache);
pWebKeyCache = NULL;
}
if (webServerCtxt != NULL)
{
SSL_CTX_free (webServerCtxt);
webServerCtxt = NULL;
}
}
CHECK_911 (BIO_free (out) > 0);
openSslShutdown ();
return SUCCESS;
}
static void
clearCertsVector(CERTS_VECTOR* certs)
{
//Not freeing certs because they will be
// freed when context is destroyed
if(certs != NULL)
{
certs->clear();
}
}
Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
Charle...@allworx.com | 585.421.5565
________________________________________
From: owner-ope...@openssl.org [owner-ope...@openssl.org] on behalf of Dr. Stephen Henson [st...@openssl.org]
Sent: Friday, November 21, 2014 12:26 PM
To: openss...@openssl.org
Subject: Re: Small memory leak on multithreaded server
On Fri, Nov 21, 2014, Barbe, Charles wrote:
I am using SSL_CTX_use_certificate and was possibly under the incorrect assumption that my code did not have to explicitly free the X509* that I pass to that argument if I am subsequently calling SSL_CTX_free on the CTX. In retrospect that doesn't sound correct. I will fix my code to free the X509s as well as the CTX and see if that is my issue.
Thank you everybody for all of the help!
Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
Charle...@allworx.com | 585.421.5565
________________________________________
From: owner-ope...@openssl.org [owner-ope...@openssl.org] on behalf of Dr. Stephen Henson [st...@openssl.org]
Sent: Friday, November 21, 2014 1:40 PM
To: openss...@openssl.org
Subject: Re: Small memory leak on multithreaded server
On Fri, Nov 21, 2014, Barbe, Charles wrote:
> Yes... sorry, forgot to include this part of my shutdown sequence. One thing
> I am noticing is that I do not call X409_free on my certs. I even have a
> comment in my code saying that I am not freeing them because I think they
> will be freed when the SSL_CTX is freed. Is that a correct assumption or
> should I be calling X509 free on them explicitly?
>
If you have an explicit X509 structure and you call SSL_CTX_use_certificate
then the reference count is increased and you have to free up the certificate.
However I'm a bit confused by the output. It *looks* like it is associated
with a certificate verification operation which could be cached certificates
in a store. Do you perform any operations with an X509_STORE structure?
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
This conversation also helped me find some other places where I wasn't properly freeing reference counted OpenSSL structures.
Thanks for the help!