SSL Cert memory leak?

[ccote]

I'm using Civetweb in a windows application built in Visual Studio 2013. I'm seeing a memory leak dump when the application exits if I provide a SSL certificate path in the options list to the CivetServer's constructor.

I can reproduce this in the sample provided in examples\embedded_cpp\embedded_cpp.cpp if I make the following changes:

// Enable memory leak debugging

#include <afx.h>

#ifdef _DEBUG

#define new DEBUG_NEW

#undef THIS_FILE

static char THIS_FILE[] = __FILE__;

#endif

int main(int argc, char *argv[])

{

const char * options[] = { "document_root", DOCUMENT_ROOT,

"listening_ports", "8081,8083s",

"ssl_certificate", "c:\\server.pem", 0   // Provide an SSL cert

};

    /* CivetServer instance is now created on the heap */

    CivetServer* server = new CivetServer (options);

    ExampleHandler h_ex;

    server->addHandler(EXAMPLE_URI, h_ex);

    ExitHandler h_exit;

    server->addHandler(EXIT_URI, h_exit);

    AHandler h_a;

    server->addHandler("/a", h_a);

    ABHandler h_ab;

    server->addHandler("/a/b", h_ab);

    FooHandler h_foo;

    server->addHandler("**.foo$", h_foo);

    printf("Browse files at http://localhost:%s/\n", PORT);

    printf("Run example at http://localhost:%s%s\n", PORT, EXAMPLE_URI);

    printf("Exit at http://localhost:%s%s\n", PORT, EXIT_URI);

    while (!exitNow) {

#ifdef _WIN32

        Sleep(1000);

#else

        sleep(1);

#endif

    }

    printf("Bye!\n");

delete server;

    return 0;

}

Note that I do not see the leak of the instance of CivetServer is created on the stack. Any ideas as to why this is the case? Do I need to clean up the certificate some how before destroying the CivetServer?

[bel]

From my understanding, using new and delete is just the same as putting the class on the stack and returning from the function.
Do you have more details - what does VS2013 say exactly?
Does it show an error if you do not put the object on the stack in main, but in another function you call from there?

[ccote]

The memory dump is far to large to post here. It makes no difference if I create the object in main or in another function.

[ccote]

Also, this is not simply a case of not deleting the CivetServer instance. If I fail to do that, I see a much larger memory leak. Also, I do not see the leak if I do not pass the SSL cert into the CivetServer options.

[bel]

How large is the leak?
If you create two civetweb instances, and then delete both, does the size of the leak double?
If you create an instance, delete it, create another one, delete it, does the size of the leak double?

It could be a leak with the initialization of OpenSSL as well, or in the handling of the thread local storage objects for OpenSSL.

I do not have Visual Studio 2013 here, so I can not reproduce it myself right now - I can access to a PC with Visual Studio 2013 later, but this takes some days.

[ccote]

It's hard to say exactly what the memory leak size is. Visual Studio reports each object it detects as leaked separately. There are over 2000 items it says are leaked that range in size from 12 bytes to 8192 bytes. Most are 12 or 16 bytes. Rough estimate would be 42k.

I tried the first case you asked for, and it crashes when the second instance is deleted, when destroying the SSL mutexes. 

In the second case, the total leak is the same as if only one instance was created.

Thanks for looking into this.

[bel]

>> If you create an instance, delete it, create another one, delete it, does the size of the leak double?

> In the second case, the total leak is the same as if only one instance was created.

The first time civetweb requires openSSL, it loads the ssl libraries and initializes them.
It never unloads these libraries, so this could be the memory leak shown by Visual Studio.
It is not a "real" leak but just some DLL that is initialized only on demand.
It does not need to do these initialization again, so the memory demand will not grow.

>> If you create two civetweb instances, and then delete both, does the size of the leak double?

> I tried the first case you asked for, and it crashes when the second instance is deleted, when destroying the SSL mutexes.

A crash must not happen here! So this is something I have to look at.
Did you use different listening ports for these two instances? Did both mg_start commands succeed?

One of the reasons SSL is only loaded "on demand" and never unloaded is, there is not "mg_init_library" function the user has to call before any other request. Adding such a function would be a change in the API - but it would make some things easier.

[ccote]

Hi Bel, have you had a chance to test this with Visual Studio? 

On your question below, I was using different listening ports for the two instances of CivetServer. Both mg_start commands succeeded and I could hit URLs on both ports.

One further note, I have tested this with several versions of OpenSSL. The version we are using is 1.0.2.d. I only see the memory dump when I'm using the debug SSL dlls.

[bel]

Not with VS 2013 yet.
I did not get this message with VS2010, but I did not know this only occurs with the debug version of OpenSSL.
I never used the debug version and actually do not really want to - I take care about CivetWeb, but I'm not OpenSSL developer.

Are you sure the leak is within CivetWeb and not within the debug version of OpenSSL? Maybe the debug version collects some log data?

[ccote]

I doubt it's a leak in the debug build of OpenSSL. We use OpenSSL in other products. I typically build and debug with the debug version of OpenSSL, and have never seen a leak like this.

[bel]

I was not able to reproduce this with Visual Studio 2013.
But probably I do not have the same openssl DLL you have.
Did you download this debug version from somewhere? Can you provide a link?

[ccote]

We build the OpenSSL libraries. The debug dlls are generated by building the debug configuration of the solution provided in the OpenSSL distribution. You should be able to build that with VS2013. If you can't, I can try and send you the dlls I'm using.

[bel]

The OpenSSL archives do not have a Visual Studio Solution file. It seems they require some Perl and nmake to build them - something I've never used. I'm currently busy with getting civetweb built with the CMake system, so I have to get more familiar with CMake. Currently I can not spent the time to care about yet another build system (nmake).

There are some projects which offer OpenSSL binaries for Linux, but I found only Visual Studio files for V0.9.? OpenSSL.
Which version are you using?
Or do you have a link where I can download a Visual Studio project?

[bel]

Are you using the NO_SSL_DL define?

Could you use call
/* static */ int initialize_ssl(struct mg_context *ctx)
(can be done with ctx == NULL), and check if there is a leak?
Could you then do your test, and check if the leak increased?

[ccote]

Yes, you are correct that you need to user Perl and nmake to build the OpenSSL libraries. Sorry for the confusion. I'm currently using version 1.0.2.d, but I've tested it with several older versions, and I see the same behavior.

I will try what you suggested. I am not using NO_SSL_DL. Can you use that if you're using SSL? Are you suggesting to define that, and then call initialize_ssl() as a test?

Thanks again.

[bel]

Civetweb has a default build with the two defines NO_SSL and NO_SSL_DL not defined.
NO_SSL means disable SSL/TLS completely.
NO_SSL_DL means do not link openSSL dynamically.

If none of these defines is set, CivetWeb will load the openSSL libraries (ssleay32.dll + libeay32.dll for Windows, libssl.so + libcrypto.so for Linux) only when it finds some https port configuration in the config file. It will use LoadLibrary (Win32) or dlopen (Linux) to do this at runtime.

This is not the usual way of using a dynamic library / shared object - usually one adds them as a dependency, and the operating system will load the libraries when it loads the executable. This typical behavior can be reached if NO_SSL_DL (no dynamic linking of SSL) is set.

However, doing this means civetweb will always require these two libraries, even if it should run a configuration which only uses http and no https. That's why it makes sense to keep the default of dynamically loading it when it is used.

However, this behavior may also confuse programs that attempt to detect memory leaks: If there is a https configuration, civetweb will load the openSSL dlls, but it will never unload it using FreeLibrary / dlclose - it will only unload the libraries if the civetweb process is shut down. This is the same behavior, as if you would have used NO_SSL_DL: only if civetweb.exe is closed, the DLLs will be unloaded.

Now if your memory leak detection system thinks "everything loaded with LoadLibrary must be unloaded with FreeLibrary", which is certainly a reasonable approach in many cases, it will consider this way of loading openSSL a memory leak.

I tend to not consider it a leak, since it would not be different if you use NO_SSL_DL - but using NO_SSL_DL has the disadvantage that I either have to have the openSSL libraries even if I never call any function from them for a pure http server, or I would have to have two executables, one with https and another one with NO_SSL set.

Now, it is only an assumption that this might be the memory leak Visual Studio is showing you, and my first approach would be to verify this assumption.

This can be done by defining NO_SSL_DL, and checking if there is still a memory leak.

Another approach would be to call the initialize_ssl (even if NO_SSL_DL is not set) and then uninitialize_ssl, without calling mg_start.
I think this will create some memory leak, at least due to the dll as discussed above.
The interesting question is, if it does create the same memory leak as if you call mg_start and then mg_stop.

[ccote]

I tried the first suggestion - defining NO_SSL_DL. Once I figured out how to statically link OpenSSL in the sample VS project, it runs and works as before, and I still see the memory leak. 

Should I still try the second experiment to call initialize_ssl without calling mg_start?

[bel]

> Should I still try the second experiment to call initialize_ssl without calling mg_start?

Yes, please.

[bel]

Un-initializing OpenSSL without memory leaks seems to be a science in it's own.

Since I got several reports of small memory leaks by several users, I want to concentrate the discussion at this issue:
https://github.com/civetweb/civetweb/issues/263