Memory footprint of SSL connection

[michal]

Hi all.

We wrote a SSL server using nss. Clients are connecting to the server
and keep the connection. We want to handle as much connections as
possible. As memory usage grows when clients are connecting to the
server I observed that every connection takes around 200Kb of memory.
This means that our server cannot handle more than 3000-4000 clients
with 1GB RAM. So I was experimenting with 20 clients on server that had
enabled trace-malloc logging. I found out that every connection takes
really around 185kb but 135kb of that memory is somehow related to the
SSL (i.e. in the backtrace was libssl3.so). The most of this memory is
5x sslBuffer (5X 18432 bytes) and 1x sslSocket (11272 bytes).
Backtraces are:

18432 bytes
PR_Malloc
PORT_Alloc
sslBuffer_Grow [sslsecur.c:423]
ssl3_AppendHandshake [ssl3con.c:2863]
ssl3_AppendHandshakeNumber [ssl3con.c:2912]
ssl3_AppendHandshakeHeader [ssl3con.c:2948]
ssl3_SendServerHello [ssl3con.c:6041]
ssl3_SendServerHelloSequence [ssl3con.c:5330]
ssl3_HandleV2ClientHello [ssl3con.c:5979]
ssl2_HandleClientHelloMessage [sslcon.c:3485]
ssl_Do1stHandshake [sslsecur.c:149]
ssl_SecureRecv [sslsecur.c:1032]
ssl_Recv [sslsock.c:1352]
PR_Recv [priometh.c:221]
apFileDescWrapper::Peek(void*, unsigned int, unsigned int*)
[apFileDescWrapper.cpp:251]

18432 bytes
PR_Malloc
PORT_Alloc
sslBuffer_Grow [sslsecur.c:423]
ssl_CreateSecurityInfo [sslsecur.c:777]
ssl_NewSocket [sslsock.c:2122]
ssl_DupSocket [sslsock.c:243]
SSL_ImportFD [sslsock.c:1116]
apSSLServer::AcceptProtocol(apIProtocolServer*, apISocketWrapper*)
[apSSLServer.cpp:356]

18432 bytes
PR_Malloc
PORT_Alloc
sslBuffer_Grow [sslsecur.c:423]
ssl_SaveWriteData [sslsecur.c:451]
4ssl3_SendRecord [ssl3con.c:1985]
ssl3_SendChangeCipherSpecs [ssl3con.c:2440]
ssl3_HandleFinished [ssl3con.c:7474]
ssl3_HandleHandshakeMessage [ssl3con.c:7723]
ssl3_HandleHandshake [ssl3con.c:7791]
ssl3_HandleRecord [ssl3con.c:8054]
ssl3_GatherCompleteHandshake [ssl3gthr.c:206]
ssl_GatherRecord1stHandshake [sslcon.c:1258]
ssl_Do1stHandshake [sslsecur.c:149]
ssl_SecureRecv [sslsecur.c:1032]
ssl_Recv [sslsock.c:1352]
PR_Recv [priometh.c:221]
apFileDescWrapper::Peek(void*, unsigned int, unsigned
int*)[apFileDescWrapper.cpp:251]

18432 bytes
PR_Malloc
PORT_Alloc
sslBuffer_Grow [sslsecur.c:423]
ssl_InitGather [sslgathr.c:437]
ssl_NewSocket [sslsock.c:2125]
ssl_DupSocket [sslsock.c:243]
SSL_ImportFD [sslsock.c:1116]
apSSLServer::AcceptProtocol(apIProtocolServer*, apISocketWrapper*)
[apSSLServer.cpp:356]

18432 bytes
PR_Malloc
PORT_Alloc
sslBuffer_Grow [sslsecur.c:423]
ssl3_GatherData [ssl3gthr.c:149]
ssl3_GatherCompleteHandshake [ssl3gthr.c:195]
ssl_GatherRecord1stHandshake [sslcon.c:1258]
ssl_Do1stHandshake [sslsecur.c:149]
ssl_SecureRecv [sslsecur.c:1032]
ssl_Recv [sslsock.c:1352]
PR_Recv [priometh.c:221]
apFileDescWrapper::Peek(void*, unsigned int, unsigned int*)
[apFileDescWrapper.cpp:251]

11272 bytes
PR_Calloc [prmem.c:474]
PORT_ZAlloc [secport.c:140]
ssl_NewSocket [sslsock.c:2074]
ssl_DupSocket [sslsock.c:243]
apSSLServer::AcceptProtocol(apIProtocolServer*, apISocketWrapper*)
[apSSLServer.cpp:356]

The question is simple. Is it possible to minimize memory needed for
one SSL connection?

Michal

[Nelson B]

michal wrote:
> Hi all.
>
> We wrote a SSL server using nss. Clients are connecting to the server
> and keep the connection. We want to handle as much connections as
> possible. As memory usage grows when clients are connecting to the
> server I observed that every connection takes around 200Kb of memory.
> This means that our server cannot handle more than 3000-4000 clients
> with 1GB RAM. So I was experimenting with 20 clients on server that had
> enabled trace-malloc logging. I found out that every connection takes
> really around 185kb but 135kb of that memory is somehow related to the
> SSL (i.e. in the backtrace was libssl3.so). The most of this memory is
> 5x sslBuffer (5X 18432 bytes) and 1x sslSocket (11272 bytes).

Thanks for reporting this, and especially for including the stack traces.

This is the result of an performance optimization that dramatically
reduced the number of memory allocations and re-allocations by
increasing the minimum memory allocation size for certain buffers.

Your results show that there are 5 different buffers being used,
and the same minimum is being applied to them all. I think some
of those buffers can use much lower minumums without hurting
performance.

Please file a bug about this in bugzilla.mozilla.org, product=NSS,
component=libraries, release=3.11. If you file the bug, you will
receive email updates as it progresses. Alternatively, if you
prefer, I will file the bug (you won't get any email notices).

--
Nelson B

[michal]

> Thanks for reporting this, and especially for including the stack traces.
>
> This is the result of an performance optimization that dramatically
> reduced the number of memory allocations and re-allocations by
> increasing the minimum memory allocation size for certain buffers.
>
> Your results show that there are 5 different buffers being used,
> and the same minimum is being applied to them all. I think some
> of those buffers can use much lower minumums without hurting
> performance.
>
> Please file a bug about this in bugzilla.mozilla.org, product=NSS,
> component=libraries, release=3.11. If you file the bug, you will
> receive email updates as it progresses. Alternatively, if you
> prefer, I will file the bug (you won't get any email notices).

Thank you very much for your reply. I'll file the bug in bugzilla.

Michal