Out of memory (and other exception) handling

[Herve]

Hi,

We're planning to move our application storage to Cassandra.
As our app is a server written in C, we naturally looked at the cpp-driver and it's C API, to understand a bit more how it works.

When doing so, we noticed the driver does not seem to handle the out of memory situation (nor other C++ exception situation).


Example,in session.cpp:

CassFuture cass_session_execute(CassSession session,
const CassStatement* statement) {
return CassFuture::to(session->execute(statement->from()));
}


Future Session::execute(const Request statement) {
RequestHandler* request_handler = new RequestHandler(statement);
Future* future = request_handler->future();
execute(request_handler);
return future;
}

Out of memory situation on RequestHandler allocation is not particularly handled, so the default behavior is a bad_alloc exception thrown, causing the above application to abort as exception in not caught in the C API cass_session_execute, nor in the C application itself.

Writing a wrapper around the driver calls to catch such exceptions, does not seem the solution neither as there's other cases requiring handling this at a lower level.

Example:

still session.cpp

Future* Session::prepare(const char* statement, size_t length) {
PrepareRequest* prepare = new PrepareRequest();
prepare->set_query(statement, length); // bad_alloc can be thrown here by
// std:string -> would require to
// delete 'prepare' to avoid memory
// leak
RequestHandler* request_handler = new RequestHandler(prepare); // same here
// bad_alloc can be also thrown
// would also require to also delete
// 'prepare' to avoid memory leak
ResponseFuture* future = request_handler->future();
future->statement.assign(statement, length);
execute(request_handler);
return future;
}

Those are simple cases, but I realize there's much more complex situations, including ones where if we can't allocate memory, we just can't complete the action or properly free some resources.

Example:

CassFuture* cass_session_close(CassSession* session) {
cass::SessionCloseFuture* close_future = new cass::SessionCloseFuture();
session->close_async(close_future);
return CassFuture::to(close_future);
}

Here, failure to allocate close_future will not allow to do proper calls to release resources allocated for the session in any ways ...

So as the API currently stands, exception handling can only be achieved by catching all exceptions thrown by the API, exit and restart the application...

Did I get anything wrong?
Is there a better way to handle this? What would be the plans to address this, allowing to write more robust applications?

Thanks

[Michael Penick]

You're right, exceptions are not handled, but if an exception is thrown the driver is in an unrecoverable state (as implemented now). I do agree that the API should return an error result when such situation can be handled. I've created a story to resolve this: https://datastax-oss.atlassian.net/browse/CPP-208. I definitely see this as something that should be handled sooner than later. I'll make sure this is given priority for the next release.

I'm wary about attempting to handle std::bad_alloc internally. I could see potentially handling this by shutting down the driver gracefully and returning error results requests that follow. I'll look into handling this. 

Out of curiosity, because it's interesting, how does your application currently response to out of memory situations? Does it have "scratch" memory that it can release in such situations to relieve memory pressure?

Mike

To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-us...@lists.datastax.com.

[Herve]

Hi Mike,

thanks for the prompt answer.

Our application does not implement any particular strategy on out of memory situation.
The code is in C, and all memory allocations are tested, and on error, cleanup and fallback code is executed and appropriate response code sent to client.

The application uses 'growing memory pools' (our own implementation) to handle memory allocation. In a nutshell, a memory pool allocates large buffer chunks from the heap, then allocates individual smaller pointers for the application within the buffer chunks. When a chunk is full, the pool allocates an other chunk from the heap.
Using this helps to keep the code simpler and leak-free (rather than freeing individual pointers allocated for the app, just free the pool that frees the chunks).
It also minimizes memory fragmentation over time.
But even growing the pool can fail when the system runs out of memory.

The out of memory situation remains a rare situation, but it happens, usually on high peak load or specific heavy operation ran on the server.
For us, failing current operations and cleaning after ourself is enough to revert to a stable situation after some time.


We have other apps using pieces of C++ code. For these, we use ref counted objects pointers to automatically clean allocated object on exceptions,
very similar to your RefCounted class...

I'm probably over-simplifying a bit, but simply generalizing the use of RefCounted class instead of actual pointers accross the cpp-driver code would help to execute adequate destructors and clean-up code on exception to address these issues.
Then for each C API, just adding a try catch would complete the effort.