Write bytes water mark

[Jan Kristof Nidzwetzki]

Hello,

I am using the cpp-driver 2.0 and I wrote an application, that executes a lot of insert statements. After a certain time, I get some error messages, complaining about the pending bytes high watermark. For example:

-+-
1431620344.707 [WARN] (src/connection.cpp:232:bool cass::Connection::write(cass::Handler*, bool)): Exceeded write bytes water mark (current: 129207 water mark: 65536) on connection to host XXX.XXX.XXX.XXX
-+-

I know, I can configure the water mark to a higher value. However, I would prefer to slow down my application when my cassandra cluster is not capable to handle the amount of statements and the number of pending write bytes increases. Is it possible, to ask the cpp-driver about the currently pending write bytes?

I noticed, that the information is stored in the Class 'Connection' in the variable 'pending_writes_size_', but I don't find a way to access the variable.

Best Regards
Jan

[Michael Penick]

Currently, there isn't a way to get configuration values. If this is something that would be helpful to applications I would consider adding it.

You could start with default (available in cassandra.h) in a external variable and keep it update as you set the configuration value.

Mike

To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-us...@lists.datastax.com.

[Jan Kristof Nidzwetzki]

Hello Mike,

thanks for your reply. For my application, it would be very useful, if I could determine the configured write byte watermark and the pending write bytes.

I have to store a huge amount of data in Cassandra. My current problem is, I don't know how much concurrency my Cassandra cluster can handle. If I am using too few concurrent insert statements, the application runs unnecessarily long. If I am using to too many concurrent insert statements, I will reach the write byte watermark.

With both values, I should be able, to determine how much concurrent insert statements my Cassandra cluster can handle. When the amount of pending write bytes approximates the watermark, my application would reduce the amount of concurrency. Or there is a better solution for this problem?

Best Regards
Jan

[Alex Popescu]

That sounds like an interesting approach. Could you draft how that code would look like ideally for you?

--
Bests,

Alex Popescu | @al3xandru
Sen. Product Manager @ DataStax

[Jan Kristof Nidzwetzki]

Hello Alex,

thank you for your feedback. A basic approach to determine the amount of concurrency in my application, could be the following code:

-+-
while(true) {
// Generating new insert statements
char* insertStatement = "INSERT INTO .....";

size_t highBytesWatermark = session->getHighBytesWatermark();
size_t insertStatementSize = strlen(insertStatement);

// Ask the session, if there is a connection available that can handle the insert statement
bool connectionAvailable = session->isAConnectionAvailableWithLessThenPendingBytes(highBytesWatermark - insertStatementSize);

if(! connectionAvailable) {
// Doing a spinlock or maybe calling a blocking function e.g.
// session->waitForPendingBytesBelow(highBytesWatermark - insertStatementSize);
}

future insertFuture = session->execute(insertStatement);
}
-+-

Depending on the amount of pending write bytes, a new async insert statement is executed or the application waits until a connection is available, that can handle the statement.

The Cassandra java driver provides some information about the session, which can be used to achieve something similar. The session can be asked for the amount of 'in flight queries' [1]. As soon as the amount of in flight queries increases, the application could reduce the amount of generated insert statements or wait for some time.

Best Regards
Jan

====
URLs:

[1] http://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/Session.State.html#getInFlightQueries%28com.datastax.driver.core.Host%29

[Elia Mazzuoli]

Hi,

There is news about that features?

BR,

Elia.