RocksDb Configuration setter on Kafka Streams properties

[Stathis Souris]

Hi,

Any particular reason why RocksDB `WriteOptions` and `FlushOptions` are not exposed through the `RocksDbConfigSetter` interface?

In particular I would like to test my setup on our cluster with WriteOptions.setSync to false and measure performance.

Anyway my issue here is how do I configure RocksDB running inside inside a docker container.

Do I really need it to do sync writes.

Thanks in advance.

[Matthias J. Sax]

Not sure if I understand the question.

When you implement a `RocksDbConfigSetter`, Kafka Streams calls
setConfigs method and you can modify the provide `org.rocksdb.Options`
object.

I don't know all available configs of the top of my head, but if options
are missing, it's a limitation of RocksDB JNI binding -- Kafka Streams
does not "hide" and options but exposes everything JNI exposes.

-Matthias

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/93ce58c0-d4e0-4bfc-ac87-b787c9afa1e6%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/93ce58c0-d4e0-4bfc-ac87-b787c9afa1e6%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Stathis Souris]

Thanks for answering.

Basically I am curious why `WriteOptions` and `FlushOptions` are not exposed through the `RocksDbConfigSetter`

private Options options;
private WriteOptions wOptions;
private FlushOptions fOptions;

but apart from that I think I found the configurations I was looking for.

Thanks for taking the time though.

On Thursday, 15 March 2018 18:43:36 UTC, Matthias J. Sax wrote:

Not sure if I understand the question.

When you implement a `RocksDbConfigSetter`, Kafka Streams calls
setConfigs method and you can modify the provide `org.rocksdb.Options`
object.

I don't know all available configs of the top of my head, but if options
are missing, it's a limitation of RocksDB JNI binding -- Kafka Streams
does not "hide" and options but exposes everything JNI exposes.

-Matthias



On 3/15/18 7:14 AM, Stathis Souris wrote:
> Hi,
>
> Any particular reason why RocksDB `WriteOptions` and `FlushOptions` are
> not exposed through the `RocksDbConfigSetter` interface?
> In particular I would like to test my setup on our cluster with
> WriteOptions.setSync to false and measure performance.
>
> Anyway my issue here is how do I configure RocksDB running inside inside
> a docker container.
> Do I really need it to do sync writes.
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

[Guozhang Wang]

Hello Stathis,

Just curious what option overrides are you currently considering?

Guozhang

[Stathis Souris]

Hi,

Basically,

WriteOptions.sync

FlushOptions.waitForFlush

but in the end I found that I didn't really need to touch those.

and currently I am experimenting with properties:

options.writeBufferSize
options.maxWriteBufferNumber
options.minWriteBufferNumberToMerge

Stathis