rocksdb config

[Shannon Ma]

Hi,

I was asking rocksdb group for the issue i had, after it was resolved, they asked me to show them the LOG file for the configs i use, and here is the comment

The block cache (block_cache_size: 104857600) is set to only 100 MB and might impact read-benchmark.
And only 1 simultaneous background compaction is enabled, which can potentially adversely impact write-benchmarks.

Is there a way to change the config?

Thanks

Shannon

[Damian Guy]

Hi Shannon,

You can override the config by providing an instance of RocksDBConfigSetter:

/**
 * An interface to that allows developers to customize the RocksDB settings
 * for a given Store. Please read the <a href="https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide">RocksDB Tuning Guide</a>.
 */
public interface RocksDBConfigSetter {

    /**
     * Set the rocks db options for the provided storeName.
     * 
     * @param storeName     the name of the store being configured
     * @param options       the Rocks DB options
     * @param configs       the configuration supplied to {@link org.apache.kafka.streams.StreamsConfig}
     */
    void setConfig(final String storeName, final Options options, final Map<String, Object> configs);
}

You can set it on the StreamsConfig, i.e, 

properties.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, YourClass.getName());

Thanks,

Damian

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6c44b774-11a5-46cc-a996-888331e42edb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shannon Ma]

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

[Shannon Ma]

I just need to put the configs that i want to override, right?

Thanks

Shannon

[Michael Noll]

Yes.

--

You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/e0efd3d7-131f-480c-9c19-e9e960409e40%40googlegroups.com.

[Shannon Ma]

Hi,

Can you give an example how to set a config, say update/set block_cache_size? Do I just do

public void setConfig(String storeName, Options options, Map<String, Object> configs) {

 configs.put("block_cache_size", xxx);

}

[Eno Thereska]

Hi Shannon,

I'm realising we haven't documented this super well. One example for the block cache would be:

@Override
public void setConfig(final String storeName, final Options options, final Map<String, Object> configs) {
    final BlockBasedTableConfig tableConfig = new BlockBasedTableConfig();
    tableConfig.setBlockCacheSize(50 * 1024 * 1024L);
    tableConfig.setBlockSize(4096L);
    options.setTableFormatConfig(tableConfig);

    options.setMaxBackgroundCompactions(Runtime.getRuntime().availableProcessors());
    
}

Note that the compactions are already set correctly in trunk, but not in 0.10.2. A good reference for these values internally is around the code:

https://github.com/apache/kafka/blob/0.10.2/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L117

Thanks

Eno

[Shannon Ma]

Thanks. I have not updated to 0.10.2, still 0.10.1, so i can have my custom config with

options.setMaxBackgroundCompactions(Runtime.getRuntime().availableProcessors());

For block cache size, 0.10.2 changed to 50MB (was 100), rocksdb suggested more, whats the reason to change to 50 (i saw somewhere says bigger setting might cause memory issue)?

Thanks

Shannon

[Shannon Ma]

Hi,

this is the cache log, any suggestion on how to improve or change setting? Let me know if you need more details.

2017/04/11 11:00:29.143[DEBUG][](NamedCache.java): Named cache 0_2-txnwindow stats on flush: #hits=0, #misses=0, #overwrites=0, #flushes=28

2017/04/11 11:00:29.143[DEBUG][](ThreadCache.java): Thread dupdetection-1-StreamThread-1 cache stats on flush: #puts=18, #gets=0, #evicts=18, #flushes=85

2017/04/11 11:00:29.144[DEBUG][](NamedCache.java): Named cache 0_2-lanewindow stats on flush: #hits=0, #misses=0, #overwrites=0, #flushes=28

2017/04/11 11:00:29.144[DEBUG][](ThreadCache.java): Thread dupdetection-1-StreamThread-1 cache stats on flush: #puts=18, #gets=0, #evicts=18, #flushes=86

2017/04/11 11:00:29.145[DEBUG][](NamedCache.java): Named cache 0_2-tagwindow stats on flush: #hits=0, #misses=0, #overwrites=0, #flushes=28

2017/04/11 11:00:29.146[DEBUG][](ThreadCache.java): Thread dupdetection-1-StreamThread-1 cache stats on flush: #puts=18, #gets=0, #evicts=18, #flushes=87

2017/04/11 11:00:29.146[DEBUG][](NamedCache.java): Named cache 0_2-multitag stats on flush: #hits=0, #misses=0, #overwrites=0, #flushes=22

2017/04/11 11:00:29.147[DEBUG][](ThreadCache.java): Thread dupdetection-1-StreamThread-1 cache stats on flush: #puts=18, #gets=0, #evicts=18, #flushes=88

Thanks

Shannon

[Shannon Ma]

Okay i was setting  cache.max.bytes.buffering to 0, now 

2017/04/11 12:14:01.566[DEBUG][](ThreadCache.java): Thread dupdetection-1-StreamThread-2 cache stats on flush: #puts=800, #gets=0, #evicts=0, #flushes=76

2017/04/11 12:14:03.786[DEBUG][](ThreadCache.java): Thread dupdetection-1-StreamThread-1 cache stats on flush: #puts=400, #gets=0, #evicts=0, #flushes=73

2017/04/11 12:14:03.786[DEBUG][](ThreadCache.java): Thread dupdetection-1-StreamThread-1 cache stats on flush: #puts=400, #gets=0, #evicts=0, #flushes=74

but how gets is 0, i am fetching from store, what does that mean? or there is another setting i missed?

[Eno Thereska]

Hi Shannon,

I'd ignore the cache.max.bytes.buffering for now. Your original question was for the RocksDb cache. Let me answer your question on that first:

"For block cache size, 0.10.2 changed to 50MB (was 100), rocksdb suggested more, whats the reason to change to 50 (i saw somewhere says bigger setting might cause memory issue)?"

We changed to 50 for scenarios where the cache doesn't help and you have a lot of stores, in which case a lot of memory per store would quickly add up (e.g., if you have 100 stores, with 100 MB each, you'd need 10GB).

However, if you have just a few stores, making this bigger is the way to go, as the folks in RocksDb suggested.

About "cache.max.bytes.buffering": we'd recommend using this (called the Thread/Named cache) for dedupping cases (see KIP-63), not necessarily for caching scenarios. So setting this to 0, while upping the RocksDb cache is the way to go. Note that all the logs above are for the Thread/Named cache, so it will be hard to see the RocksDb hits/misses.

Thanks

Eno

[Shannon Ma]

Thanks, are you referring to these?

2017/04/12 16:20:34.047[StreamThread-2][DEBUG][](NamedCache.java): Named cache 0_0-lanewindow stats on flush: #hits=800, #misses=0, #overwrites=0, #flushes=10

2017/04/12 16:20:34.047[StreamThread-2][DEBUG][](NamedCache.java): Named cache 0_0-tagwindow stats on flush: #hits=0, #misses=0, #overwrites=0, #flushes=10

2017/04/12 16:20:34.047[StreamThread-2][DEBUG][](NamedCache.java): Named cache 0_0-multitag stats on flush: #hits=0, #misses=0, #overwrites=0, #flushes=10

2017/04/12 16:20:34.048[StreamThread-3][DEBUG][](NamedCache.java): Named cache 0_2-lanewindow stats on flush: #hits=800, #misses=0, #overwrites=0, #flushes=10

2017/04/12 16:20:34.048[StreamThread-3][DEBUG][](NamedCache.java): Named cache 0_2-tagwindow stats on flush: #hits=0, #misses=0, #overwrites=0, #flushes=10

2017/04/12 16:20:34.048[StreamThread-1][DEBUG][](NamedCache.java): Named cache 0_1-txnwindow stats on flush: #hits=400, #misses=0, #overwrites=0, #flushes=10

2017/04/12 16:20:34.048[StreamThread-1][DEBUG][](NamedCache.java): Named cache 0_1-lanewindow stats on flush: #hits=400, #misses=0, #overwrites=0, #flushes=10

2017/04/12 16:20:34.048[StreamThread-3][DEBUG][](NamedCache.java): Named cache 0_2-multitag stats on flush: #hits=0, #misses=0, #overwrites=0, #flushes=10

2017/04/12 16:20:34.049[StreamThread-1][DEBUG][](NamedCache.java): Named cache 0_1-tagwindow stats on flush: #hits=790, #misses=0, #overwrites=394, #flushes=10

2017/04/12 16:20:34.050[StreamThread-1][DEBUG][](NamedCache.java): Named cache 0_1-multitag stats on flush: #hits=0, #misses=0, #overwrites=0, #flushes=10

[Eno Thereska]

Yes.

[Shannon Ma]

The value in my state store is ArrayList<SpecificRecord>, i noticed sometime when the list size is big (1000+), the store.put() takes quite some time (up to 1-2 sec), but it does not happen for all the cases that size is big. Is there a log can help find the root cause, or in this case (writing large data), what configs i can try?

Thanks

Shannon