question about MongoRocks configuration

[MarkCallaghan]

Setting storage.rocksdb.compression=none doesn't have the effect that I expected. It only determines the compression type for level 2, no compression is hardwired for levels 0 and 1 and maybe zlib is hardwired for levels 3 and larger. I need to explicitly set compression_per_level to avoid this. Is this expected?

I am using Percona MongoDB 3.4.2-1.2 with this mongo.conf:

processManagement:

  fork: true

systemLog:

  destination: file

  path: /home/mdcallag/b/pmon342/log

  logAppend: true

storage:

  syncPeriodSecs: 60

  dbPath: /data/m/mo

  journal:

    enabled: true

  mmapv1:

    journal:

      commitIntervalMs: 100

operationProfiling.slowOpThresholdMs: 2000

replication.oplogSizeMB: 2000

storage.rocksdb.cacheSizeGB: 2

storage.rocksdb.compression: none

In the mongod log after startup I see:

[RocksDB] Block Cache Size GB: 2

[RocksDB] Compression: none

[RocksDB] MaxWriteMBPerSec: 1024

Engine custom option: compression_per_level=kNoCompression:kNoCompression:kNoCompression:kZlibCompression:kZlibCompression:kZlibCompression:kZlibCompression;compression_opts=-14:1:0;

[initandlisten] options: { config: "mongo.conf", master: true, operationProfiling: { slowOpThresholdMs: 2000 }, processManagement: { fork: true }, replication: { oplogSizeMB: 2000 }, storage: { dbPath: "/data/m/mo", engine: "rocksdb", journal: { commitIntervalMs: 100, enabled: true }, rocksdb: { cacheSizeGB: 2, compression: "none", configString: "compression_per_level=kNoCompression:kNoCompression:kNoCompression:kZlibCompression:kZlibCompression:kZlibCompression:kZlibCompression;compression_opt..." }, syncPeriodSecs: 60.0 }, systemLog: { destination: "file", logAppend: true, path: "/home/mdcallag/b/pmon342/log" } }

2017/03/11-07:38:43.499777 7feaa68fbec0        Options.compression[0]: NoCompression

2017/03/11-07:38:43.499778 7feaa68fbec0        Options.compression[1]: NoCompression

2017/03/11-07:38:43.499779 7feaa68fbec0        Options.compression[2]: NoCompression

2017/03/11-07:38:43.499780 7feaa68fbec0        Options.compression[3]: Zlib

2017/03/11-07:38:43.499782 7feaa68fbec0        Options.compression[4]: Zlib

2017/03/11-07:38:43.499783 7feaa68fbec0        Options.compression[5]: Zlib

2017/03/11-07:38:43.499796 7feaa68fbec0        Options.compression[6]: Zlib

From mongo-rocks/src/rocks_engine.cpp I see this, meaning my use of "storage.rocksdb.compression=none" only determines the compression type for level 2. If I want to set it for other levels I uave to set compression_per_level.

    rocksdb::Options RocksEngine::_options() const {

        // default options

        rocksdb::Options options;

        options.rate_limiter = _rateLimiter;

        rocksdb::BlockBasedTableOptions table_options;

        table_options.block_cache = _block_cache;

        table_options.filter_policy.reset(rocksdb::NewBloomFilterPolicy(10, false));

        table_options.block_size = 16 * 1024; // 16KB

        table_options.format_version = 2;

        options.table_factory.reset(rocksdb::NewBlockBasedTableFactory(table_options));

        options.write_buffer_size = 64 * 1024 * 1024;  // 64MB

        options.level0_slowdown_writes_trigger = 8;

        options.max_write_buffer_number = 4;

        options.max_background_compactions = 8;

        options.max_background_flushes = 2;

        options.target_file_size_base = 64 * 1024 * 1024; // 64MB

        options.soft_rate_limit = 2.5;

        options.hard_rate_limit = 3;

        options.level_compaction_dynamic_level_bytes = true;

        options.max_bytes_for_level_base = 512 * 1024 * 1024;  // 512 MB

        // This means there is no limit on open files. Make sure to always set ulimit so that it can

        // keep all RocksDB files opened.

        options.max_open_files = -1;

        options.optimize_filters_for_hits = true;

        options.compaction_filter_factory.reset(new PrefixDeletingCompactionFilterFactory(this));

        options.enable_thread_tracking = true;

        options.compression_per_level.resize(3);

        options.compression_per_level[0] = rocksdb::kNoCompression;

        options.compression_per_level[1] = rocksdb::kNoCompression;

        if (rocksGlobalOptions.compression == "snappy") {

            options.compression_per_level[2] = rocksdb::kSnappyCompression;

        } else if (rocksGlobalOptions.compression == "zlib") {

            options.compression_per_level[2] = rocksdb::kZlibCompression;

        } else if (rocksGlobalOptions.compression == "none") {

            options.compression_per_level[2] = rocksdb::kNoCompression;

        } else if (rocksGlobalOptions.compression == "lz4") {

            options.compression_per_level[2] = rocksdb::kLZ4Compression;

        } else if (rocksGlobalOptions.compression == "lz4hc") {

            options.compression_per_level[2] = rocksdb::kLZ4HCCompression;

        } else {

            log() << "Unknown compression, will use default (snappy)";

            options.compression_per_level[2] = rocksdb::kSnappyCompression;

        }

[MarkCallaghan]

Never mind. It behaves as expected. I was using a different mongo.conf then I shared below.

[Shrikanth N]

Hi Mark,

I am getting below error while changing MongoRocks variable on config file, could you pls suggest me on this

Error parsing YAML config file: yaml-cpp: error at line  47, column 5:

The variables i would like to change are 

max_background_compactions=8

max_background_flushes=4

table_options.block_size=16*1024

write_buffer_size=512MB

max_write_buffer_number=6

min_write_buffer_number_to_merge=2

target_file_size_base=206*1024*1024