High latency after setting row_cache_size_in_mb

[mohsin k]

Hi,

I have been looking in different configurations for tuning my cassandra servers. So, initially I loadtested server using cassandra-stress tool, with default configs and then tuning one by one config to measure impact of change. First config, i tried was by setting "row_cache_size_in_mb" to 1000 (MB) in yaml, adding caching {'keys': 'ALL', 'rows_per_partition': 'ALL'}. I am running a cluster with 4 nodes.

Following is my schema:

CREATE TABLE stresstest.user_to_segment (

    userid text,

    segmentid text,

    PRIMARY KEY (userid, segmentid)

) WITH CLUSTERING ORDER BY (segmentid DESC)

    AND bloom_filter_fp_chance = 0.1

    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}

    AND comment = 'A table to hold blog segment user relation'

    AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}

    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}

    AND crc_check_chance = 1.0

    AND dclocal_read_repair_chance = 0.1

    AND default_time_to_live = 0

    AND gc_grace_seconds = 864000

    AND max_index_interval = 2048

    AND memtable_flush_period_in_ms = 0

    AND min_index_interval = 128

    AND read_repair_chance = 0.0

    AND speculative_retry = '99PERCENTILE';

Following are node specs:

RAM: 4GB

CPU: 4 Core

HDD: 250BG

Following is the output of 'nodetool info' after setting row_cache_size_in_mb:

ID                     : d97dfbbf-1dc3-4d95-a1d9-c9a8d22a3d32

Gossip active          : true

Thrift active          : false

Native Transport active: true

Load                   : 10.94 MiB

Generation No          : 1517571163

Uptime (seconds)       : 9169

Heap Memory (MB)       : 136.01 / 3932.00

Off Heap Memory (MB)   : 0.10

Data Center            : dc1

Rack                   : rack1

Exceptions             : 0

Key Cache              : entries 125881, size 9.6 MiB, capacity 100 MiB, 107 hits, 126004 requests, 0.001 recent hit rate, 14400 save period in seconds

Row Cache              : entries 125861, size 31.54 MiB, capacity 1000 MiB, 4262684 hits, 4388545 requests, 0.971 recent hit rate, 0 save period in seconds

Counter Cache          : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds

Chunk Cache            : entries 273, size 17.06 MiB, capacity 480 MiB, 325 misses, 126623 requests, 0.997 recent hit rate, NaN microseconds miss latency

Percent Repaired       : 100.0%

Token                  : (invoke with -T/--tokens to see all 256 tokens)

Following is output of nodetool cfstats:

Total number of tables: 37

----------------

Keyspace : stresstest

Read Count: 4398162

Read Latency: 0.02184742626579012 ms.

Write Count: 0

Write Latency: NaN ms.

Pending Flushes: 0

Table: user_to_segment

SSTable count: 1

SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]

Space used (live): 11076103

Space used (total): 11076103

Space used by snapshots (total): 0

Off heap memory used (total): 107981

SSTable Compression Ratio: 0.5123353861375962

Number of partitions (estimate): 125782

Memtable cell count: 0

Memtable data size: 0

Memtable off heap memory used: 0

Memtable switch count: 2

Local read count: 4398162

Local read latency: 0.030 ms

Local write count: 0

Local write latency: NaN ms

Pending flushes: 0

Percent repaired: 0.0

Bloom filter false positives: 0

Bloom filter false ratio: 0.00000

Bloom filter space used: 79280

Bloom filter off heap memory used: 79272

Index summary off heap memory used: 26757

Compression metadata off heap memory used: 1952

Compacted partition minimum bytes: 43

Compacted partition maximum bytes: 215

Compacted partition mean bytes: 136

Average live cells per slice (last five minutes): 5.719932432432432

Maximum live cells per slice (last five minutes): 10

Average tombstones per slice (last five minutes): 1.0

Maximum tombstones per slice (last five minutes): 1

Dropped Mutations: 0

Following are my results:

[mohsin k]

The blue graph is before setting row_cache_size_in_mb, orange is after.

[Andrew Tolbert]

Hello,

I would recommend asking this question to the cassandra-user mailing list.

Thanks,

Andy

[mohsin k]

Thanks Andrew, will do the same.