Last week in scylladb.git master (issue #150; 2022-10-09)

[Avi Kivity]

This short report brings to light some interesting commits to scylladb.git master from the last weeks. Commits in the 060dda8e00..0923cb435f range are covered.

There were 143 non-merge commits from 16 authors in that period. Some notable commits:

The log-structured allocator is used to manage cache and memtable memory. When memory runs out, the allocator tries to reclaim memory by evicting cache items and by defragmenting memory. If this takes too long, the allocator logs a stall report. Due to a bug, if the report threshold was set too low then the report is generated even if a stall did not happen, slowing down the system and flooding the logs. This is now fixed.

The view builder is responsible for building materialized views when a view is created and after repair. To control its memory consumption, it reads base table sstables in chunks. Due to a bug, it could forget the partition tombstone or an open range tombstone when moving between chunks, leading to a discrepancy between the base table data and the view data. This is now fixed.

The compaction manager now ignores out-of-disk-space (ENOSPC) exceptions when shutting down, so the server doesn't crash in these scenarios.

An inccuracy in the per_partition_rate_limit read metric was corrected. Tables with the per_partition_rate_limit property can throttle read and write activity on a per-partition basis.

ScyllaDB uses an interval map data structure from the Boost library to quickly locate sstables needed to service a read. Due to the way we interface with the library, updating the interval map was unnecessarily slow. This is now fixed.

The "snitch" is used to derive the node's location (data center and rack). The AWS EC2 snitch now uses Instance Metadata Service V2 (IMDSv2) to access instance metadata, per AWS recommendations.

The CQL CONTAINS and CONTAINS KEY operators are used to check if a collection contains an element. CONTAINS and CONTAINS KEY have been changed to return false if it is asked whether a collection contains NULL, since no collection can contain a NULL. This is a behavior change, but is not expected to affect users since the query is not useful.

ScyllaDB records large partitions, large cells, and large rows in system tables so that the primary key can be used to deal with them. It now additionally records collections with large numbers of elements, since these can cause degraded performance. The warning threshold is configurable.

See you in the next issue of last week in scylladb.git master!