[RELEASE] Scylla 4.3 Release Candidate 1 (RC1)
Tzach Livyatan
<tzach@scylladb.com>The Scylla team is pleased to announce the release of Scylla Open Source 4.3 RC1, the first Release Candidate for the Scylla Open Source 4.3 minor release. Moving forward, only critical bugs will be fixed in branch-4.3. We will continue to fix bugs and add features to the master branch.
Scylla 4.3 includes production ready Change Data Capture (CDC) as well as an experimental version of AWS DynamoDB compatible (Alternator) streams, off line (air gap) installer, the first step toward removing the seed node concept, as well as many other improvements and bug fixes (below).
Find the Scylla Open Source 4.3 repository for your Linux distribution here.
Scylla 4.3 RC1 Docker is also available.
Use the release candidate with caution; RC1 is not production-ready yet. You can help stabilize Scylla Open Source 4.3 by reporting bugs here.
Please note that only the last two minor releases of Scylla Open Source project are supported. Once Scylla Open Source 4.3 is officially released, only Scylla Open Source 4.3 and Scylla 4.2 will be supported, and Scylla 4.1 will be retired.
Related Links
* Get Scylla Open Source 4.3 (on each distro)
New features in Scylla 4.3
Change Data Capture (CDC)
Change Data Capture (CDC) allows users to track data updates in their Scylla database. While it is similar to the feature with the same name in Apache Cassandra and other databases, how it is implemented in Scylla is unique, more accessible, and more powerful.
Originally introduced in Scylla 4.1 as experimental, CDC is now graduated to a production ready feature.
Note that you still need an experimental flag for rc1.
CDC in 4.3 is not fully compatible with previous versions so it is recommended to turn CDC off on all tables before upgrading to 4.3 and enabling it again on selected tables after the upgrade.
One of the biggest changes was a rename of the table that contains CDC streams description from “cdc_description“ in 4.1 to “cdc_streams” in 4.2 and finally to “cdc_streams_descriptions” in 4.3.
With Scylla’s CDC, you can choose to keep track of the updates, original values and/or new values. The data is stored in regular Scylla tables (SSTables) and can be queried asynchronously using a standard Scylla/Cassandra CQL driver. Data in a CDC table is set with a TTL, minimizing the possibility of an overflow.
Example: creating a table with CDC for preimage only:
CREATE TABLE base_table (
pk text,
ck text,
val1 text,
val2 text,
PRIMARY KEY (pk, ck)
) WITH cdc = { 'enabled':true, 'preimage':true};
CDC is also used internally to implement the AWS DynamoDB API (Alternator) new Streams feature (see below)
More on information:
Using Change Data Capture (CDC) in Scylla
Alternator Streams
Based on CDC, the following DynamoDB API stream operations are now supported in Scylla:
DescribeStream
GetRecords
GetShardIterator
ListStreams
Offline Installer
Scylla is now available as a binary tar file. You can download the tar file from the download page, move it to an air gap server (without an external networking connection) and install it. #6626 #6949
(we will share more details and download link soon)
Non root installer
You can use Scylla offline installation without root permissions.
Deployment
Scylla is now available in Ubuntu 20
Scylla now has an official GCP Image
Scylla AMI is now based on Amazon Linux 2 (previously based on CentOS)
Support from direct upgrade from 2018 vintage releases was removed. One can still upgrade via intermediate versions (e.g. 2.3 -> 3.1 -> 3.1 -> 3.3 -> 4.0 -> 4.1).
Amazon Linux 2
Scylla Open Source Community AMI is now based on Amazon Linux 2.
The AMI supports the same “centos” and “scylla” users, backward compatible with previous versions of the AMI
GCP Images
Scylla is now available as a GCP image on Google Compute Platform (GCP) #7080 #6631
Default login to the image is (as for AMI) “centos”.
(we will share more details and download link soon)
Remove the seed concept in gossip
The concept of seed and the different behavior between seed nodes and non seed nodes generate a lot of confusion, complication and error for users. Starting from this release, seed nodes are ignored in Gossip protocol. They are still in use (for now) as part of the node bootstrap process. #6845
More on seedless nodes here
Security related updates
gnutls vulnerability fix: GNUTLS-SA-2020-09-04 #7212
New optional CQL ports (19042, 19142, see Shard aware CQL ports below)
Allow users to disable CQL unencrypted native transport by setting it to zero #6997
CQL Extensions
A new CQL protocol extension allows client drivers to distinguish between lightweight transactions (LWT) and non-LWT statements. The intent is to prefer the primary replica when sending LWT statements, to reduce transaction aborts due to contention.
Shard aware CQL ports: new CQL port (19042 by default) is being open for the so-called “advanced shard-aware drivers”. It works exactly as the typical 9042 works for existing drivers (connector libraries), but it allows the client to choose the specific shard to connect to by precise binding of the client-side (ephemeral) port. Also, a TLS alternative is supported, under port 19142.
The extensions above are already implemented or in progress in Scylla Java Driver.
Additional Features
Support for SSTable "md" format (CASSANDRA-14861) #4442
Scylla Enterprise Docker: a new '--io-setup N' command line option, which users can pass to specify whether they want to run the "scylla_io_setup" script or not. This is useful if users want to specify I/O settings themselves in environments such as Kubernetes, where running "iotune" is problematic. #6587
Requests Role or User are now tracked in the tracing output #6737
REST API: Add long polling to StorageServiceRepairAsyncByKeyspaceGet #6445
Docker: Scylla on Docker now supports the passing of extra arguments to scylla #7458
Tools
node_exporter is an agent used to report OS level metrics to Scylla Monitoring Stack. In this release installed node_exporter is update from 0.17.0 to 1.0.1
Nodetool getendpoints can return wrongs nodes list #7134
The JMX management interface was enabled for Java 11 for debian systems. It was already supported, but a packaging error resulting in installation problems.
nodetool now support the gettraceprobability command #7265
Performance Optimizations
Cleanup: nodetool cleanup procedure is used after adding nodes, to remove data that was migrated to the new nodes. The calculation that determines which sstables needed to be rewritten (cleaned up) was inefficient and could cause stalls in queries running at the same time, so it was optimized #6662
JSON: More of the code base was migrated from the jsoncpp library to the rjson library, improving json performance, JSON is heavily used in Scylla Alternator.
MV: After a repair, bootstrap, or decommission (or a similar operation), the node receiving new data must update materialized views. This is done by reading staging sstables containing the new data, row by row, and updating the view rows corresponding to those rows. There can be large numbers of such sstables (one per vnode per peer), and reading from such large numbers of sstables requires a lot of memory. This memory usage is now dramatically reduced by exploiting the property that per-vnode sstables have few overlaps, and reusing the partitioned_sstable_set (which we use for leveled compaction strategy, which has similar properties) to avoid reading from all those sstables at once. #6707
Repair: moving from to btree_set for repair_hash eliminate the need for large allocation which cause stalls #6190
The Scylla cache and memtable implementations now use a btree instead of a red-black tree. This improves performance in cache-intensive and write-intensive workloads. More here
utf8 validation of large cells causes latency spikes. In this release, UTF8 validation update to work on fragmented buffers to fix this. #7448
Configuration
The following new configuration parameters are available in this release:
max_concurrent_requests_per_shard: Maximum number of concurrent requests a single shard can handle before it starts shedding extra load. By default, no requests will be shed. Default: max (disabled)
Native_shard_aware_transport_port: Like native_transport_port, but clients-side port number (modulo smp) is used to route the connection to the specific shard. Default: 19042 (see CQL Extensions above)
native_shard_aware_transport_port_ssl: Like native_transport_port_ssl, but clients are forwarded to specific shards, based on the client-side port numbers. Default: 19142. (see CQL Extensions above)
enable_sstables_md_format: Enable SSTables 'md' format to be used as the default file format (requires enable_sstables_mc_format). Default: true
max_memory_for_unlimited_query was replaced by two new parameters;
max_memory_for_unlimited_query_soft_limit Maximum amount of memory a query, whose memory consumption is not naturally limited, is allowed to consume, e.g. non-paged and reverse queries. This is the soft limit, there will be a warning logged for queries violating this limit. Default: 1 MB
max_memory_for_unlimited_query_hard_limit: Maximum amount of memory a query, whose memory consumption is not naturally limited, is allowed to consume, e.g. non-paged and reverse queries. This is the hard limit, queries violating this limit will be aborted. Default: Max 100 MB
schema_registry_grace_period: Time period in seconds after which unused schema versions will be evicted from the local schema registry cache. Default: 1 second
alternator_streams_time_window_s: CDC query confidence window for alternator streams. Default 10 second
Redis API
The following commands have been added to Scylla Redis API:
Build and Debugging
New documentation on Scylla debugging
Toolchain: regenerate for gcc 10.2, As a side effect, this also brings in xxhash 0.7.4 #6813
Other bugs fix in the release.
For a full list use git log
CQL: Numbers of the 'decimal' type that had negative scale (which translates to positive exponent: 1.2e1 has negative scale, while 1.2e-1 has positive scale), when casted to a floating-point type ('CAST x AS float'), thew the node into a long loop, effectively making it unavailable. #6720
CQL: A bug where impossible range restrictions (WHERE a > 0 AND a < 0) was processed incorrectly #5799
Stability: Internal schema change CQL queries should not be used for distributed tables #6700
CQL: min()/max() return wrong results on some collections #6768
Stability: a rare memory leak caused by improper commitlog usage from hints manager #6776 #6409
Stability: Row-level repair was made more robust against hash collisions. Row-level repair uses a hash to identify mismatched rows. A weak hash is used to reduce computation and network costs, but this results in the possibility of collisions. Now, when a collision is detected, repair will copy the colliding rows (even if there was no problem) rather than fail the repair.
Stability: Repair now uses a uuid to identify repair jobs #6786
Stability: compaction should print a unique id to correlate start and finish log messages #6840
Alternator: tracing was fix for GetItem and BatchItem #6891
Stability: TWCS: compaction: partition estimate can become 0, causing an assert in sstables::prepare_summary() #6913
scyllatop: using `metricPattern` fails with "dictionary changed size during iteration" #7488
Stability: Cleanup compaction in KA/LA sstables may crash the node in some cases #7553
Stability: secondary index updates failing after upgrade to 4.2.0 due to missing system_schema.computed_columns. The problem is limited to secondary indexes created *before* Scylla 3.2, which had their `idx_token` column incorrectly not marked as computed #7515
Stability: schema integrity issues that can arise in races involving materialized views may cause segmentation fault and coredump happened during starting after scylla was killed #7420
Stability: provide strong exception guarantees from load_sstable() #6658
Stability: Incrementally delete resharded sstables as they're retired #7463
Stability: Node may get stuck in schema disagreement loop when bootstrapping sequentially #7396
CQL: token_restriction: invalid_request_exception on SELECTs with both normal and token restrictions #7441
lwt: store column_mapping's for each table schema version upon a DDL change
Stability (refactoring): Move write() methods from class sstable to class sstable_writer #3012
Stability: Clean cluster issued 'Exceptional future ignored' right after been started with no load on it #7352
UX: Make batchlog size warning clearer #7367
Stability: ascii validation of large cells causes large allocations #7393
Stability: Scylla 4.1.7 crashing on repairs (uncaught exceptions) #7285
Stability: Shutting down database hangs in dtest-debug #7331
Stability: Useless linearization on large data during validation, of either type bytes or string-derived, potentially cause stalls due to reclaiming #7318
Stability: NEW_NODE should be sent after listening for CQL clients has started #7301
Stability (counters): runtime error: signed integer overflow cannot be represented in type 'long int' #7330
Non root install (Logging): can't find scylla log by journalctl --user -xe #7131
Non root install: nonroot install: ubuntu18 failed to start for `NOFILE rlimit too low` #7133
CQL (found with Jepsen): Aborted reads of writes which fail with "Not enough replicas available" #7258
Stability: Unsynchronized cross-shard memory operations caused by incorrectly used updateable_value #7310
Stability: Reduce unnecessary VIEW_BACKLOG updates in gossip #5970
Non root install: nonroot: systemctl --user enable scylla-server.service failed on Ubuntu 18 #7288
Non root install: Got offline mode warnings on nonroot mode #7286
CQL: (LOCAL_/EACH_)QUORUM consistency calculation is broken when RF=0 #6905
Install: scylla_prepare: 'get_set_nic_and_disks_config_value' is not defined #7276
Stability: RPC server still has handlers registered in dtests involving repair-based operations #7262
Stability: Query pager can try to get keys from empty vector of result::partitions #7263
Install: scylla_cpuscaling_setup: Got warning when installing scylla-cpupower.service #7230
Stability: disable_autocompaction_nodetool_test failed: std::runtime_error (Too early: storage service not initialized yet) #7199
Stability: abstract_replication_strategy::do_get_ranges is passed a reference to token_metadata that may be invalidated if it yields #7044
Install: scylla_setup doesn't support to skip install of hugepages or libhugetlbfs-bin package #7182
Stability: init - Startup failed: std::runtime_error (Could not find tokens for 10.0.0.155 to replace) during large-partition-4d test #7166
Init: scylla4 process fails to restart (perftune.py: error: argument --mode: invalid choice: 'None') #6847
CQL: Forbid adding new fields to UDTs used in partition key columns #6941
Stability: Make allocation_section decay the reserves over time #325 (from 2015!)
Init: scylla_setup failing with dependency errors #7153
CQL (found with Jepsen): Weird return values from batch updates #7113
Non root install: scylla-python3 isn't loaded for setup scripts #7130
Stability: some non-prepared statements can leak memory (with set/map/tuple/udt literals) #7019
Stability: Reactor stall for 6 ms in sstables::seal_summary() #7108
Stability: coredump while node hit enospc: "Assertion `!this->_future && this->_state && !this->_task' failed #7085
Stability: gossip: Apply state for local node in shadow round #7073
Stability (CDC): CDC: within a batch, partition deletes and range deletes do not affect postimage #6900
Stability: repair_joint_row_3nodes_1_diff_shard_count_test fails on unexpected rx_row_nr #7086
CDC: cdc delta == keys does not produce cdc$operation nor cdc$ttl #7095
Stability: Hinted handoff is using very long timeout to sending some hints #7051
Build: dist: scylla-python3 should be separated repository #6751
Stability: Track repair_meta created on both repair follower and master #7009
Stability: Failed compaction : compaction_manager - compaction failed: sstables::malformed_sstable_exception (consumer not at partition boundary #6529
Stability: sstable code needs to close files in error paths #6448
Stability: Scylla setup failed: unit mdmonitor.service is not found or invalid #7000
UX: scylla_setup: include swap size in the prompt #6947
Untyped result sets may cause segfaults when parsing disengaged optionals #6915
UX: scylla_setup: default "done" when there is no disk to choose from #6760
Stability: simple_kill_streaming_node_while_bootstrapping_test: Startup failed: std::runtime_error ({shard 0: fmt::v6::format_error (invalid type specifier), shard 1: fmt::v6::format_error (invalid type specifier)})']) #6874
Setup: scylla_setup does not abort RAID setup when no free disk available #6860
CQL: Filtering captures uninitialized/deleted values of certain types #6295
Stability: repair: Relax node selection in bootstrap when nodes are less than RF #6744
UX: scylla_prepare: Improve error message on missing CPU features #6528
CQL: NULL counters treated as 0 #6382
CQL: IN(NULL) yields different results with prepared statements #6369
CQL: LIKE filter ignored on column key #6371
CQL: NULL and empty text are considered equal #6372
Stability: scylla_coredump_setup always fails on CentOS7 #6789
Setup: node_exporter_install --force failing #6782
repair: inaccurate log from check_failed_ranges #6785
repair: log recoverable errors as warnings rather than info messages #5612
scylla setup fails on Oracle Linux Artifact: OS variant not recognized #6761
scylla_util.py: duplication on detecting distribution #6691
scylla_setup: on RAID prompt, strange output when passing comma separated multiple disks #6724
Stability: connection storm when attempting to achieve shard-per-connection #5239
Stability: Nodetool Repair failing on keyspace with std::runtime_error (row_diff.size() != set_diff.size()) #6252
some non-prepared statements can leak memory (with set/map/tuple/udt literals) #7019
Regards