[RELEASE] Scylla 4.6 Release Candidate 1 (RC1)
Tzach Livyatan
<tzach@scylladb.com>The Scylla team is pleased to announce Scylla Open Source 4.6 RC1, the first Release Candidate for the Scylla Open Source 4.6 minor release.
Scylla 4.6 includes ARM support, new replace-node streaming, a new restore (load and stream) operation, and other performance and stability improvements and bug fixes (below).
Find the Scylla Open Source 4.6 repository for your Linux distribution here. Scylla 4.6 RC1 Docker is also available.
Use the release candidate with caution; RC1 is not production-ready yet. You can help stabilize Scylla Open Source 4.6 by reporting bugs here.
Only the last two minor releases of the Scylla Open Source project are supported. Once Scylla Open Source 4.6 is officially released, only Scylla Open Source 4.6 and Scylla 4.5 will be supported, and Scylla 4.4 will be retired.
Related Links
* Get Scylla Open Source 4.5 (for each distro)
New features in Scylla 4.6
ARM support
Scylla 4.6 is supported ARM architecture, including:
EC2 ARM base AMI, ready for Graviton2
Running Scylla Docker on ARM , including Mac M1
Repair Base Node Operations (RBNO)
Repair Base Node Operations was introduced as an experimental feature in Scylla 4.0, which uses repair to stream data for node-operations like replace, bootstrap and others. While still considered experimental, we continue to work on this feature.
Repair is oriented towards moving small amounts of data, not an entire node's worth. This resulted in many sstables being created in the node, creating a large compaction load. To fix that, offstrategy compaction is now used to efficiently compact these sstables without impacting the primary workload. #5226
In 4.6, RBNO is enabled by default only for replace node operation.
Example from scylla.yaml:
Enable_repair_based_node_ops: true
allowed_repair_based_node_ops: replace
To enable other operations (experimental), add them as a comma-separated list to allowed_repair_based_node_ops. Available operations are: bootstrap, replace, removenode, decommission and rebuild
Service Level properties
Service Levels allows the user to attach attributes to Rules and Users. These attributes apply to each session the user opens to Scylla, enabling granular control of the session properties, like time out and shedding (overload handling).
So far, service levels have been used to implement Workload Prioritization in Scylla Enterprise.
In this release, service levels are merged to Scylla Open Source to implement two features:
per service level timeouts
Workload types
per service level timeouts
You can now create service levels with customized read and write timeouts and attach them to rules and users. This is useful when some workloads, like ETL, are less sensitive to latency than others.
For example:
CREATE SERVICE LEVEL sl2 WITH timeout = 500ms;
ATTACH SERVICE LEVEL sl2 TO scylla;
ALTER SERVICE LEVEL sl2 WITH timeout = null;
Workload types
It's possible to declare a workload type for a service level, currently out of three available values:
unspecified - generic workload without any specific characteristics; default
interactive - workload sensitive to latency, expected to have high/unbounded concurrency, with dynamic characteristics, OLTP; example: users clicking on a website and generating events with their clicks
batch - workload for processing large amounts of data, not sensitive to latency, expected to have fixed concurrency, OLAP, ETL; example: processing billions of historical sales records to generate useful statistics
Declaring a workload type provides more context for Scylla to decide how to handle the sessions. For instance, if a coordinator node receives requests with a rate higher than it can handle, it will make different decisions depending on the declared workload type:
For batch workloads it makes sense to apply back pressure - the concurrency is assumed to be fixed, so delaying a reply will likely also reduce the rate at which new requests are sent;
For interactive workloads, backpressure would only waste resources - delaying a reply does not decrease the rate of incoming requests, so it's reasonable for the coordinator to start shedding surplus requests.
Example
ALTER SERVICE LEVEL sl WITH workload_type = 'interactive';
Load and stream
This feature extends nodetool refresh to allow loading arbitrary sstables
that do not belong to a particular node into the cluster. It loads the sstables from disk, calculates the data's owning nodes, and automatically streams the data to the owning nodes.
For example, say the old cluster has 6 nodes and the new cluster has 3 nodes.
We can copy the sstables from the old cluster to any of the new nodes and
trigger the load and stream process.
This can make restores and migrations much easier:
You can place sstables from any node from the old cluster to any node in the new cluster
No need to run nodetool cleanup to remove data that does not belong to the local node after refresh
Load_and_stream option also update the relevant Materialized Views #9205
Examples:
> Nodetool refresh --load-and-stream
> curl -X POST "http://{ip}:10000/storage_service/sstables/{keyspace}?cf={table}&load_and_stream=true
Reverse Queries
A reverse query is a query SELECT that uses a reverse order compared to the one used in the table schema. If no order was defined, the default order is ascending (ASC).
For example, the following table schema orders the rows in a partition by “time” in an ascending order:
CREATE TABLE heartrate (
pet_chip_id uuid,
owner uuid,
time timestamp,
heart_rate int,
PRIMARY KEY (pet_chip_id, time)
);
The following SELECT worked in Scylla 4.5 but might be very inefficient:
SELECT * FROM heartrate LIMIT 1 ORDER BY time DESC
Improving the performance of reverse query is an ongoing process, with the following updates in Scylla 4.6:
The internal layer for managing queries now supports reversed queries natively. This lays the groundwork for reversed reads in memtables, cache, and sstables, so that reversed queries will perform efficiently.
The mx sstable reader (reading mc and md format sstables) can now read partitions in reversed order. This is a step towards supporting reversed reads of large partitions.
Memtables now efficiently support reversed reads (for CQL WITH CLUSTERING ORDER). Together with the already merged sstable reversed reader, reversed reads with BYPASS CACHE are now more efficient, especially with memory consumption.
Scylla 4.6 Updates
Install, deployment and packaging
Deprecated: Ubuntu 16 support. Ubuntu 16.04 is EOL and will not be supported following 4.6
Newer Linux distributions use systemd-timesyncd instead of ntp or chrony for time synchronization. This is now supported. Scylla will now detect if you already have time synchronization set up and leave it alone if so. #8339
scylla_setup script now supports disabling the NVMe write-back cache on disks that allow it. This is useful to reduce latency on Google Cloud Platform local disks. The machine images built using scylla-machine-image will do this automatically.
The Unified (tarball) Installer now works correctly when SElinux is enabled. #8589
The docker image base has been switched from CentOS 7 to Ubuntu 20.04, similar to Scylla AMI, and GCP images. PR#8849
The installer now offers to set up RAID 5 on the data disks in addition to RAID 0; this is useful when the disks can have read errors, such as on GCP local disks. #9076
The install script now supports supervisord in addition to systemd. This was brought in from the container image, where systemd is not available, and is useful in some situations where root access is not available.
Automatic I/O configuration during setup now supports AWS ARM instances.#9493
The setup utility now recognizes Persistent Disks on Google Cloud Platform and Azure. PR#9395 PR#9417
Raft
We are building up an internal service in Scylla, useful for this and other applications. The changes have no visible effect yet. Among other, the following was added:
The Raft implementation now updates RPC about new and removed nodes.
It is now required to enable Raft with a configuration item (experimental: raft). This lets the implementation mature without requiring backwards compatibility efforts. #9239
The schema for storing Raft snapshots has been updated to avoid blobs.
CDC
Change Data Capture (CDC) now fills in the cdc$deleted_columns column in pre-image correctly
Alternator
Alternator is Scylla's implementation of the DynamoDB API.
Alternator: rudimentary implementation of TTL expiration service (PR #9624)
ConditionExpression wrong comparison of two non-existent attributes #8511
Incorrect set equality comparison inside a nested document #8514
Incorrect inequality check of two sets #8513
Alternator, ScyllaDB's implementation of the DynamoDB API, now supports the ADD operation. #5893
Guardrails
We continue to add default limitations (guardrails) to Scylla, making it harder for users to use non production settings by mistake. Each new configuration added to restriction mode (tri_mode_restriction) has three options:
True: restricted, disable risky feature
False: non restricted, enable risky feature
Warn: non restricted, log warning about risky feature
Additions in this release are:
It's now possible to prevent users from using SimpleReplicationStrategy, using config parameter restrict_replication_simplestrategy
The goal is to first default to warning and then default to actual prevention. SimpleReplicationStrategy can make it hard to later grow the cluster by adding data centers. #8586
DateTieredCompactionStrategy is deprecated in favor of TimeWindowCompactionStrategy for a long time now. A new warning will let you know if you are still using it. If you are nostalgic for the old strategy, use “restrict_dtcs” to disable this warning #8914
CQL
SELECT statements that used an index, and also restricted the token (e.g. SELECT ... WHERE some_indexed_column = ? AND token(pk) = ?) incorrectly ignored the token restriction. The issue was found by using spark connector filtering on a secondary index. This is now fixed. #7043
User-Defined Aggregates (UDA) have been implemented. User-defined aggregates allow the creation of custom aggregate functions, like count, min, and max. Note that UDA, like UDF are experimental and can be enable with enable_user_defined_functions parameter in scylla.yaml #7201
Selecting a partition range with a slice/equality restriction on clustering keys (e.g. "SELECT * FROM tab WHERE ck=?", with no partition key restrictions) now demands ALLOW FILTERING again (since this query can potentially discard large amounts of data without returning anything). To avoid breaking applications that accidentally did not specify ALLOW FILTERING, it will only generate a warning for now. #7608
Scylla now correctly rejects CQL insert/update statements with NULLs in key columns. #7852
Queries that are performed using an index can now select Static Columns. #8869
User Defined Functions and Aggregates (UDF/UDA) now support WebAssembly in experimental mode. The bindings to Scylla data types will likely change, but this is sufficient to play with.
Hinted Handoff API
Hinted Handoff is an anti-entropy mechanism to replay mutations to a node which was unreachable for some time.
A new HTTP API for waiting for hinted handoff replay to be completed. This can be used to reduce repair work.
`/hints_manager/waiting_point` (POST) - Create a sync point: given a set of target hosts, creates a sync point at the end of all HH queues pointing to any of the hosts.
`/hints_manager/waiting_point` (GET) - Wait or check the sync point: given a description of a sync point, checks if the sync point was already reached. If you provide a non-zero `timeout` parameter and the sync point is not reached yet, this endpoint will wait until the point reached or the timeout expires.
Scylla-sstable
A tool which can be used to examine the content of sstable(s) and
execute various operations on them. The currently supported operations
are:
dump - dumps the content of the sstable(s), similar to sstabledump;
dump-index - dumps the content of the sstable index(es), replacing scylla-sstable-index;
writetime-histogram - generates a histogram of all the timestamps in
the sstable(s);
custom - a hackable operation for the expert user (until scripting
support is implemented);
validate - validate the content of the sstable(s) with the mutation
fragment stream validator, same as scrub in validate mode;
Performance Improvements
Streams of mutation data are represented in Scylla by a flat_mutation_reader, which provides a means for a function to consume a stream. This was made faster, which improves operations such as flushing a memtable. See PR#8359 for micro benchmark results.
When Scylla receives sstables from outside the replica (such as via a repair operation, or after restoring a snapshot) it first reshapes them to conform to the compaction strategy.
Reshape was improved for:
Leveled Compaction Strategy, by checking first if the sstables happen to be disjoint.
Time Window Compaction Strategy, by reducing write amplification.
More code paths can now work with non-contiguous memory for table columns and intermediate values: comparing values, the CQL write path. This reduces CPU stalls due to memory allocation when large blobs are present. PR#8357
SSTable parser will avoid large allocations, reducing latency spikes. #6376, #7457
Repair is now delayed until hints for that table are replayed. This reduces the amount of work that repair has to do, since hint replay can fill in the gaps that a downed node misses in the data set. #8102
SSTables will now automatically choose a buffer size that is compatible with achieving good latency, based on disk measurements by iotune.
The setup scripts will now format the filesystem with 1024 byte blocks if possible. This reduces write amplification for lightweight transaction (LWT) workloads.
Performance: read latency increase after deletion of high percentage of the data, as many rows covered by a single range tombstone, which go through row cache are very slow #8626
Authentication had a 15 second delay, working around dependency problems. But it is long unneeded and is now removed, speeding up node start.
Unintended quadratic behavior in the log-structured allocator (which manages Scylla memtable and cache memory) has been fixed. #8542
Off-strategy compaction is now enabled for repair. After repair completes, the sstables generated by repair will first be merged together, then incorporated into the set of sstables used for serving data. This reduces read amplification due to the large number of sstables that repair can generate, especially for range queries where the bloom filter cannot exclude those sstables. #8677
Off-strategy compaction, a method by which sstables are reshaped to fit the compaction strategy, is now enabled for bootstrap and replace operation using standard streaming. #8820
The read path has been optimized to remove unnecessary work, leading to a small performance increase.
The common case of single-partition query was treated as an IN query with a 1-element tuple. This case is now specialized to avoid the extra post-processing work.
SSTable index files are now cached, both at the page level and at an object level (index entry). This improves large partition workloads as well as intermediate size workloads where the entire sstable index can be cached. #7079
The row cache behavior was quadratic in certain cases where many range tombstones were present. This has been fixed. #2581
Recently the sstable index has gained the ability to use the cache to reduce I/O; but it did so even when BYPASS CACHE was requested in the CQL statement. The index now respects BYPASS CACHE like data access.
After adding a node, a cleanup process is run to remove data that was copied to the new node. This is a compaction process that compacts only one sstable at a time. This fact was used to optimize cleanup. In addition, the check for whether a partition should be removed during cleanup was also improved. #6807
Scylla uses reader objects to read sequential data. It caches those readers so they can be reused across multiple pages of the result set, eliminating the overhead of starting a new sequential read each time. However, this optimization was missed for internal paging used to implement aggregations (e.g. SUM(column)). Scylla now uses the optimization for aggregates too. 9127
There is now an effective replication map structure, which contains the application of a replication strategy and its parameters to a topology (node->token mapping). This reduces the amount of run-time computation needed by the coordinator.
Time Window Compaction strategy reshape gained two optimizations. Reshape happens when changing compaction strategies or after streaming data to a new node. The optimizations reduce write amplification and therefore the time spent when adding a new node.
In Time Window Compaction Strategy compactions, fully expired sstables will be compacted separately, since that can be done by just dropping them. #9533
Scrub compaction, which re-sorts unsorted sstables, will now use memtables as a sorting mechanism instead of generating many small sstables. PR#9548
Usually a memtable is flushed into one sstable. In some cases, however, it can be flushed into several sstables. We now adjust the partition estimate for these sstables so the bloom filters allocated by these sstables will not occupy too much memory. #9581
Size-tiered compaction strategy will prefer compactions with larger fan-in in order to improve efficiency. Moreover, once a compaction with large fan-in is started, compactions with lower fan-in will be delayed in order to improve overall write amplification.
Major compaction will now process tables from smallest to largest, to increase the probability of success in case the node is running low on space.
Stability Improvements
Thrift has now partial admission control, to reduce the chance of server overload.
A recent regression caused requests to data centers where the local replication factor is zero to crash. This is now fixed. #8354
A bug in Time Window Compaction Strategy's selection of sstables for single-partition reads caused sstables that did not have data for the key to be consulted, reducing performance. This is now fixed. #8415
Continuing on the path of allowing non-contiguous allocations for large blobs, memory linearizations have been removed from Change Data Capture (CDC). This reduces CPU stalls when CDC is used in conjunction with large blobs. #7506
The underlying json parser used by alternator, rapidjson, has been hardened against out-of-memory errors. #8521
A bug in the row cache that can cause large stalls on schemas with no clustering key has been fixed.
Scylla uses a log-structured memory allocator (LSA) for memtable and cache. Recently, unintentional quadratic behavior in LSA was discovered, so as a workaround the memory reserve size is decreased. Since the quadratic cost is in terms of this reserve size, the bad behavior is eliminated. Note the reserves will automatically grow if the workload really needs them. #8542
Repair allocates working memory for holding table rows, but did not consider memory bloat and could over-allocate memory. It is now more careful. #8641
Failure detection is now done directly by nodes pinging each other rather than through the gossip protocol. This is more reliable and the information is available more rapidly. Impact on networking is low, since Scylla implements a fully connected mesh in all clusters smaller than 256 nodes per datacenter, much larger than the typical cluster. #8488
Change Data Capture (CDC) uses a new internal table for maintaining the stream identifiers. The new table works better with large clusters.#7961
Scylla will now close the connection when a too-large request arrives; previously Scylla would read and discard the request. The new behavior protects against having to read and discard potentially gigabytes of data. #8798
A SELECT statement could result in unbounded concurrency (leading to an out-of-memory error) in some circumstances; this is now fixed. #8799
Speculative retry will now no longer consider failed responses when calculating the 99th percentile of a request. Doing so let to the latency threshold being continuously raised, since failed (timed out) requests have very long latency; so the effectiveness of the feature is much reduced. Now it will only consider successful responses. #3746
#7342The bootstrap process was made more robust over failures to communicate the token ranges to the new node. Such problems will be resolved more fully with Raft. #8889
A regression in how Scylla computes compaction backlog has been fixed. Compaction backlog estimates the total number of bytes that remain to be compacted (which can be greater than the table size due to write amplification). Scylla then uses the backlog to decide how to allocate resources to compaction. The bug caused the backlog to be overestimated, thus devoting too many resources (CPU and disk bandwidth) to compaction #8768
Logging of corrupted sstables has been improved to report the file name and offset of the corrupted chunk.
A problem with COMPACT STORAGE tables representation in Change Data Capture has been corrected. #8410
The reader concurrency semaphore is Scylla's main replica-side admission control mechanism. It tracks query memory usage and timeouts, and delays or rejects queries that will overload memory. Previously, it was engaged only on a cache miss. It is now engaged before the cache is accessed. This prevents some cache intensive workloads from overloading memory. These workloads typically have queries that take a long time to execute and consume a lot of memory, even from cache, so their concurrency needs to be limited. #4758 #5718
A limitation of 10,000 connections per shard has been lifted to 50,000 connections per shard, and made tunable with max-networking-io-control-blocks seastar config. #9051
The currenttime() and related functions were incorrectly marked as deterministic. This could lead to incorrect results in prepared statements. They are now marked as non-deterministic. #8816
If Scylla stalls while reclaiming memory, it will now log memory-related diagnostics so it is easier to understand the root cause.
Repair now reads data with a very long timeout (instead of infinite timeout). This is a last-resort defence against internal deadlocks. #5359
The cache abstraction that is used to implement the prepared statement cache has been made more robust, eliminating cases where the entry expired prematurely. #8920
When streaming data from a Time Window Compaction Strategy table, Scylla segregates the data into the correct time window to follow the compaction strategy rules. This however causes a large number of sstables to be created, overloading the node in extreme cases. We now postpone data segregation to a later phase, off strategy compaction, which happens after the number of sstables has been reduced. #9199
Scrub compactions no longer purge tombstones. This is in order to give a tombstone hiding in a corrupted sstable the chance to get compacted with data.
Scrub compactions are now serialized wrt. other compactions, to prevent an sstable undergoing compaction from not being scrubbed. #9256
The CQL protocol server now responds with an overloaded exception if it decides to shed requests. This avoids a resource leak at the client driver side, when the driver may be waiting for the response indefinitely. #9442
A scrub compaction takes an sstable with corruption problems and splits it into non-corrupted sstables. This can result in an sstable being split into a large number of new sstables. In this case the bloom filter of the new sstables could occupy too much memory and run the node out of memory. We now reduce the bloom filter size dynamically to avoid this. #9463
Infrastructure
Infrastructure for a new style of virtual tables has been merged. While Scylla already supported virtual tables, it was hard to populate them with data. The new infrastructure reuses memtables as a simple way to populate a virtual table. #8343
The code to write ka/la format sstables has been removed. Scylla can now only write mc/md format sstables. Note that Scylla can still read ka/la format sstables. #8352
The commitlog code has been converted to coroutines in order to simplify further work. PR#8954
Scylla uses a home-grown Interface Definition Language (IDL) to help preserve compatibility during upgrades. It now supports generating RPC verbs, not just data types.
Security
Thrift listen port (start_rpc) is disabled by default, as it is not often used. Thrift is still fully supported. #8336
Configuration
New and updated configuration changes:
experimental - deprecated. The flag used to enable all experimental features, which is not very useful. Instead, you should enable experimental features one by one. Experimental feature in this release are: possible values: 'lwt', 'cdc', 'udf', 'alternator-streams', 'raft' #9467
enable_sstables_mc_format: parameter is now ignored; mc or md format is mandatory. Scylla continues to be able to read older ka and la format sstables. #8352
Batch_size_warn_threshold_in_kb: from 5 to 128, Batch_size_fail_threshold_in_kb: from 50 to 1024
The batch size warning and failure thresholds have been increased to 128 kiB and 1 MiB respectively. The original limits (5 and 50 kiB) were unnecessarily restrictive. #8416
Max_hinted_handoff_concurrency: when a node is down, writes to that node are collected in hint files. When the node rejoins, hint files are replayed. It turned out that hint replay was too fast and interfered with the normal workload, so its concurrency was reduced and made configurable.
restrict_dtcs(false): see restricted above
experimental::alternator-ttl see above
reversed_reads_auto_bypass_cache(true) : Use new implementation of reversed reads in sstables when performing reversed queries
replace_node and replace_token were ignored, and node removed
flush_schema_tables_after_modification(true): Flush tables in the system_schema keyspace after schema modification. This is required for crash recovery, but slows down tests and can be disabled for them
strict_allow_filtering(warn): Match Cassandra in requiring ALLOW FILTERING on slow queries. Can be true, false, or warn. When false, Scylla accepts some slow queries even without ALLOW FILTERING that Cassandra rejects. Warn is the same as false, but with warning.
commitlog_use_hard_size_limit(false): Whether or not to use a hard size limit for commitlog disk usage. Default is false. Enabling this can cause latency spikes, whereas the default can lead to occasional disk usage peaks.
enable_repair_based_node_ops(true): Set true to use enable repair based node operations instead of streaming based (more on RBNO above)
allowed_repair_based_node_ops(replace): A comma separated list of node operations which are allowed to enable repair based node operations. The operations can be bootstrap, replace, removenode, decommission and rebuild (more on RBNO above)
sanitizer_report_backtrace(false): In debug mode, report log-structured allocator sanitizer violations with a backtrace. Do not use this in production, as is slow Scylla significantly.
restrict_replication_simplestrategy(false): see restricted above
failure_detector_timeout_in_ms(20,000ms): Maximum time between two successful echo message before gossip mark a node down in milliseconds
wait_for_hint_replay_before_repair(true): If set to true, the cluster will first wait until the cluster sends its hints towards the nodes participating in repair before proceeding with the repair itself. This reduces the amount of data needed to be transferred during repair.
Build
The command "ninja help" will now list available targets when building Scylla. PR#8454
It is now possible to build Scylla under the Nix package manager.
There is a new process for building docker images, based on the buildah tool. The process makes it easier to build outside the continuous integration pipeline (e.g. for individual developers). PR#8730
The toolchain used to build Scylla is now based on Fedora 34, with the clang 12 compiler.
Testing
The performance benchmark perf_simple_query now returns the number of allocations performed and tasks executed per query. These metrics are less noisy than raw performance and so can be used to quantify improvements more easily.
There is now rudimentary support for code-coverage reports in unit tests.
The perf_simple_query benchmark now reports how many instructions were executed by the CPU per query.
Scylla comes with extensive integration with gdb, to make inspecting code dumps easier. This infrastructure now has a unit test.
A new type of compaction, validation. A validation compaction will read all sstables and perform some checks, but write nothing. This is useful to make sure all sstables can be read and pass sanity checks. #7736
Tooling
sstablemetadata: add support to inspect sstable using zstd compression #8887
CQLsh use the Scylla Python Driver instead of the Cassandra driver
There is now a crawling sstable reader, that does not use the sstable index file. This is useful in scenarios such as scrub, when the index file is suspected to be corrupted.
When rewriting sstables (e.g. for the nodetool upgradesstables command), Scylla will now abort the rewrite if shutting down.
Infrastructure
There is a new internal keyspace, "system_distributed_everywhere", which is used to propagate internal information that needs higher consistency and bandwidth than gossip. The first user will be Change Data Capture internal data. PR#8457
The redis protocol server relied on code that was copy-pasted from the cql transport server. The two implementations are now unified into a generic tcp server.
The internal representation of range tombstones is changing. Instead of a {start, end} pair, after which rows that may be affected by the range tombstone (and rows that are not) are emitted, we emit a {start} marker, then all affected rows, then the {end} marker. This fits well with 'm' format sstables, and once the work is complete will improve workloads which have many range tombstones.
The sstable parsers have been converted from a state machine to a C++ coroutine. There should be no use-visible effect, but the code complexity has been significantly reduced, from a tangle of switches and gotos to structured loops.
The system.status and system.describe_ring virtual tables have been renamed to the more descriptive and conventional names system.cluster_status and system.token_ring respectively. Since the tables are new, users are not affected.
Additional bug fixes
Stability: LWT sporadic failure if a non-deterministic function is used to assign partition key #8604
Monitoring: API uses incorrect plus<int> to sum up cf.active_memtable().partition_count(), which can result with the value wrapped around if bigger than 232, and return the wrong metric. #9090
CQL: Multi column restrictions ignored on indexed queries #9085
Stability: Reads which page due to memory and need to reconcile may miss writes in the results #9119
Packaging: Docker run arguments not passed to scylla #9247
A performance regression in the memtable reader was fixed. The regression was introduced when adding support to reverse queries. #9502
UX: large data warnings do not contains the sstable name #9524
All bug fixed in this release (from Git log)
More sstable code need a close() method to avoid use-after-free bugs #1076
Make result of trichotomic comparison not convertible to bool #1449
Cache reads generate more range tombstones than necessary #2581
Change flat_mutation_reader::is_end_of_stream() to reflect the state of the consumer side #3067
Speculative reads based on read latency profile may cease to be effective temporarily #3746
Request for enhancement nodetool toppartitions - need to be more general #4520
cache-mostly read workloads can overload memory #4758
hints_manager - Exception when draining 10.0.63.76: std::filesystem::__cxx11::filesystem_error (error system:39, filesystem error: remove failed: Directory not empty [/var/lib/scylla/hints/3/10.0.63.76] #5087
reader_concurrency_semaphore: admit new reads only when all current ones are stalled on I/O #5718
alternator - missing support for UpdateItem ADD operation #5893
Error applying view update (Operation timed out for mview...) #6187
large allocation when calculating schema digest with 5000 stables #6376
Optimize partition filtering of cleanup compaction #6807
scylla vs cassandra compatibility issue (spark connector filtering on secondary index) #7043
Share sstable index page cache between readers #7079
UDA creation error #7201
Transform mutation data into result_row_view when invoking expr::is_satisfied_by() #7215
C-s latency peaks up to 12 s with OPS fall down from 80k to 300 when scylla killed on one node. #7342
Avoid large contiguous allocation for large cells in the sstable parser / writer #7457
cdc code linearizes all values #7506
Restriction on clustering-key only should require ALLOW FILTERING #7608
New need_filtering() returns false for a query that needs filtering #7708
nodetool scrub: add dry run flag #7736
Provide an option to bypass reshape on boot #7738
Add metrics for the amount of range tombstones processed during read #7749
Azure Ls v2 local disk setup #7807
Inserting a row with a null key column should be forbidden #7852
Frozen nested set may be created out-of-order via a prepared statement #7856
Introduce service levels #7867
Add per-service-level timeouts #7913
docs: moved latest_version to conf.py #7957
large allocation in untyped_result_set reading cdc generations (seastar_memory - oversized allocation: 2023424 bytes. This is non-fatal, but could lead to latency and/or fragmentation issues) #7961
Cassandra stress fails to achieve consistency during replace node operation #8013
scylla_io_setup on GKE: found more than one disk mounted at root #8032
gossip: Introduce direct failure detector #8036
Replaying hints before repair #8102
Some IDL compiler cleanups #8148
scylla_setup: set XFS block size to 1k #8156
cql3: remove linearizations in the write path #8160
sstables: remove large allocations when parsing cells #8182
If ALLOW FILTERING excludes a long string of rows, the scan can stop prematurely #8203
expr: give a name to a tuple of columns #8208
sstableloader doesn't work with Alternator tables if "-nx" option is used #8230
The LWT stress is stopped during the prepare phase #8270
var-lib-scylla.mount mount process failed: can't find UUID #8279
CDC generations: refactors and improvements #8283
storage_service/removenode: update gossiper state before excise #8284
dist/common/scripts/scylla_util.py doesn't support Azure images #8325
repair: Switch to use NODE_OPS_CMD for replace operation #8330
scylla_ntp_setup: Use systemd-timesyncd instead of chrony when it's installed by default #8339
Add infrastructure for virtual tables #8343
- Warning: Could not get data for |8344| on repo scylla (tried: #8344)
Retire la/ka writer #8352
Unable to initialize new nodes on scylla 4.3.2: what(): Not an IPv4 address #8354
config: ignore enable_sstables_mc_format flag #8360
Commitlog can get stuck after reaching disk size limit, causing writes to time out #8363
dist: hardcoded product name on scylla_setup #8367
commitlog: Fix race and edge condition in delete_segments #8372
handle commitlog recycle errors #8376
Stuck server while upgrading to 4.4 from 4.3.2 #8378
listen-address flag description saying `Never specify 0.0.0.0; it is always wrong.` is wrong #8381
Tracing can remain not shut down if start is aborted #8382
scylla_setup: stop hardcode product name on scylla_setup #8384
nonroot: when /var/log/journal exists, install.sh does not generate scylla_sysconfdir.py correctly #8385
nonroot: generate scylla_sysconfdir.py correctly #8386
hints: use token_metadata to tell if node has left the ring #8387
Unify CQL and Redis server code #8388
install.sh: document pathname components #8389
Spurious EINVAL errors from TLS sockets #8391
utils: fragment_range: fix FragmentedView utils for views with empty fragments #8397
Infinite loop in write_fragmented for empty fragments in non-empty views #8398
composite: replace enable_if with constraints #8399
treewide: don't include "db/system_distributed_keyspace.hh" from headers #8403
types: convert has_empty predicate to a concept #8404
utils: rjson: convert enable_if to concept #8405
caching_options: detemplate from_map() #8406
caching_options.hh: move code to .cc #8408
config: correct description of listen_address #8409
coredump during requesting cdc log table #8410
"Reader was forwarded before returning partition start" Exception using TWCS in 4.4 #8411
Add a (failing) test for picking secondary indexes in order #8414
`TimeWindowCompactionStrategy` not using specialized reader for single partition queries #8415
Annoying warnings for small batches #8416
config: relax batch size warning and failure thresholds #8417
Reject replacing a node that has left the ring #8419
storage_service: Reject replacing a node that has left the ring #8420
test: perf: perf_fast_forward: report allocation rate and tasks #8422
test: perf: perf_simple_query: collect allocation and task statistics #8425
cdc: log: avoid linearizations #8429
build: drop lld from install-dependencies.sh on s390x #8430
scylla_ntp_setup: detect already installed ntp client #8431
Optmized TWCS single-partition reader opens sstables unnecessarily #8432
sstables: fix TWCS single key reader sstable filter #8433
`cache_flat_mutation_reader` position-fast-forwards readers that are not inside a partition, if they immediately return eof #8435
stream_session uses escaped curly braces in format strings #8436
Make sure that cache_flat_mutation_reader::do_fill_buffer does not fast forward finished underlying reader #8437
Name resolution with tcp fails #8442
dist/common/scripts/scylla_util.py: GCP nvme: fail silently if you ca… #8444
`clustering_order_reader_merger` may immediately return end-of-stream if some (but not necessarily all) underlying readers are empty #8445
clustering_order_reader_merger: handle empty readers #8446
`time_series_sstable_set::create_single_key_sstable_reader` may return an empty reader even if the queried partition exists (in some other sstable) #8447
time_series_sstable_set: return partition start if some sstables were ck-filtered out #8448
Avoid two compaction passes when reshaping STCS table for repair-based operations #8449
utils: data_input: replace enable_if with tightened concept #8450
hints: delay repair until hints are replayed #8452
Add a ninja help build target #8454
artifacts-centos7-test: scylla_util.py fails with - ModuleNotFoundError: No module named 'scylla_sysconfdir' #8456
sys_dist_ks: new keyspace for system tables with Everywhere strategy #8457
tracing: test/boost/tracing: fix use after free #8461
mislicensed source files #8465
treewide: correct mislicensed source files #8466
service level controller is used after stop #8468
qos: make sure to wait for service level updates on shutdown #8470
Switch to use NODE_OPS_CMD for decommission operation #8471
Switch to use NODE_OPS_CMD for bootstrap operation #8472
High latency during bootstrap, background reclaim ineffective #8473
main: start background reclaim before bootstrap #8474
commitlog_test: Add test for deadlock in shutdown w. segment wait #8475
gossip: Relax failure detector update #8476
Cant Install using sites default install pkg #8479
Switch to use NODE_OPS_CMD for decommission and bootstrap operation #8481
Ubuntu AMI: Scylla server isn't up after reboot #8482
unified: abort install when non-bash shell detected #8484
Improve validation of "enable", "postimage" and "ttl" CDC options #8486
Introduce direct failure detector #8488
reader_concurrency_semaphore: readmission leaks resources #8493
scylla_raid_setup: Ubuntu does not able to enable mdmonitor.service #8494
dist: add DefaultDependencies=no to .mount units #8495
bootstrap_with_repair badly formatted exception: expected 1 node losing range but found more nodes={} #8503
repair: Handle everywhere_topology in bootstrap_with_repair #8505
main: add a debug symbol for service level controller #8506
Potential data corruption when reading range tombstone from MC sstable #8507
database::get_reader_concurrency_semaphore() uses system semaphore as the catch-all semaphore #8508
Materialized views: fix possibly old views coming from other nodes #8509
alternator ConditionExpression wrong comparison of two non-existent attributes #8511
scylla_io_setup: configure "aio-max-nr" before iotune #8512
Alternator incorrect inequality check of two sets #8513
Alternator - incorrect set equality comparison inside a nested document #8514
doc: Update bootstrap with everywhere_topology #8518
rapidjson's default allocator does not handle allocation failures properly #8521
tools: toolchain: dbuild: define die() earlier #8523
treewide: remove inclusions of storage_proxy.hh from headers #8524
Calculate partition ranges from expr::expression #8525
dist: Add support for disabling writeback cache #8526
dist/debian: when PRODUCT != scylla, scylla-node-exporter.default does not added with node exporter package #8527
dist/debian: rename .default file correctly #8528
rjson: Add throwing allocator #8529
scylla_raid_setup: enabling mdmonitor.service on Debian variants #8530
Reshape may ignore overlapping in level L where L > 0 #8531
Cannot bootstrap a node in presence of Everywhere strategy tables with RBO enabled #8534
token_metadata: Fix get_all_endpoints to return nodes in the ring #8536
repair: remove partition_checksum and related code #8537
test: make rjson allocator test working in debug mode #8539
Forward-port service level fixes #8540
logalloc: reclaim_from_evictable evicts up to 3(N^2) segments when asked to reclaim N segments #8542
Added r5b to ena instance_class. #8546
Alternator tracing does not include valid user info #8547
Add username to alternator tracing #8548
test: perf: don't truncate allocation/req and tasks/req report #8550
gdb: Fix heapprof() dereferencing of backtrace #8551
Huge reactor stalls in `token_metadata` during topology changes when `EverywhereStrategy` tables are present #8555
sstables: Add debug info when create_sharding_metadata generates zero ranges #8557
messaging service: be more verbose when shutting down servers and clients #8560
test: perf: report instructions retired per operations #8563
Tests for Alternator's TTL feature #8564
cql3: Check if partition-key restrictions are all EQ at preparation time #8565
repeated message: service_level_controller - service level default was deleted #8567
cdc: log: fill cdc$deleted_ columns in pre-images #8568
TWCS: sometimes SSTables are not compacted together when time window finishes #8569
logalloc: reduce minimum lsa reserve in allocating_section to 1 #8572
Off-strategy compaction with LCS keeps reshaping the last remaining sstable #8573
workload prioritization: Reduce the logging sensitivity to "glitches" in #8574
Service level controller: fix wrong default service level removal log #8576
Scylla shutdown process stuck when Shutting down drain storage proxy #8577
commitlog: make_checked_file for segments, report and ignore other errors on shutdown #8578
Reactor stall in consume_mutation_fragments_until #8579
flat_mutation_reader: consume_mutation_fragments_until: maybe yield for each popped mutation_fragment #8580
cql/cdc_batch_delete_postimage_test - rename test files + fix result #8581
STCS get_buckets insertion algorithm may break the bucket invariant #8584
dist: scylla_raid_setup: reduce xfs block size to 1k #8585
Add option to forbid SimpleStrategy in CREATE KEYSPACE #8586
scylla_io_setup failed with error: seastar - Could not setup Async I/O on aws instances (r5, r5b) and gp3 ebs volumes #8587
Unified Installer: Incorrect file security context cause scylla_setup to fail #8589
TWCS: initialize _highest_window_seen #8590
storage_proxy::get_max_result_size() uses the unlimited max-size for service levels #8591
storage_proxy: use small_vector for vectors of inet_address #8592
Support Microsoft Azure snitch #8593
scylla_util.py: Fix Azure support for machine-image #8596
node doesn't switch status to UP and normal after scylla had started on node, which was previously drained #8597
IndexInfo system table lists MV name instead of index name #8600
install.sh: apply correct file security context when copying files #8602
LWT sporadic failure if a non-deterministic function is used to assign partition key #8604
storage_proxy: place unique_response_handler:s in small_vector instead of std::vector #8606
repair: Wire off-strategy compaction for decommission #8607
wild pointer dereference during shutdown after failing to create sstable component #8609
storage_service isolate log message should be an error #8610
logalloc: fix quadratic behaviour of reclaim_from_evictable #8611
phased_barrier move-assignment may leak a gate entry #8613
storage_service: Delay update pending ranges for replacing node #8614
gossip: add local application state excess logging #8616
Handle excess logging of non existing local endpoint #8617
scylla-gdb unit test #8618
Creating a table which looks like a secondary index breaks the secondary idnex creation mechanism #8620
cql-pytest: add nodetool flush feature and use it in a test #8622
Overlapping range tombstones can lead to very slow reads and OOMs #8625
Reads of many rows covered by a single range tombstone which go through row cache are very slow #8626
additional sstable stats (#251) #8630
Fix index name conflicts with regular tables #8632
Description in scylla-fstrim.timer is incorrect #8633
db/virtual tables: Add infrastructure + system.status example table #8634
exception phased_barrier::advance_and_await is not handled #8636
consistency_level: deinline assure_sufficient_live_nodes() #8637
repair: Consider memory bloat when calculate repair parallelism #8641
cdc: use a new internal table for exchanging generations #8643
scylla-fstrim.timer: fix wrong description from 'daily' to 'weekly' #8644
Fix service level negative timeouts #8645
hints: make hints concurrency configurable and reduce the default #8646
abstract_replication_strategy: avoid reactor stalls in `get_address_ranges` and friends #8647
Scylla unconditionally waits 15 seconds for auth to be initialized, which is wasteful #8648
auth: remove the fixed 15s delay during auth setup #8649
install.sh:set aio conf during installation #8650
Abort restore_replica_count when node is removed from the cluster #8651
repair: Consider memory bloat when calculate repair parallelism #8652
storage_service: Abort restore_replica_count when node is removed from the cluster #8655
perf: add alternator frontend to perf_simple_query #8656
Exceptions in resharding and reshaping are being incorrectly swallowed #8657
perf_fast_forward: report instructions per fragment #8660
unified/uninstall.sh: simplify uninstall.sh, delete all files correctly #8662
nonroot installation broken at 4.5.rc1 #8663
install.sh: fix not such file or directory on nonroot #8664
tree-wide: comments on deprecated functions to access global variables #8665
Mismatched types for base and view columns id: timeuuid and timeuuid #8666
Fix type checking in index paging #8667
cql3: represent lists as chunked_vector instead of std::vector #8668
row_cache_test: test_partition_range_population_with_concurrent_memtable_flushes fails with segmentation fault #8671
gdb: bypass unit test on non-x86 #8672
Ninja build feature request: Set packages to be created with architecture name #8675
Wire off-strategy compaction for regular repair #8677
repair: Wire off-strategy compaction for regular repair #8678
cql3: result_set: switch rows to chunked_vector #8679
Introduce per-service-level workload types and their first use-case - shedding in interactive workloads #8680
scylla_raid_setup: use /dev/disk/by-uuid to specify filesystem #8681
Enable -Wunused-private-field warning #8683
types: remove some dead code #8684
types: fix a case of type punning via union #8685
keys, compound: eliminate some careless copies of shared pointers #8686
alternator: executor: eliminate some pointless reserializations #8687
keys, compound: take the argument to from_single_value() by reference #8688
types: don't linearize values in abstract_type::hash #8689
collection_mutation: don't linearize collection values #8690
Alternator's health-check request doesn't work properly with HTTPS #8691
test: serialized_action_test: prevent false-positive timeout in test_phased_barrier_reassignment #8692
Segfault during forced repair termination #8693