tombstone problem

[Mathieu Dutour]

We have a problem with the tombstone in the database.

We can trigger it in a reproducible way by the following way:

Step 1: Insert 200000 key/values.

Step 2: Delete 100000 keys selected at random in those.

Step 3: Then read the remaining data.

The tombstone problem occurs with the third step. Note that

we are using the "query_paged" command to read the data.

If this can help, we can provide a short rust program that triggers

the problem.

  Mathieu

[Avi Kivity]

What is the problem you see?

What ScyllaDB version are you using?

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/9c04afa6-b46e-45b5-ad40-44680f5c6ccen%40googlegroups.com.

[Mathieu Dutour]

The problem I see is the error
"Tombstones processed by unpaged query exceeds limit of 10000"
this is in direct contradiction with my use of "query_paged" in
my code.

About the version, I cannot tell since I am using the docker
and it does not have a version attached to it.

But I can see from "docker image inspect scylladb/scylla"
        "Id": "sha256:d75cb647338013684c7c098742c07fdd7f9d473d5922c60f0e9745c27b242e0f",
        "Created": "2023-12-31T09:50:22.830874168Z",

[Avi Kivity]

You can run 'docker run --rm --entrypoint /usr/bin/scylla scylladb/scylla --version' to get the version.

It's recommended to pull an explicit tag, not whatever random version happens to be at the time you pull.

[Mathieu Dutour]

The version is 5.4.1-0.20231231.3d22f42cf9c3

[Mathieu Dutour]

The version is 

5.4.1-0.20231231.3d22f42cf9c3

On Wednesday 7 February 2024 at 17:52:25 UTC+1 Avi Kivity wrote:

[Avi Kivity]

You can work around it by changing the config item query_tombstone_page_limit.

See also https://github.com/scylladb/scylladb/issues/17241.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/cb1c3ff2-ab13-4c54-863b-6633297256a5n%40googlegroups.com.

[Mathieu Dutour]

Yes, I know that.

But that is not the point.

The error message says "Tombstones processed by unpaged query exceeds limit of 10000"

But I am using "query_paged".

Therefore the error should not happen.

So either:

* The error message is incorrect.

* There is another bug in the querying system.

  Mathieu

[Avi Kivity]

Well, that shouldn't happen.

It could be that the server converted a paged query to an unpaged query.

What was the query string?

[Mathieu Dutour]

Thank you for the agreement.

The queries used are:

"SELECT k FROM kv.pairs WHERE dummy = 0 AND k >= ? AND k < ? ALLOW FILTERING"

Before that for inserting the batch, the operations are:

"INSERT INTO kv.pairs (dummy, k, v) VALUES (0, ?, ?)"

"DELETE FROM kv.pairs WHERE dummy = 0 AND k = ?"

An integrated test that shows up exactly the problem is illustrated

on https://github.com/MathieuDutSik/minimal_example_tombstone_pb

It works for less than 10000 deletion and it crashes for more than 10000

deletions because of the tombstone problem.

[Avi Kivity]

Botond, how does this fit with #17241? That addressed unpaged queries but here we see the problem for paged queries.

Did we lose the paged/unpaged flag somehow?

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/3e03e926-3da7-472c-b4b6-e28db964da93n%40googlegroups.com.

[Avi Kivity]

Well wireshark shows the problem, the query is actually unpaged (see
the Page Size flag).

Cassandra CQL Protocol
Version: 0x04
.... 0100 = Protocol version: 4
0000 .... = Direction: Request (0x0)
Flags: 0x00
.... ...0 = Compression: False
.... ..0. = Tracing: False
.... .0.. = Custom Payload: False
.... 0... = Warning: False
0000 .... = Reserved: 0x0
Stream Identifier: 0
.000 0111 = Opcode: QUERY (7)
Message Length: 96
Query
String Length: 75
String: SELECT k FROM kv.pairs WHERE dummy = 0 AND k >= ? AND k
< ? ALLOW FILTERING
Consistency: LOCAL_QUORUM (0x0006)
Flags: 0x11, Values, Serial Consistency
.... ...1 = Values: True
.... ..0. = Skip Metadata: False
.... .0.. = Page Size: False
.... 0... = Paging State: False
...1 .... = Serial Consistency: True
..0. .... = Default Timestamp: False
.0.. .... = Names for Values: False
0... .... = Reserved: 0x0
Value count: 2
Bytes length: 1
Bytes: 00
Bytes length: 1
Bytes: 01
Consistency: LOCAL_SERIAL (0x0009)



Command: tshark -V -i lo port 9042




So either query_paged doesn't do what one expects, or something else
went wrong.

[Karol Baryla]

`query_paged` in Rust driver is actually a bit misleading name - I opened an issue about it now: https://github.com/scylladb/scylla-rust-driver/issues/940

The only difference between `query()` and `query_paged()` is that the latter accepts `paging_state` argument.

Whether the query is paged or not is actually controlled by `page_size` attribute on `Query` struct. Look into those methods:

```

Query::with_page_size()

Query::set_page_size

Query::disable_paging

Query::get_page_size

```

Also, do you know about `query_iter()` method on `Session`? This is easier way to perform paged selects as you won't need to fetch next pages manually, driver will take care of it.

It's also not possible to forget about enabling paging using `query_iter` because it will set the default page size (5000) if none is set.

[Avi Kivity]

It would be better to have two layers for this, a general purpose one that hides paging, and a low-level one that exposes the paging mechanics.

Meanwhile I recommend to add better names and deprecate the misleading names.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/250d0c2d-d308-467a-b705-838f6fdc3c61n%40googlegroups.com.