40ms delay issue for TCP ACK DELAY happened on 10% requests

[Cong Guo]

Hi, I met p99 latency delay days before. After digging into the whole path one by one, I found the delay is caused in the network phase on some packets with specific length (< MSS).

You can see the snapshot in the link below.

1. Client (ddb API) send http request to scylla for read.
2. Scylla sends back 1 TCP segment of size 8258 but stops sending the remaining end segment.
3. Client waits for 40ms (seems the ack timeout) and reply the ack to scylla.
4. Scylla send the end tcp segment client.
5. Client ACK.

https://files.slack.com/files-pri/T02B9QAPR-F02LWD0DVEZ/image.png

In other data item scenario, scylla keeps sending TCP segment no matter received ACK or not. Don't understand why scylla hold the last TCP Segment for Client's ACK.

[Cong Guo]

uploaded the snapshot with bad read v.s good read.

[Cong Guo]

The MSS size is 8961B.

[Cong Guo]

Wonder if use cql via Scylla Driver would met the same issue or not?

[Cong Guo]

I just thought if scylla disabled Nagle's algorithm, then we should not meet such issue.

[Cong Guo]

I am sure the Nagle's algorithem is disabled in AWS ddb sdk, for I printed it to the stdout.

cguo: test3

cguo: test2

cguo: test1

cguo: 111 set TcpNoDelay true

[Nadav Har'El]

I think I know where the problem is, and opened https://github.com/scylladb/scylla/issues/9619 and also https://github.com/scylladb/seastar/issues/968 for the underlying Seastar library.

If we fix the problem, would you be able to test the result? How do you consume Scylla - can you run a nightly version if it contains a fix?

--
Nadav Har'El

n...@scylladb.com

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/061cdf0a-8aa4-4bb3-8079-224d7b89b650n%40googlegroups.com.

[Cong Guo]

Hi Nadav, 

Sure, I can do that, it's much better if the nightly version support upgrade from 4.5.1. Wonder when will it be able?

Thanks,

Cong

[Avi Kivity]

You can upgrade from 4.5.1 (or any version) to the nightly.

In general, you can upgrade from the nightly to a later release (e.g. 4.7.0; 4.6 was already branched), but we do not guarantee that it will be possible. The nightly will of course be less stable than release branches.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/dd8d321b-94f2-4afd-8c43-791dd2d96361n%40googlegroups.com.

[Cong Guo]

Thank you.