40ms delay issue for TCP ACK DELAY happened on 10% requests

144 views
Skip to first unread message

Cong Guo

<cong.guo@smartnews.com>
unread,
Nov 11, 2021, 4:47:03 AM11/11/21
to ScyllaDB users
Hi, I met p99 latency delay days before. After digging into the whole path one by one, I found the delay is caused in the network phase on some packets with specific length (< MSS).

You can see the snapshot in the link below.

1. Client (ddb API) send http request to scylla for read.
2. Scylla sends back 1 TCP segment of size 8258 but stops sending the remaining end segment.
3. Client waits for 40ms (seems the ack timeout) and reply the ack to scylla.
4. Scylla send the end tcp segment client.
5. Client ACK.

https://files.slack.com/files-pri/T02B9QAPR-F02LWD0DVEZ/image.png

In other data item scenario, scylla keeps sending TCP segment no matter received ACK or not. Don't understand why scylla hold the last TCP Segment for Client's ACK.

Cong Guo

<cong.guo@smartnews.com>
unread,
Nov 11, 2021, 4:54:19 AM11/11/21
to ScyllaDB users
uploaded the snapshot with bad read v.s good read.
Scylla 40ms delay issue.pdf

Cong Guo

<cong.guo@smartnews.com>
unread,
Nov 11, 2021, 4:56:20 AM11/11/21
to ScyllaDB users
The MSS size is 8961B.

Cong Guo

<cong.guo@smartnews.com>
unread,
Nov 11, 2021, 5:11:10 AM11/11/21
to ScyllaDB users
Wonder if use cql via Scylla Driver would met the same issue or not?

Cong Guo

<cong.guo@smartnews.com>
unread,
Nov 11, 2021, 5:54:47 AM11/11/21
to ScyllaDB users
I just thought if scylla disabled Nagle's algorithm, then we should not meet such issue.

Cong Guo

<cong.guo@smartnews.com>
unread,
Nov 11, 2021, 5:59:44 AM11/11/21
to ScyllaDB users
I am sure the Nagle's algorithem is disabled in AWS ddb sdk, for I printed it to the stdout.


cguo: test3

cguo: test2

cguo: test1

cguo: 111 set TcpNoDelay true

Nadav Har'El

<nyh@scylladb.com>
unread,
Nov 11, 2021, 8:56:50 AM11/11/21
to ScyllaDB users, Piotr Sarna
I think I know where the problem is, and opened https://github.com/scylladb/scylla/issues/9619 and also https://github.com/scylladb/seastar/issues/968 for the underlying Seastar library.

If we fix the problem, would you be able to test the result? How do you consume Scylla - can you run a nightly version if it contains a fix?

--
Nadav Har'El
n...@scylladb.com


--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/061cdf0a-8aa4-4bb3-8079-224d7b89b650n%40googlegroups.com.

Cong Guo

<cong.guo@smartnews.com>
unread,
Nov 11, 2021, 9:10:44 PM11/11/21
to ScyllaDB users
Hi Nadav, 

Sure, I can do that, it's much better if the nightly version support upgrade from 4.5.1. Wonder when will it be able?

Thanks,
Cong

Avi Kivity

<avi@scylladb.com>
unread,
Nov 14, 2021, 6:23:59 AM11/14/21
to scylladb-users@googlegroups.com

You can upgrade from 4.5.1 (or any version) to the nightly.


In general, you can upgrade from the nightly to a later release (e.g. 4.7.0; 4.6 was already branched), but we do not guarantee that it will be possible. The nightly will of course be less stable than release branches.

Cong Guo

<cong.guo@smartnews.com>
unread,
Nov 14, 2021, 10:18:21 PM11/14/21
to ScyllaDB users
Thank you.
Reply all
Reply to author
Forward
0 new messages