About read performance

32 views
Skip to first unread message

workatf...@gmail.com

unread,
Nov 7, 2021, 7:42:40 AM11/7/21
to rqlite
Hello,

While executing some parallel reading tests , i.e., several simultaneous queries against the same entity with different where clauses, I notice a huge degradation in response performance, by about 10x. (single query takes 0.500ms, parallel queries go up to between 4s and 5s) and I'm a bit puzzled to why.

I'm using latest version, in-memory and a single node. 

I know it is hard to answer this question without additional context, but do you think this can be attributed to rqlite? And can I provide you with aditional info, or test cases?

 

Philip O'Toole

unread,
Nov 8, 2021, 8:08:19 AM11/8/21
to rql...@googlegroups.com
I see very good parallel read performance (10,000+ queries per sec on a single instance, using SSDs), I would need precise details on the queries you are running, including the read-consistency level you are setting, if any. I would need to see the exact requests you make, in terms of the HTTP requests.

https://www.youtube.com/watch?v=JLlIAWjvHxM (from about min 20 on, and I discuss performance)

--
You received this message because you are subscribed to the Google Groups "rqlite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rqlite+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/ab7de866-547c-4857-8d09-28b58f7e3980n%40googlegroups.com.

workatf...@gmail.com

unread,
Nov 8, 2021, 11:23:15 AM11/8/21
to rqlite
I ended up doing the following test, to exclude rqlite from the equation:

1) started rqlite in disk mode and created the data base

2) made a simple c# client to connect to the data base and execute parallel queries

3) the problem is the same: 2 queries in simultaneous (creating a new connection for each read), almost doubles the response time comparing to a single query.

So this as something to do with sqlite, I suppose.

Regarding rqlite options I'm not setting any read-consistency level and i'm using only one node.

The query is a bit complex. But I will try to provide a testing sample.

Philip O'Toole

unread,
Nov 8, 2021, 12:13:25 PM11/8/21
to rql...@googlegroups.com
Yeah, without more details, it's hard for me say. You're queries could be locking tables, something your client code.

Philip O'Toole

unread,
Nov 8, 2021, 12:29:34 PM11/8/21
to rql...@googlegroups.com
Here's a simple example using the load-testing tool that comes with rqbench, showing that rqlite can support 1000s of requests a second on my machine:

workatf...@gmail.com

unread,
Nov 11, 2021, 4:39:57 PM11/11/21
to rqlite
I'm sure that's the case for really fast sqlite queries. The problem seems to be in the complexity of the database, causing queries to process in hundreds of milliseconds (0.200 - 0.900). It seems sqlite can't handle concurrent reads very well and I get almost sequential read times when multithreading several queries.

Philip O'Toole

unread,
Nov 11, 2021, 5:07:47 PM11/11/21
to rql...@googlegroups.com
Sounds like you think might have a SQLite-specific issue, not a rqlite-issue. Have you considered measuring the query performance of a SQLite database directly? With your data loaded?

workatf...@gmail.com

unread,
Nov 11, 2021, 5:15:12 PM11/11/21
to rqlite
Yes, I created a on-disk sqlite db. And sqlite executes very slow concurrent reads.
Reply all
Reply to author
Forward
0 new messages