About read performance

[workatf...@gmail.com]

Hello,

While executing some parallel reading tests , i.e., several simultaneous queries against the same entity with different where clauses, I notice a huge degradation in response performance, by about 10x. (single query takes 0.500ms, parallel queries go up to between 4s and 5s) and I'm a bit puzzled to why.

I'm using latest version, in-memory and a single node. 

I know it is hard to answer this question without additional context, but do you think this can be attributed to rqlite? And can I provide you with aditional info, or test cases?

 

[Philip O'Toole]

I see very good parallel read performance (10,000+ queries per sec on a single instance, using SSDs), I would need precise details on the queries you are running, including the read-consistency level you are setting, if any. I would need to see the exact requests you make, in terms of the HTTP requests.

https://www.youtube.com/watch?v=JLlIAWjvHxM (from about min 20 on, and I discuss performance)

https://www.philipotoole.com/2021-rqlite-cmu-tech-talk

https://github.com/rqlite/rqlite/blob/master/DOC/CONSISTENCY.md

--
You received this message because you are subscribed to the Google Groups "rqlite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rqlite+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/ab7de866-547c-4857-8d09-28b58f7e3980n%40googlegroups.com.

[workatf...@gmail.com]

I ended up doing the following test, to exclude rqlite from the equation:

1) started rqlite in disk mode and created the data base

2) made a simple c# client to connect to the data base and execute parallel queries

3) the problem is the same: 2 queries in simultaneous (creating a new connection for each read), almost doubles the response time comparing to a single query.

So this as something to do with sqlite, I suppose.

Regarding rqlite options I'm not setting any read-consistency level and i'm using only one node.

The query is a bit complex. But I will try to provide a testing sample.

[Philip O'Toole]

Yeah, without more details, it's hard for me say. You're queries could be locking tables, something your client code.

To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/7717290d-3b19-48ed-9902-3532cc794e10n%40googlegroups.com.

[Philip O'Toole]

Here's a simple example using the load-testing tool that comes with rqbench, showing that rqlite can support 1000s of requests a second on my machine:

https://gist.github.com/otoolep/f2f650dc24a9408ddaed2d08bde6ac89

[workatf...@gmail.com]

I'm sure that's the case for really fast sqlite queries. The problem seems to be in the complexity of the database, causing queries to process in hundreds of milliseconds (0.200 - 0.900). It seems sqlite can't handle concurrent reads very well and I get almost sequential read times when multithreading several queries.

[Philip O'Toole]

Sounds like you think might have a SQLite-specific issue, not a rqlite-issue. Have you considered measuring the query performance of a SQLite database directly? With your data loaded?

To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/b8d78b73-ede1-4d00-8f80-f43363d593f0n%40googlegroups.com.

[workatf...@gmail.com]

Yes, I created a on-disk sqlite db. And sqlite executes very slow concurrent reads.