Performance issue (10x slower) using redis as middleware rediSQL

[Simone Mosciatti]

Happy New Year Everybody!

with my module (rediSQL) I am facing a performance problem.

I cannot understand why it is slow to insert elements.

I documented everything with precision and some graph in this github issues: https://github.com/RedBeardLab/rediSQL/issues/13#issuecomment-270449852

Anyhow, when I insert using rediSQL a get a peek of performance at the beginning of around 24k inserts per second, and then these number decrease down to 4k inserts per second in a (at least) linear way based on the row count.

My first though was that the problem was SQLite, so I asked help on the mail list, but they very gently show me that SQLite is not the bottleneck, on my machine using TCL it reach 240k inserts per second and it is stable at that number if when more than a million of elements are inserted.

Now I am completely lost, SQLite is fast enough, I do not do any weird operation on the Redis side, I simply stream each statement to a DB connection; but I cannot explain why the performance are slow and why they decrease.

Of course I am expecting some performance drop, but 10x is a little too much, there must be something more. Similarly I cannot explain why the performance using rediSQL decrease, while they are stable using vanilla SQLite.

Do you have any idea?

Cheers,

Simone