What will be the expected performance boost if I migrate code from Lua script to a C Redis Module?

[hamid alaei]

I have a relatively slow read-only lua script that performs some expensive computations. The performance is currently not satisfactory, so I need to know should I try migrating the code to a C Redis module or not. I don't think I can be that fast in doing the migration, so here I am asking before actually going for it.
I know the performance boost is highly depending on the kind of processes I need to do. Here is a typical workflow:

1. In a database of 10 million keys, get 100 sets given the keys.
2. For each found set, scan the whole set and does some computation on the values (which all are integer). Sets are of size 1 to 5000. However, the majority of sets are of size between 10 and 100.
3. Get 20 strings each of size about 100KB. Json decode the strings and perform some linear-time computation on each strings.
4. Return the result of computation.

I am doing this calculation on 8 Redis instances each storing a portion of my data. Currently, my code can take as much as 5 seconds with nearly 100% CPU utilization. If you have any estimate or feeling on the performance boost, it will be a great help to share it with me. Do you think I can expect gaining 10x speed boost or even more?

Thanks.

[Dvir Volk]

Even with all that you can’t really tell. However with modules you are not bound to the redis data structures and can implement your own optimized ones. Once you do that, it opens the door to massive improvements.

I can tell you for example that set intersections in redisearch are about 20 times faster than redis set intersections because we are using highly optimized structures and algorithms that are not generic for any set, but for record sets in inverted indexes.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[hamid alaei]

Thanks a lot dvirsky. Your 20x improvement experience is really amazing and inviting. I am using sets to implement inverted index too. But not for text applications. I will take a look at your code to learn more.

[Dvir Volk]

You can maybe just use my module. The aggregation engine will give you a lot of flexibility as to what you can achieve with numeric operations on sets. 

On Mon, May 7, 2018 at 3:53 PM hamid alaei <hami...@gmail.com> wrote:

Thanks a lot dvirsky. Your 20x improvement experience is really amazing and inviting. I am using sets to implement inverted index too. But not for text applications. I will take a look at your code to learn more.

--