I have a relatively slow read-only lua script that performs some expensive computations. The performance is currently not satisfactory, so I need to know should I try migrating the code to a C Redis module or not. I don't think I can be that fast in doing the migration, so here I am asking before actually going for it.
I know the performance boost is highly depending on the kind of processes I need to do. Here is a typical workflow:
1. In a database of 10 million keys, get 100 sets given the keys.
2. For each found set, scan the whole set and does some computation on the values (which all are integer). Sets are of size 1 to 5000. However, the majority of sets are of size between 10 and 100.
3. Get 20 strings each of size about 100KB. Json decode the strings and perform some linear-time computation on each strings.
4. Return the result of computation.
I am doing this calculation on 8 Redis instances each storing a portion of my data. Currently, my code can take as much as 5 seconds with nearly 100% CPU utilization. If you have any estimate or feeling on the performance boost, it will be a great help to share it with me. Do you think I can expect gaining 10x speed boost or even more?
Thanks.