Redis Indexing Mystery?

[Runming Wu]

Hi:

We are working on the SONiC project: https://github.com/sonic-net.

It uses Redis as the centralize DB. In SONiC, most of the data is stored as Redis hash.

In the software layer, the Redis hash is abstracted as "table". The same keys in the same table will have the same table name as prefix.

For example, the Redis hash key will look like "<table_name>:<key>".

To get all the keys in a table, we can search "<table_name>:*".

We found some weird timing issue in this LUA script: https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua

What the script does is simple. It dumps all the data in a table (dump all Redis hash that has the given prefix).

We found out that the time it took to run this script on different tables differed a lot. Here is our example run:

-------------------------------------------------------------------------------------------------------------------------------

$ redis-cli -n 1 keys "ASIC_STATE*" | wc -l

77665

$ redis-cli -n 1 keys "TEMP_ASIC_STATE*" | wc -l

77641

$ redis-cli -n 1 SCRIPT LOAD "local keys = redis.call(\"KEYS\", KEYS[1] .. \":*\")

> local res = {}

> 

> for i,k in pairs(keys) do

>    local sres={}

> 

>    local flat_map = redis.call('HGETALL', k)

>    for j = 1, #flat_map, 2 do

>        sres[flat_map[j]] = flat_map[j + 1]

>    end

> 

>    res[k] = sres

> end

> 

> return cjson.encode(res)"

"db4ed79b13424810826cd9ad2a5b18367f876409"

$ time redis-cli -n 1 EVALSHA db4ed79b13424810826cd9ad2a5b18367f876409 1 ASIC_STATE > /dev/null

real 0m2.847s

user 0m0.050s

sys 0m0.021s

$ time redis-cli -n 1 EVALSHA db4ed79b13424810826cd9ad2a5b18367f876409 1 TEMP_ASIC_STATE > /dev/null

real 0m31.268s

user 0m0.040s

sys 0m0.028s

-------------------------------------------------------------------------------------------------------------------------------

There are two tables: ASIC_STATE and TEMP_ASIC_STATE.

We can confirmed that the two tables have similar size (in both number of keys and data contents). The main difference is the table name.

But ASIC_STATE took 3 seconds to run the script and TEMP_ASIC_STATE took 30 seconds.

That is a lot of difference. And it is causing performance issue in our software due to this.

We are using Redis version 6.2.7.

Thanks in advance on any hints or clues on how we can improve this.

Runming Wu