recommendation for storage pattern of A:B association that can be deleted on either element

[jona...@findmeon.com]

We use Redis to handle a 30s cache of "displaysafe" permissions in an app.  The cache isn't consulted for "write" operations, but "read" operations on a given object can be up to 30 seconds out-of date.  For our production needs on this app, this 30s delay on viewing an object is fine.

The caching strategy had been a simple key:

    "displaysafe|a2b|{id_a}:{id_b}" = value

I'm overseeing a ticket fix on our app for an unrelated edge case, and am considering updating our caching component to handle the structure of the new integrated tests associated with the fix. The tests would run best with functionality that will clear out cached values without waiting for the timeout.  The alternatives are to set the timeout to 0 during these tests or only run the delete in development, but I want to keep DEV and PRODUCTION in sync.

I know that I can write an EVALSHA to handle scanning/deleting the keys on the current pattern, but I also know that can create performance problems in large keysets (production).

I looked into using a HASH like: "displaysafe|{id_a}[{id_b}] = value", but am concerned as we'd need to delete based on the id_a AND the id_b, -- which seems like it might have even less performance during scanning.

I'm sure many others have encountered this need and come up with a brilliant solution before.  Can anyone share a recommendation to jumpstart our development?

Thanks in advance, Jonathan

[Pierre Chapuis]

Not sure I understand exactly what you want, but it looks like you are trying to associate a value to two different keys.

What I typically do in this case is use a counter as the primary key for values and indices for the other keys.

Supposing *both* id_a and id_b are unique you can do something like this (pseudocode) to add a value:

n = INCR displaysafe:counter
HSET displaysafe:$n { value = $the_value, id_a = $id_a, id_b = $id_b }
HSET displaysafe:i:id_a $id_a $n
HSET displaysafe:i:id_b $id_b $n

... and to remove by id_a for instance:

n = HGET displaysafe:i:id_a $id_a
id_b = HGET displaysafe:$n id_b
DEL displaysafe:$n
HDEL displaysafe:i:id_a $id_a
HDEL displaysafe:i:id_b $id_b

If you don't have unicity for id_a and id_b, you can use several hashes instead of a hash for indices:

n = INCR displaysafe:counter
HSET displaysafe:$n { value = $the_value, id_a = $id_a, id_b = $id_b }
HSET displaysafe:i:$id_a $id_b $n
HSET displaysafe:i:$id_b $id_a $n

Removal is a bit more complicated and left as an exercise. :)

[jona...@findmeon.com]

Thank you, Pierre.

On Friday, June 15, 2018 at 5:33:26 AM UTC-4, Pierre Chapuis wrote:

Not sure I understand exactly what you want, but it looks like you are trying to associate a value to two different keys.

Yes. Using a counter like that to autogenerate a space for hashes is interesting.  i'll try a test with some sample data that represents our use cases.  That looks more efficient than a similar version i tested, which created a hash for the (value, id_a, id_b), and two regular keys that pointed id_a and id_b to the hash.

A way to describe our use-case can be explained with the Google Groups that we're reading right now.  We currently have about 100k id_a and 10k id_b, and are growing at around a 10:1 ratio.  Each A is a "Person" who is associated to one or more B "Groups".  The association value is if the person is allowed to read the "Group", and extremely readheavy on the "A" index.  When a "Group" has a configuration change, I would like to expire the existing "Person" permissions (which is the only time the "B" index would be consulted).

[jona...@findmeon.com]

If anyone has time for feedback, I could use a second set of eyes on this:

Instead of having the "A2B" and "B2A" lookups point to a single record under another index, I decided it would be lighter to just duplicate the payload in each lookup, as the value is tiny (a comma-separated string of two numbers; the first is at-most 1 character, the second number is at-most 5 characters).

# this deletes them by id_a

eval "local ids_b = redis.call('hkeys', 'a2b|' .. ARGV[1])\nfor i, id_b in ipairs(ids_b) do\n redis.call('hdel', 'b2a|' .. id_b, ARGV[1])\n end\n redis.call('del', 'a2b|' .. ARGV[1]) \nreturn 1" 0 10

# this deletes them by id_b

eval "local ids_a = redis.call('hkeys', 'b2a|' .. ARGV[1])\nfor i, id_a in ipairs(ids_a) do\n redis.call('hdel', 'a2b|' .. ids_a, ARGV[1])\n end\n redis.call('del', 'b2a|' .. ARGV[1]) \nreturn 1" 0 10

does this look decent/performant?  does anyone have suggestions to maximize making things faster?

my primary concern is "what happens if the hkeys are too long?".