Dynamic values passed as lua arguments

[mailb...@gmail.com]

var db = redis.createClient();

var s1 = "some string calculated in Node.js elsewhere";

var s2 = "some string calculated in Node.js elsewhere";

var s3 = "some string calculated in Node.js elsewhere";

var rs = "myLua script"

 

db.EVAL(rs, 0, ****3 SET names here***, function(err, replies) { 

        if (err) { console.log (err);

   return console.error("error response - " + err);

}

        res.json (JSON.parse(replies));

  });

I want to set DYNAMICALLY values of s1, s2, s3 where I have ****3 SET names here*** in the EVAL call above.

In the rs lua script, I do:

local c = redis.call('SINTER', unpack(ARGV))

When passing those 3 set names as arguments in the EVAL call:

db.EVAL(rs, 0 , 's1', 's2', 's3', function(err, replies) { … => works

db.EVAL(rs, 0 , "s1", "s2", "s3", function(err, replies) { … => works

db.EVAL(rs, 0 , s1, s2, s3, function(err, replies) { … => does NOT work, c=[ ]

How do I pass dynamically generated values as 3 arguments, NOT string literals (Also, no KEY here, since it's a SINTER operation)?

Am I missing something obvious here?

[Josiah Carlson]

The reason why you get no results is simply because the strings that you are calculating in other parts of your Node.js application either don't have relevant keys in Redis, or their contents don't have anything that fits the intersection criteria. Have you output the *values* of your calculated strings s1, s2, and s3, then verified that Redis has those keys (maybe with redis-cli), and verified that there should be a result to the intersection?

As for your "(Also, no KEY here, since it's a SINTER operation)", you are wrong. SINTER accepts *keys* to look up SETs in order to intersect. While what you are doing will technically work in vanilla Redis, it will not work when confronted with key-sharded Redis cluster, or any other key-sharded clients.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Duraz]

Sorry, thanks for the pointers, perhaps I wasn't clear. 

's1', 's2' and 's3' are in fact the names of 3 sets is Redis. What are 'passed as variables' above within the EVAL statement in fact resolve to string variables 's1', 's2' and 's3'. Easily verified with SMEMBERS to see they are populated and also SINTER s1 s2 s3 does in fact give the right intersection, as I mentioned above. I've checked this with lua script loaded and ran via redis-cli, also sent via a Node.js script. That's not the problem I'm having. 

To state slightly more narrowly: 

db.EVAL(rs, 0 , 's1', 's2', 's3', function(err, replies) { … => works in every way.

But

myVar1 = 's1';

myVar2 = 's2';

myVar3 = 's3';

db.EVAL(rs, 0 , myVar1, myVar2, myVar3, function(err, replies) { … => does NOT work

db.EVAL(rs, 3 , myVar1, myVar2, myVar3, function(err, replies) { … => does NOT work either

I just want to call in Node.js: 

db.EVAL(rs, 0 , myVar1, myVar2, myVar3, function(err, replies) { …

and use in lua script in Node.js:

local c = redis.call('SINTER', unpack(ARGV)) which should unpack to 

local c = redis.call('SINTER', s1, s2, s3) but doesn't and is my problem.

 

(This is a controlled app, so it'd be impossible to pick a set variable that doesn't exist or is empty. It's possible to get an empty intersection, but that's not the problem here. I just can't seem to pass dynamically selected myVar1, myVar2, myVar3 to EVAL unless they are string literals, which I don't know ahead of time.)

As to the SINTER KEYS issue, is the name of the SET considered the KEY? If so, the ARGV is the actual contents? 

In that case, in db.EVAL(rs, 0 , myVar1, myVar2, myVar3, function(err, replies) { …

myVar1, myVar2, myVar3 would/should be considered KEYS, no?

local c = redis.call('SINTER', unpack(ARGV)) does work to unpack(KEYS) instead of ARGVs (when I use string literals 's1', 's2' and 's3'). I don't know why. Is there an unpack(KEYS) command or are KEYS passed considered ARGVs in this case? 

Thanks.

[Josiah Carlson]

Replies inline.

On Sun, Mar 2, 2014 at 12:28 AM, Duraz <mailb...@gmail.com> wrote:

Sorry, thanks for the pointers, perhaps I wasn't clear. 

's1', 's2' and 's3' are in fact the names of 3 sets is Redis. What are 'passed as variables' above within the EVAL statement in fact resolve to string variables 's1', 's2' and 's3'. Easily verified with SMEMBERS to see they are populated and also SINTER s1 s2 s3 does in fact give the right intersection, as I mentioned above. I've checked this with lua script loaded and ran via redis-cli, also sent via a Node.js script. That's not the problem I'm having. 

To state slightly more narrowly: 

db.EVAL(rs, 0 , 's1', 's2', 's3', function(err, replies) { … => works in every way.

But

myVar1 = 's1';

myVar2 = 's2';

myVar3 = 's3';

db.EVAL(rs, 0 , myVar1, myVar2, myVar3, function(err, replies) { … => does NOT work

db.EVAL(rs, 3 , myVar1, myVar2, myVar3, function(err, replies) { … => does NOT work either

That is very strange and seems to violate my expectations of what the meaning of variables are in Javascript. Are you *really* sure that your variables are exactly those strings? Can you try to cut your script down to one of the following:

return redis.call('SINTER', unpack(ARGV))

return redis.call('SINTER', unpack(KEYS))

Or heck, have you tested:

return ARGV

return KEYS

Or even:

redis.call('SADD', 'TEMP', unpack(KEYS))

return redis.call('SMEMBERS', 'TEMP')

... just to make sure that you are getting what you expect

 

I just want to call in Node.js: 

db.EVAL(rs, 0 , myVar1, myVar2, myVar3, function(err, replies) { …

and use in lua script in Node.js:

local c = redis.call('SINTER', unpack(ARGV)) which should unpack to 

local c = redis.call('SINTER', s1, s2, s3) but doesn't and is my problem.

No, it's going to unpack to:

local c = redis.call('SINTER', 's1', 's2', 's3')

You were missing some quotes.

 

 (This is a controlled app, so it'd be impossible to pick a set variable that doesn't exist or is empty. It's possible to get an empty intersection, but that's not the problem here. I just can't seem to pass dynamically selected myVar1, myVar2, myVar3 to EVAL unless they are string literals, which I don't know ahead of time.)

As to the SINTER KEYS issue, is the name of the SET considered the KEY? If so, the ARGV is the actual contents? 

I'm not sure you understand what KEYS and ARGV are supposed to represent. Your later statement says that you should know, but this question suggests that you don't. KEYS are supposed to be keys that you will be reading from or writing to in Redis. They are separated from ARGV so that if you are selecting your Redis server based on where certain keys are stored, you can determine *before the script is running* what server it should run on, and if all keys should be on that server so it can run there.

ARGV is supposed to contain arguments that you use to manipulate your script behavior. Little things like timestamps to scan over a ZSET for, entries to add to a SET, keys to pull from a HASH named in KEYS, limits for how much data to return from a sub ZRANGE call, etc.

You can also freely use any value in KEYS as an argument (like ARGV), though you aren't supposed to be dynamically generating keys inside of Lua scripts (though this is not enforced except at the Redis cluster level at present), or using ARGV values as KEYS (also not enforced except by Redis cluster and some clients).

In that case, in db.EVAL(rs, 0 , myVar1, myVar2, myVar3, function(err, replies) { …

myVar1, myVar2, myVar3 would/should be considered KEYS, no?

I would consider them to be KEYS, but then you'd have to pass a 3 instead of a 0.

local c = redis.call('SINTER', unpack(ARGV)) does work to unpack(KEYS) instead of ARGVs (when I use string literals 's1', 's2' and 's3'). I don't know why. Is there an unpack(KEYS) command or are KEYS passed considered ARGVs in this case? 

Both KEYS and ARGV are just tables in Lua. The unpack() function performs some magic to extract the contents of the table to pass as arguments to the function. It's sort-of like a flipped around fcn.apply from Javascript.

More than anything, I suspect that either your values aren't really what you expect them to be, you aren't using your client properly, or your client is broken in some *very* strange way. What client library are you using, and can you show us a 10-line example that doesn't do what you expect?

 - Josiah 

Thanks.

[Duraz]

Here's where I originally got stuck, while testing:

s1 = 's1';

s2 = 's2';

s3 = 's3';

('s1', 's2' and 's3' are in fact the names of 3 actual SETS in Redis.)

If I pass string literals to get at those sets like so:

db.eval(rs, 0 , 's1', 's2', 's3' , function(err, replies) {…  => everything is passed to lua, and in lua this:

 

c = redis.call('SINTER', unpack(ARGV)) also works and generates the correct intersection of the 3 SETs

if however I pass the variables created in JS above: 

db.eval(rs, 0 , s1, s2, s3 , function(err, replies) {…  => fails  

c = redis.call('SINTER', unpack(ARGV))  => fails

 

db.eval(rs, 3 , s1, s2, s3 , function(err, replies) {…  => fails

c = redis.call('SINTER', unpack(ARGV))  => fails

Even though I've been on pain killers in the last 36 hours, I thought this was nuts.

So I thought perhaps naming the variables the same as quoted string literals may have been the oversight. Renamed them:

v1 = 's1';

v2 = 's2';

v3 = 's3';

Lo and behold, this works:

db.eval(rs, 0 , v1, v2, v3 , function(err, replies) {…  => works in combo with:

c = redis.call('SINTER', unpack(ARGV)) => generates exactly what's expected.

As you point out, I was not clear on KEYS/ARGV business, mostly because I didn't know unpack(KEYS) existed and was forcing the notion of ARGV into my model. So

db.eval(rs, 2 , v1, v2, function(err, replies) {…  combo with:

c = redis.call('SINTER', unpack(KEYS)) => works.

And so does: 

db.eval(rs, 3 , v1, v2, v3 , function(err, replies) {…  combo with:

c = redis.call('SINTER', unpack(KEYS)) => works.

Finally, I'm still groggy but happy it got sorted, thanks for your guidance.

One final question on this:

var myArgsTable = ['v1', 'v2', 'v3'];

var length = myArgsTable.length;

db.eval(rs, length, myArgsTable, function(err, replies) {… fails with "[Error: ERR Number of keys can't be greater than number of args]" showing  myArgsTable doesn't translate into 'v1', 'v2', 'v3' unsurprisingly.

Can you think of a way to insert a variable number of KEYS into the EVAL call?

Thanks.

[Josiah Carlson]

Replies inline...

On Sun, Mar 2, 2014 at 10:23 PM, Duraz <mailb...@gmail.com> wrote:

Here's where I originally got stuck, while testing:

s1 = 's1';

s2 = 's2';

s3 = 's3';

('s1', 's2' and 's3' are in fact the names of 3 actual SETS in Redis.)

If I pass string literals to get at those sets like so:

db.eval(rs, 0 , 's1', 's2', 's3' , function(err, replies) {…  => everything is passed to lua, and in lua this:

 

c = redis.call('SINTER', unpack(ARGV)) also works and generates the correct intersection of the 3 SETs

if however I pass the variables created in JS above: 

db.eval(rs, 0 , s1, s2, s3 , function(err, replies) {…  => fails  

c = redis.call('SINTER', unpack(ARGV))  => fails

 

db.eval(rs, 3 , s1, s2, s3 , function(err, replies) {…  => fails

c = redis.call('SINTER', unpack(ARGV))  => fails

Even though I've been on pain killers in the last 36 hours, I thought this was nuts.

So I thought perhaps naming the variables the same as quoted string literals may have been the oversight.

Are you sure that you aren't missing the use of a 'var' somewhere? I've seen and read about more "oops, I forgot 'var', so a bunch of different unrelated calls are sharing variables in ways I didn't expect" than I'd care to remember.

 

Renamed them:

v1 = 's1';

v2 = 's2';

v3 = 's3';

Lo and behold, this works:

db.eval(rs, 0 , v1, v2, v3 , function(err, replies) {…  => works in combo with:

c = redis.call('SINTER', unpack(ARGV)) => generates exactly what's expected.

As you point out, I was not clear on KEYS/ARGV business, mostly because I didn't know unpack(KEYS) existed and was forcing the notion of ARGV into my model. So

db.eval(rs, 2 , v1, v2, function(err, replies) {…  combo with:

c = redis.call('SINTER', unpack(KEYS)) => works.

And so does: 

db.eval(rs, 3 , v1, v2, v3 , function(err, replies) {…  combo with:

c = redis.call('SINTER', unpack(KEYS)) => works.

Finally, I'm still groggy but happy it got sorted, thanks for your guidance.

You're quite welcome, though I'm pretty sure that your s1, s2, and s3 variables are still not the values 's1', 's2', and 's3', because anything else would imply bad things about Javascript.

One final question on this:

var myArgsTable = ['v1', 'v2', 'v3'];

var length = myArgsTable.length;

db.eval(rs, length, myArgsTable, function(err, replies) {… fails with "[Error: ERR Number of keys can't be greater than number of args]" showing  myArgsTable doesn't translate into 'v1', 'v2', 'v3' unsurprisingly.

Can you think of a way to insert a variable number of KEYS into the EVAL call?

Yes, one of these might work, or you may need to pass something else as the first argument to apply:

db.eval.apply(undefined, [rs, length] + myArgsTable + [function(err, replies) {...}]);

db.eval.apply(db, [rs, length] + myArgsTable + [function(err, replies) {...}]);

[Duraz]

var myArgsTable = ['v1', 'v2', 'v3'];

var length = myArgsTable.length;

db.eval(rs, length, myArgsTable, function(err, replies) {… fails with "[Error: ERR Number of keys can't be greater than number of args]" showing  myArgsTable doesn't translate into 'v1', 'v2', 'v3' unsurprisingly.

Can you think of a way to insert a variable number of KEYS into the EVAL call?

Yes, one of these might work, or you may need to pass something else as the first argument to apply:

db.eval.apply(undefined, [rs, length] + myArgsTable + [function(err, replies) {...}]);

db.eval.apply(db, [rs, length] + myArgsTable + [function(err, replies) {...}]);

I tried numerous things for the first item (db, undefined, eval, obj, this, etc) to no avail; even left it blank, hoping it would somehow assume thisObj and end up making a proper reference. No luck. The EVAL call and the arguments passed therein are still opaque to me. Very frustrating that there doesn't seem to be an obvious way to pass dynamically generated/variable arrays from JS > Lua > Redis and back.  

[Josiah Carlson]

... Well, it's not that the eval() function is strange, it's that Javascript's apply method on functions is strange, and it's not always obvious what the 'this' object is supposed to be. And just because you haven't discovered a way, doesn't mean there isn't a way. Which client are you using? There may be an undocumented interface hidden in there somewhere, or you might be able to convince the author to provide an API that supports passing arrays instead of a fixed number of arguments, or you might be able to extend it yourself.

Tell me what client you are using, and I'll help look for a solution, because I'm sure this won't be the last time someone asks about this.

 - Josiah

[Duraz]

mranney/node_redis client.

It's a node.js app with Express. 

Routine app.get() acquires (variable number of) Redis set KEYS from req.params.id passed in URL.

I then EVAL those, into the lua script, to get JSON from Redis, which is then templated client-side directly into DOM in one go.

[Josiah Carlson]

Looking at:

https://github.com/mranney/node_redis/blob/master/examples/eval.js

It would seem that you can use:
db.eval([rs, length] + myArgsTable. function(err, replies) {...});

 - Josiah

[Duraz]

Looking at:

https://github.com/mranney/node_redis/blob/master/examples/eval.js

It would seem that you can use:
db.eval([rs, length] + myArgsTable. function(err, replies) {...});

After several trials, this is the only format that worked:

 

var myArgsTable = ['v1', 'v2', 'v3'…];

var length = myArgsTable.length;

 

myArgsTable = [rs, length].concat(myArgsTable); 

db.eval(myArgsTable, function(err, replies) {...});  //single array variable

Your help much appreciated.

[Josiah Carlson]

Ahh yes, you are right. I could have swore that array concatenation worked in Javascript.

I'm glad that we got you up and running :)

 - Josiah

--