Pipe the results of a command as arguments to another

[Joran Greef]

It would be great to be able to "pipe" the results of a command as
arguments to another:

ZRANGE index 0 1000 | MGET

(Return the first 1000 objects represented by the keys in "index")

A good use case for this (even if you could just pipe certain commands
like ZRANGE to just MGET) would be to eliminate unnecessary network
overhead, since it removes the need to have to send results out to the
application, only for the application to have to send them back in.

[Jak Sprats]

i like this idea, for the example you are talking about,
you can always put an appserver on the same machine as redis and then
call that appserver which does the 2 requests locally and just sends
the results.
this will cut out the additional network overhead.

but being able to combine redis commands server-side would allow for
some very complex server-side data manipulation, so the idea is worth
exploring, its tricky to come up w/ good use-cases for it though.

[Joran Greef]

Yes, even if only MGET supported this for results returned by ZRANGE
it would solve my problem.

I like your idea of putting an appserver on the same machine as Redis,
but it would still be a waste of CPU to transfer say 500000 keys back
and forth. This is my exact use-case: I need to MGET the objects
represented by keys in a sorted set.

For my API I do authentication in my application code, but then
authorization and filtering within Redis, using set union and
intersection commands on various primary and secondary indexes. I then
MGET the keys returned, and return these objects directly to the end-
user as strings. This keeps authorization centralized, and eliminates
unnecessary JSON deserialization/serialization overhead, which can be
costly for large responses. The end result is that the user's data
goes directly from database to user, with the data opaque to the API
server. Redis seems well-suited for this since it's already keeping
the indexes in memory and provides great set support.

[Michel Martens]

On Thu, Sep 16, 2010 at 7:22 AM, Joran Greef <joran...@gmail.com> wrote:
> Yes, even if only MGET supported this for results returned by ZRANGE
> it would solve my problem.
>
> I like your idea of putting an appserver on the same machine as Redis,
> but it would still be a waste of CPU to transfer say 500000 keys back
> and forth. This is my exact use-case: I need to MGET the objects
> represented by keys in a sorted set.

DEL is another command that could benefit from this.

[Kijin Sung]

On Sep 16, 10:01 am, Michel Martens <sove...@gmail.com> wrote:

> DEL is another command that could benefit from this.

Exactly. Sometimes I wish I could do "KEYS prefix:* | DEL" or
something like that. I don't mind the network round-trip, but I do
care about atomicity. MULTI/EXEC doesn't work well when the the result
of the first command needs to be fed into the second command, because
the first command won't be executed until after the second command is
written.

Over time, piping could replace a whole array of compound commands
such as SINTERSTORE, SDIFFSTORE, ZUNIONSTORE, RPOPLPUSH. It would also
make SORT a lot more flexible than the current SORT...GET solution.
Piping is a consistent and flexible mechanism that can assume the
function of a whole class of existing and potential commands, and it's
also a familiar feature of Unix command-line programming. Go for it!

Of course, we'll need to have some rules concerning how to treat the
output of certain commands (ZRANGE...WITHSCORES, HGETALL) when piped.

[Jak Sprats]

Kijin raises a real interesting point. The thing why pipes dont work
automatically in redis is all commands give one result per line and
many requests take 2 or 3 arguments per line. There has to be a 1to1
mapping or a 2to2 or a NtoN betweeen the left and right hand side of
the pipe, but it has to be definable. (the best example is piping a
ZRANGE WITH SCORES into a HSET which is a very powerful data
structure, but the keys do not appear on the same line as the fields,
so a HSET is meaningless)

The idea is brilliant, because you can combine redis data-structures,
but syntactically its not a good fit, and changing existing commands
is a nogo as there is code already built against them.

[Aníbal Rojas]

Piping + something like SED or AWK in Redis would practically get rid
of the necessity of a higher level scripting support.

This idea is really cool.

--
Aníbal Rojas  -  @anibalrojas
Ruby / Rails focused Devops
More info: http://www.google.com/profiles/anibalrojas
My free/busy schedule: http://tungle.me/anibal

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[Aníbal Rojas]

[Not sure what happened with my previous response]

Piping + something very simple ala AWK or SED would practically
replace any need for a high level scripting support.

--
Aníbal Rojas  -  @anibalrojas
Ruby / Rails focused Devops
More info: http://www.google.com/profiles/anibalrojas
My free/busy schedule: http://tungle.me/anibal

On Fri, Sep 17, 2010 at 10:27 AM, Jak Sprats <jaks...@gmail.com> wrote:
>

[Kijin Sung]

On Sep 17, 10:57 am, Jak Sprats <jakspr...@gmail.com> wrote:

> but syntactically its not a good fit, and changing existing commands
> is a nogo as there is code already built against them.

Agreed. The ZRANGE | HSET case is particularly tricky.

However, with regard to the 1to1/NtoN mapping thing, I think it would
be easy to find workarounds. For example, if COMMAND_1 returns 1
value, and COMMAND_2 needs two args, you could do

COMMAND_1 | COMMAND_2 arg1

where the second argument for COMMAND_2 will be filled with the output
of COMMAND_1. Or maybe even

COMMAND_1 | COMMAND_2 - arg2

where the first argument for COMMAND_2 will be filled with the output
of COMMAND_1.

Heck, you could even do printf, though I suspect it would be overkill.
Anyway, people do command piping and argument juggling all the time in
the Unix shell. Each command does one thing and one thing well, and
you combine commands to do what you want to do. The protocol just
needs to be consistent.

Redis already does a similar thing in one case: SORT...GET. Normally,
the result of SORT is a list, one result per line. This is "piped"
into MGET as arguments, so that the corresponding keys are returned
instead. One could generalize from this example, and implement some
sort of "duck typing", so that piping works if and only if the
commands on either side are compatible. Otherwise you'd get an error.

A similar item on my wishlist would be simple logical operators such
as AND/OR. This would make it super easy to make commands conditional
on the result of a boolean-returning command, such as EXISTS, all the
while maintaining atomicity and reducing round-trips. Some existing
commands such as SETNX is actually equivalent to EXISTS OR SET. One
could make a "reverse SETNX" by writing EXISTS AND SET. Similar things
can be done with other boolean-returning commands, such as SADD,
SISMEMBER, HEXISTS, etc. Again, the Unix shell provides this
capability, and it's amazing what one can do with simple logical
primitives such as AND/OR and piping without using anything as
convoluted as SQL.

Relying on a combination of primitives might also take the pressure
off the Redis developers to keep adding slightly different commands
that do slightly different things. You'd just tell the user to combine
existing commands, which is so much easier to do when you don't have
to rely on a delicate balance of WATCH/UNWATCH, MULTI/EXEC and so on.

Maybe I've been spoiled by the new redis-cli, which is probably why I
keep thinking of importing Unix shell features to Redis... But Redis
does look to me like an exemplary embodiment of Unix principles. Maybe
we could get something along these lines in Redis 3.x?

[Jak Sprats]

I would write everything with bash scripts if I could, so I like this
idea :)

why dont we propose a syntax. I like:
"COMMAND1 | COMMAND2 $1 $2" for "ZRANGE WITHSCORES | HSET HT $1 $2"
or
"COMMAND1 | COMMAND2 $@" for "ZRANGE | MGET $@"

and throw in some logical operators
"COMMAND1 && COMMAND2"

the point being, before any of this brainstorming is taken seriously,
this stuff has to be written down and picked apart, and it has to be
shown how useful it can be. I love bash scripts, but they have a bad
rep from 4th generation language guys (php, ruby, etc...) cause bash
is not as natural as gen4 languages.

But if the point can be made that this syntax can allow for powerful
server side scripting, and provide a lot of functionality, and would
not take forever to code, then it is worth developing.

Additionally, I always try to figure out how this stuff would be
effected by redis-cluster ... piping commands could involve network
hops server side, which is not a good thing.

[Kijin Sung]

Bump... ;)

Maybe the examples we've been throwing around in this thread are way
too complicated. Just looking at some of my code, I see myself doing
the following a lot of the time:

key = LPOP list_of_keys
GET key

Or any other variant where I use a a list or set or zset to store my
keys (or parts of keys, such as the numeric ID). The actual value is
usually stored under another key, as a string or a hash.

This is an extremely common use case, and almost mandatory if your
values are large. So it would be great if I could retrieve the actual
value in one step, atomically. In other words:

LPOP list_of_keys | GET

Of course, the solution doesn't need to take the form of piping. Any
reasonable syntax would be fine. I just wish there were an atomic
shortcut for this very common case, and I'm not even asking for MGET
here!

[Joran Greef]

Yes, that's the same use case as mine. I was doing some benchmarks
last night, and transferring keys out and back in to Redis is
responsible for 25% of a database operation. Using Redis in this way
is actually slower than using Tokyo Tyrant with custom Lua extensions
to do the same thing but avoid the network traffic. Writes with Redis,
however, are enjoyably fast.

But I did some more looking around through the docs, and I where I
used to think that SORT..GET could not use SortedSets, I found that in
fact it can.

So you could try:

SORT somesortedsetorsetorlisthere BY nosort GET *

And that would retrieve all *objects* represented by the keys in
"somesortedsetorsetorlisthere".

This can solve my problem.

[Joran Greef]

On second thoughts, SORT ... BY nosort GET * would not work when you
need to use keys returned by ZRANGE or ZRANGEBYSCORE (since there's no
way to store the results of these operations).

What we really need (and we can skip everything else) is a way to do:
ZRANGE ... 10 +inf | GET.

Perhaps this would be easier to implement if MGET was extended to
support being passed a command and a command's arguments. MGET would
run the command (which would need to return a single key or a list of
keys) and then return the objects represented by those keys. So then
it would look more like this:

MGET ZRANGEBYSCORE sortedset 1285139822 9999999999

Perhaps instead of extending MGET, we could create a new one. Up for
bike-shedding but does "METAGET" have any takers? ;)

It seems that when people use SORT ... BY nosort GET * it's just as a
means to do bulk retrieval of an objects represented by an index, i.e.
MGET sortedset and perhaps this is a symptom of needing a command to
do this properly.

[Jak Sprats]

so using the analogy of SINTERSTORE and ZUNIONSTORE perhaps the best
syntax for stuff like this is:
ZRANGEPIPE command key start end

which in practice would be used like:
ZRANGEBYSCOREPIPE MGET sortedset 1285139822 9999999999

or we could just introduce an Index :)