optional STORE argument for read operations?

[Brian P O'Rourke]

I don't want to hijack Peter's thread about his specific problem[1],
so let me re-start my suggestion here... this is a bit of a thought
experiment.

It would be nice if the STORE suffix were generalized to an argument
that works wherever there's a read operation going on. I think this
change would simplify Redis's language significantly, though the
implementation would have to map return values of all commands
returning integer/bulk/multi-bulk to Redis types in a way that seemed
natural.

Examples of operations which would have new uses

Atomic copy on a string:
GET STORE

Grab a snapshot of the keys available in a hash as a new set:
HKEYS STORE

Partial copies of keys:
GETRANGE STORE
HMGET STORE
ZRANGE STORE
LRANGE STORE

And all of these special commands could be dropped in favor of their
composite alternative:

SDIFFSTORE becomes "SDIFF STORE"
SUNIONSTORE
SINTERSTORE
ZUNIONSTORE
ZINTERSTORE

Most of this can be done with Lua now, but it seems like a way to at
once simplify and add functionality - thoughts?

Cheers,
Brian P O'Rourke


[1] https://groups.google.com/forum/#!msg/redis-db/TXhODw23YW0/sruwnzIm5xQJ

[Salvatore Sanfilippo]

Hi, you can use Lua Scripting in Redis 2.6 to solve all the scenarios
you describe.

I think that while many protocol extensions like this seem natural,
the risk is to add STORE, then a generic NX option, and so forth, and
you get with half-a-language that does not benefit in the long run
IMHO. Instead just going for a Lua script is the simplest thing :-)

It is much better to provide fast Lua primitives in the future for
things we'll need. For instance we can have C-coded functions in Lua
that do things like redis.mass_add_to_set(lua_array) or alike, that
can avoid writing a loop inside Lua.

Cheers!
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>



--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Brian P O'Rourke]

On Thu, Nov 1, 2012 at 11:22 AM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
<snip>

> It is much better to provide fast Lua primitives in the future for
> things we'll need. For instance we can have C-coded functions in Lua
> that do things like redis.mass_add_to_set(lua_array) or alike, that
> can avoid writing a loop inside Lua.

This makes a lot of sense. Thanks, Salvatore.

Cheers,
Brian

>
> Cheers!
> Salvatore
>

[Josiah Carlson]

On Thu, Nov 1, 2012 at 11:22 AM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
>

> Hi, you can use Lua Scripting in Redis 2.6 to solve all the scenarios
> you describe.
>
> I think that while many protocol extensions like this seem natural,
> the risk is to add STORE, then a generic NX option, and so forth, and
> you get with half-a-language that does not benefit in the long run
> IMHO. Instead just going for a Lua script is the simplest thing :-)
>
> It is much better to provide fast Lua primitives in the future for
> things we'll need. For instance we can have C-coded functions in Lua
> that do things like redis.mass_add_to_set(lua_array) or alike, that
> can avoid writing a loop inside Lua.

So a special C-implemented API for interacting with the underlying
system, as opposed to using the redis.call() interface?

That seems to me to be rife with "I want internal API X", which is the
same problem, only now there are 2 APIs (the external and internal).

What about just offering the ability to call 3rd party C extensions
stored in .so modules via Lua? Combine that with a relatively fixed
C-API for major components in Redis for major releases, and you
basically have the Postgres 3rd party situation (which has worked
great for them).

- Josiah

[Salvatore Sanfilippo]

On Thu, Nov 1, 2012 at 7:56 PM, Josiah Carlson <josiah....@gmail.com> wrote:

> So a special C-implemented API for interacting with the underlying
> system, as opposed to using the redis.call() interface?

There is no real need to break the interface, but it is possible to
extend it considering we are calling it from Lua in some way.
For example, instead of:

redis.sadd("foo",1,2,3,4,5,...) maybe we could support
redis.sadd("foo",my_array)

Or as proposed before, Lua could have access to a few well-specified
APIs that don't have any sense called as commands, that can operate
keys in very fast ways, as mass-add to list, set, zset, ...

I don't think this will be a big issue, but this is one of those ideas
that in order to enter Redis need 6 months or more and a clear
understanding that we: 1) need more speed compared to Lua loops. 2) We
can't optimize it further. 3) A minimal set of API calls can do a
great difference in some case.

For now I'll simply not add anything :-) We'll need a lot of
experience to understand what's a sensible thing to implement or not
IMHO.

> That seems to me to be rife with "I want internal API X", which is the
> same problem, only now there are 2 APIs (the external and internal).

This is bad only if the second API makes sense as a standard command
itself, this would be odd.

But: mass_add_to_set(key,array) is in the command API simply SADD.

However I think it's much better if mass_add_to_set() can just be an
optimization the user is not aware of at all if possible.

Like if the engine detects "SADD","key", ... many args, it will take a
code path that will add stuff directly to the set instead of really
going along the normal code path to dispatch the command using the
"fake client".

> What about just offering the ability to call 3rd party C extensions
> stored in .so modules via Lua? Combine that with a relatively fixed
> C-API for major components in Redis for major releases, and you
> basically have the Postgres 3rd party situation (which has worked
> great for them).

I'm really against that for a number of reasons already exposed. But
this is a short summary:

1) Every Redis instance should be the same with a set of minimal API.
2) I don't think that external C-coded extensions will allow Redis to
stay as stable as it is today.
3) I think Redis is Redis because of it's API. If everybody can extend
the API Redis will no longer be Redis, in some way.

Cheers,
Salvatore