Is lua scripting atomic?

[Xiangrong Fang]

While doing redis.call() in a lua script, can I assume that these calls are atomic (all or none)?

Thanks,
Shannon

--

[David Yu]

On Wed, Aug 1, 2012 at 11:11 AM, Xiangrong Fang <xrf...@gmail.com> wrote:

While doing redis.call() in a lua script, can I assume that these calls are atomic (all or none)?

Yes, as long as you make sure all pre-conditions are met before doing *any write* (or you could simulate a rollback in your script).

Thanks,
Shannon

--

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--
When the cat is away, the mouse is alone.
- David Yu

[Josiah Carlson]

And as long as your script doesn't run too long and get killed via "SCRIPT KILL"

 - Josiah

[Xiangrong Fang]

This means, SCRIPT KILL may be called automatically by redis itself?

Shannon

2012/8/1 Josiah Carlson <josiah....@gmail.com>

--

[Josiah Carlson]

No, a client may call SCRIPT KILL if the script is running for longer than it is configured to do so. You can change the running limit, which defaults to 5 seconds.

Regards,

 - Josiah

[Xiangrong Fang]

Running limit is a redis setting, not client's. Even though redis won't explicitly call SCRIPT KILL, it will still kill long running scripts according to running limit setting, right ?

Thanks!

[Josiah Carlson]

As per the EVAL documentation:

Scripts should never try to access the external system, like the file system or any other system call. A script should only operate on Redis data and passed arguments.

Scripts are also subject to a maximum execution time (five seconds by default). This default timeout is huge since a script should usually run in under a millisecond. The limit is mostly to handle accidental infinite loops created during development.

It is possible to modify the maximum time a script can be executed with millisecond precision, either via redis.conf or using the CONFIG GET / CONFIG SET command. The configuration parameter affecting max execution time is called lua-time-limit.

When a script reaches the timeout it is not automatically terminated by Redis since this violates the contract Redis has with the scripting engine to ensure that scripts are atomic. Interrupting a script means potentially leaving the dataset with half-written data. For this reasons when a script executes for more than the specified time the following happens:

• Redis logs that a script is running too long.
• It starts accepting commands again from other clients, but will reply with a BUSY error to all the clients sending normal commands. The only allowed commands in this status are SCRIPT KILL and SHUTDOWN NOSAVE.
• It is possible to terminate a script that executes only read-only commands using the SCRIPT KILL command. This does not violate the scripting semantic as no data was yet written to the dataset by the script.
• If the script already called write commands the only allowed command becomes SHUTDOWN NOSAVE that stops the server without saving the current data set on disk (basically the server is aborted).

Regards,

 - Josiah

[Readis]

Looks like that if a multi-step script fails in the middle, the steps that have already been done will not be rolled back, as shown in this example:

redis 127.0.0.1:6379> set foo 100
OK
redis 127.0.0.1:6379> eval "redis.call('set', 'foo', 200); redis.call('lpush', 'foo', 300)" 0
(error) ERR Error running script (call to f_18507169045dda984b7ec9ae42a857413800dbd5): ERR Operation against a key holding the wrong kind of value
redis 127.0.0.1:6379> get foo
"200"
redis 127.0.0.1:6379> eval "redis.call('set', 'foo', 300); redis.call('lpush', 'foo', 300); redis.call('set', 'foo2', 5678)" 0
(error) ERR Error running script (call to f_ba41622270712b5cdf299f8c90be4f9f5afd7c9d): ERR Operation against a key holding the wrong kind of value
redis 127.0.0.1:6379> get foo
"300"
redis 127.0.0.1:6379> get foo2
(nil)

So it would mean one has to be ultra careful in writing a Lua redis script. Can there be an option to test run the script first and then execute it for real, in the same atomic EVAL command? like

    test_and_eval "redis.call('set', 'foo', 300); redis.call('lpush', 'foo', 300); redis.call('set', 'foo2', 5678)" 0

In typical SQL programming, one would rollback on coding error -- not nice but safer.

[Salvatore Sanfilippo]

On Wed, Aug 1, 2012 at 6:50 AM, Josiah Carlson <josiah....@gmail.com> wrote:

And as long as your script doesn't run too long and get killed via "SCRIPT KILL"

Actually Lua scripts are *always* atomic as SCRIPT KILL is only able to kill scripts without side effects (no writes performed so far).

So a script that modifies the database will either:

1) Run till its natural end.

2) Be stopped before any write, by SCRIPT KILL, so no effect at all.
3) Be stopped by SHUTDOWN NOSAVE, so no effect at all.

There is no way you end with half-writes after calling a Lua script, this is a fundamental contract between Redis and the user.

Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology because software is so complicated. Beauty is the ultimate defence against complexity.
       — David Gelernter

[Josiah Carlson]

On Mon, Feb 4, 2013 at 12:20 PM, Readis <winson...@gmail.com> wrote:

Looks like that if a multi-step script fails in the middle, the steps that have already been done will not be rolled back, as shown in this example:

Correct.

redis 127.0.0.1:6379> set foo 100

OK
redis 127.0.0.1:6379> eval "redis.call('set', 'foo', 200); redis.call('lpush', 'foo', 300)" 0
(error) ERR Error running script (call to f_18507169045dda984b7ec9ae42a857413800dbd5): ERR Operation against a key holding the wrong kind of value
redis 127.0.0.1:6379> get foo
"200"
redis 127.0.0.1:6379> eval "redis.call('set', 'foo', 300); redis.call('lpush', 'foo', 300); redis.call('set', 'foo2', 5678)" 0
(error) ERR Error running script (call to f_ba41622270712b5cdf299f8c90be4f9f5afd7c9d): ERR Operation against a key holding the wrong kind of value
redis 127.0.0.1:6379> get foo
"300"
redis 127.0.0.1:6379> get foo2
(nil)

So it would mean one has to be ultra careful in writing a Lua redis script. Can there be an option to test run the script first and then execute it for real, in the same atomic EVAL command? like

    test_and_eval "redis.call('set', 'foo', 300); redis.call('lpush', 'foo', 300); redis.call('set', 'foo2', 5678)" 0

In typical SQL programming, one would rollback on coding error -- not nice but safer.

The problem with offering that kind of option is that in order to test, you have to duplicate every key that the script touches if it writes to it, so that if it doesn't succeed, you can "roll back" the changes. In the worst-case, a script could modify every key in Redis, and could do so substantially, which could more than double Redis' memory use. With Redis' data model as it exists currently, the easiest way to do this is to fork the Redis process, block other commands, execute the script on the child process, and if it succeeds, re-execute it on the master process (doubling total execution time).

On some virtual hosting platforms, forking can be a fairly expensive operation, and even in the case where forking is fast, the potential downsides on the memory side of things might be extraordinary. And in the forking + try on the child side of things, the child could be killed for a variety of reasons (the most common being the Linux OOM killer), which would tell the client "this failed" where if it were to run on the non-test case, it would succeed.

This feature might be more trouble than it is worth right now? I don't know. But I do think it would be a neat feature.

 - Josiah

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.

To post to this group, send email to redi...@googlegroups.com.

Visit this group at http://groups.google.com/group/redis-db?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Josiah Carlson]

Sure. But if the Lua script has an error half-way through (bad Redis
call, operation on a nil, etc.), then it can quit partway through and
leave data partially updated.

- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.

> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.

> To post to this group, send email to redi...@googlegroups.com.

> Visit this group at http://groups.google.com/group/redis-db?hl=en.

[Salvatore Sanfilippo]

That's technically true, I can not deny it, but I never buy this
argument (the same argument often used for MULTI/EXEC rollback)
because there are an infinite number of errors that will corrupt data
in case of a programming error without trowing an exception. So the
point in rolling back in case of programming errors, but only for the
subset that can be trivially trapped by the system, is a bad waste of
complexity IMHO.

Cheers,
Salvatore

[Josiah Carlson]

As a point of support for the "test and run if successful" option,
with sufficient warnings about: memory use, twice the execution time,
forking potentially being expensive, ... and proper configuration on
the persistence side of things, Redis could be an ACID compliant
database.

- Josiah

[Readis]

Of course, that's true. Though in practical SQL programming, we see more syntactic and semantic errors than silent rampant/nonrecoverable data corruption. It is just like strong/static typing vs dynamic typing in programming languages.

I can think of two better approaches

1. when executing MULTI/EVAL, accumulate old values in a rollback buffer (of configured size e.g. 64MB); on error or buffer full, rollback values from buffer and raise error to client

2. when executing MULTI/EVAL, accumulate old values in a rollback buffer (of configured size e.g. 64MB), flush the buffer to rollback log if the buffers get full; on error, read back and apply values from the rollback log if created and then the buffer. this makes redis like a real ACID DB.

#1 is probably good enough for majority of these scripts.

[Salvatore Sanfilippo]

IMHO complexity in this case does not pay back, btw what you say is
not completely related to MULTI, as the new behavior of MULTI in
*very* recent versions of Redis is to abort the transaction or syntax
errors (the whole transaction).

In all the other cases I expect that errors that will raise an
exception are trapped in the testing environment, we are talking about
a class of errors that will scream aloud with an error :-) Something I
don't expect to touch the production environment.
In the production environment instead, for a DB like Redis, I expect
most users not wanting to use more CPU to accumulate changes to
rollback.

Disclaimer: this is a matter of taste / tradeoffs / personal opinion,
I don't mean you are wrong.

Cheers,
Salvatore