Re: Non-atomic execution of LUA scripts

[Alexander Gladysh]

Hi,

You probably can try some coroutine-based solution.

I.e. do some work, yield and continue after poke from a client. (Tell
me if you'd like a more complete explanation.)

Redis does not provide a framework for that, but... this would be,
IMHO, much easier to push through than full async stuff.

My 2c,
Alexander.

On Mon, Jun 3, 2013 at 8:00 AM, Lawrence <lawren...@gmail.com> wrote:
> Hi,
>
> Currently lua scripts are always executed atomically.
>
> I'm using lua scripts for fairly complex analytical (time-series) functions.
> Some functions take a "long" time. All these scripts only make read-only
> redis calls though, they never write anything to redis.
>
> Given a script that only makes read-only redis calls would it be possible to
> execute those kind of scripts in a non-atomic way so that it doesn't block
> other clients for the total duration of the script? Or would adding such an
> option to redis be out of the question due to some architectural
> constraints? (for example a lua script might not be able to work with the
> redis event loop??)
>
>
> PS. Also a script that only makes read-only redis calls wouldn't need to be
> executed on replicated slaves (as it does today) I would think.
>
>
> Cheers,
> Lawrence
>
> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Josiah Carlson]

If your data is in Redis, I would instead write functions that keep their state stored in Redis, but are aware of the fact that they can take a while. There's no need to go full continuation style, this is all doable in standard Redis with Lua right now.

How much data are you processing? Can you store your progress as you go along without taking as much time as it would take to re-process it?

 - Josiah

[Lawrence]

Hi Alexander,

I'm assuming you mean a coroutine-based solution that is written outside of redis? 

I see that redis allows lua coroutines to be used in scripts, but I don't think that yielding there would allow other redis clients to do their work (due to the atomicity constraint)? That is however what I was kind of hoping for. 

Just like how lua is integrated into nginx (http://wiki.nginx.org/HttpLuaModule#Nginx_API_for_Lua). The beauty there is that lua coroutines integrate with the nginx eventloop. All I/O is non-blocking. 

So in the redis-lua case what I was thinking was for example that when redis.call is called from a lua script it would yield to the redis eventloop allowing other redis clients in the queue to be handled and that it would resume the lua script after that. I.e. all I/O performed within a lua script would be non-blocking. 

If you'd want an atomic lua script then start the script with redis.multi.

The more I think about it, the more I wonder why the atomic constraint exists for lua scripts? Why not treat a lua script like a regular redis client?

Cheers,

Lawrence

On Monday, June 3, 2013 6:27:48 PM UTC+10, Alexander Gladysh wrote:

Hi,

You probably can try some coroutine-based solution.

I.e. do some work, yield and continue after poke from a client. (Tell
me if you'd like a more complete explanation.)

Redis does not provide a framework for that, but... this would be,
IMHO, much easier to push through than full async stuff.

My 2c,
Alexander.

[Lawrence]

Hi Josiah,

Storing progress is unfortunately not an option for us, or at least not very useful as the results of calculations often won't be used again in the near term. So all we'd achieve then is requiring a ton more memory. It's also not a scalable solution as the more analytical functions we add the more of these pre-processing jobs we'd need to do, most of which results will never be looked at.

In some scenarios we process 10,000 keys of sorted sets together holding around 5 million scores.

Cheers,
Lawrence

[Damian Janowski]

On Mon, Jun 3, 2013 at 9:18 PM, Lawrence <lawren...@gmail.com> wrote:
> Storing progress is unfortunately not an option for us, or at least not very
> useful as the results of calculations often won't be used again in the near
> term. So all we'd achieve then is requiring a ton more memory. It's also not
> a scalable solution as the more analytical functions we add the more of
> these pre-processing jobs we'd need to do, most of which results will never
> be looked at.
>
> In some scenarios we process 10,000 keys of sorted sets together holding
> around 5 million scores.

Then maybe you can set up some slaves just to run those jobs?

[Josiah Carlson]

I was going to mention this myself. This is exactly the case where a read-only replica is the right answer, especially if atomicity of processing is not a requirement.

 - Josiah

[Alexander Gladysh]

On Tue, Jun 4, 2013 at 4:04 AM, Lawrence <lawren...@gmail.com> wrote:
> Hi Alexander,
>
> I'm assuming you mean a coroutine-based solution that is written outside of
> redis?

No, inside.

> I see that redis allows lua coroutines to be used in scripts, but I don't
> think that yielding there would allow other redis clients to do their work
> (due to the atomicity constraint)? That is however what I was kind of hoping
> for.

Yielding inside Redis would — with properly written wrapper.

Problems:

1. You'd have to hackishly store coroutines somewhere, e.g. in `redis` table.

2. You'd have to manually resume running coroutines from some client —
i.e. send heartbeats from somewhere. (Probably from the same client
that initiated the calculation.)

3. Clusterisation will be somewhat more complicated. You'll have to be
able to reach the correct redis instance no matter what.

> Just like how lua is integrated into nginx
> (http://wiki.nginx.org/HttpLuaModule#Nginx_API_for_Lua). The beauty there is
> that lua coroutines integrate with the nginx eventloop. All I/O is
> non-blocking.
>
> So in the redis-lua case what I was thinking was for example that when
> redis.call is called from a lua script it would yield to the redis eventloop
> allowing other redis clients in the queue to be handled and that it would
> resume the lua script after that. I.e. all I/O performed within a lua script
> would be non-blocking.
>
> If you'd want an atomic lua script then start the script with redis.multi.

This will be a nice feature. Salvatore, I think that I can code this
if you're interested (and nobody else will step up). I've done
coroutine-based async stuff like that before.

> The more I think about it, the more I wonder why the atomic constraint
> exists for lua scripts? Why not treat a lua script like a regular redis
> client?

Speed? (Maybe, didn't give much thought to that.)

[Josiah Carlson]

On Mon, Jun 3, 2013 at 8:38 PM, Alexander Gladysh <agla...@gmail.com> wrote:

On Tue, Jun 4, 2013 at 4:04 AM, Lawrence <lawren...@gmail.com> wrote:

> The more I think about it, the more I wonder why the atomic constraint
> exists for lua scripts? Why not treat a lua script like a regular redis
> client?

Speed? (Maybe, didn't give much thought to that.)

If Lua scripts are always atomic, then you can never run into a data race condition between concurrent access. If you can never run into a data race condition, then a lot of concurrent problems become 10-1000x easier. Before the setnxex equivalent command was available, building a proper lock without Lua was a huge PITA. How do I know? There's a 11 page section in my book talking about standard acquire/release locks, which I revisit with Lua for another 3 1/2 pages. But a lot of the pieces that I revisit in Lua are so much easier to think about, because the one thing you *don't* need to think about is atomicity.

Offering new commands EVALASYNC and EVALSHAASYNC, or even an optional argument for EVAL and EVALSHA that occurs after the KEYS and ARGV that tells Redis whether it can be executed asynchronously, that seems like a better answer than changing semantics if async execution is desired.

That said, what's going to happen after async execution is allowed is that someone is going to want to have critical sections inside their Lua scripts, or is going to want to be able to try to acquire a lock via setnxex, be able to yield control to other scripts to finish executing, etc. All of those pieces adds not just one can of worms, but adds a case of different flavored worms.

 - Josiah

[Lawrence]

Yielding inside Redis would — with properly written wrapper.

Problems:

1. You'd have to hackishly store coroutines somewhere, e.g. in `redis` table.

See what you mean, that is quite hackish yes :) 

 

> I.e. all I/O performed within a lua script would be non-blocking.
>
> If you'd want an atomic lua script then start the script with redis.multi.

This will be a nice feature. Salvatore, I think that I can code this
if you're interested (and nobody else will step up). I've done
coroutine-based async stuff like that before.

You've got my vote :)

Cheers,

Lawrence 

[Lawrence]

Hi Josiah,

Offering new commands EVALASYNC and EVALSHAASYNC, or even an optional argument for EVAL and EVALSHA that occurs after the KEYS and ARGV that tells Redis whether it can be executed asynchronously, that seems like a better answer than changing semantics if async execution is desired.

Yes. The current semantics where EVAL is atomic should not change.

 

That said, what's going to happen after async execution is allowed is that someone is going to want to have critical sections inside their Lua scripts, or is going to want to be able to try to acquire a lock via setnxex, be able to yield control to other scripts to finish executing, etc. All of those pieces adds not just one can of worms, but adds a case of different flavored worms.

It's all still synchronous. Lua would be using the evented architecture in redis. With a non-blocking API to redis. So yes, it can have async behaviour (i.e. the script as a whole is non-atomic), but it's not executing asynchronously to anything else.

Given redis is single-threaded I am assuming it would be relatively easy to have a redis API for lua scripts that allows to define a critical section (esp. given it already works that way), should someone want that. I have no need for it though, nor for the worms you mention. ;)

Cheers,

Lawrence 

[Josiah Carlson]

Maybe I don't understand what you actually want then.

On Tue, Jun 4, 2013 at 10:47 PM, Lawrence <lawren...@gmail.com> wrote:

Hi Josiah,

Offering new commands EVALASYNC and EVALSHAASYNC, or even an optional argument for EVAL and EVALSHA that occurs after the KEYS and ARGV that tells Redis whether it can be executed asynchronously, that seems like a better answer than changing semantics if async execution is desired.

Yes. The current semantics where EVAL is atomic should not change.

 

That said, what's going to happen after async execution is allowed is that someone is going to want to have critical sections inside their Lua scripts, or is going to want to be able to try to acquire a lock via setnxex, be able to yield control to other scripts to finish executing, etc. All of those pieces adds not just one can of worms, but adds a case of different flavored worms.

It's all still synchronous. Lua would be using the evented architecture in redis. With a non-blocking API to redis. So yes, it can have async behaviour (i.e. the script as a whole is non-atomic), but it's not executing asynchronously to anything else.

If Lua scripts can be interrupted to execute other commands, scripts, or to perform network IO, regardless of the number of threads (which is actually 2 in some situations in Linux - a filesystem fsync() thread runs in the background for AOF-enabled Redis installs), data race conditions (which can introduce logic race conditions) are possible (but more or less likely, depending on the ultimate semantics).

Given redis is single-threaded I am assuming it would be relatively easy to have a redis API for lua scripts that allows to define a critical section (esp. given it already works that way), should someone want that. I have no need for it though, nor for the worms you mention. ;)

I'm not saying it is difficult. It's actually quite trivial (only perform Lua operations during main loop operation until the critical section or function is exited). My point was that what *seems* like a quick and easy thing that can solve an interactivity problem for some long-running scripts will inevitably lead to API expansion, increased code maintenance costs, and having to teach users what Redis means by "async Lua scripts". The latter one of those can get confusing considering Redis' limited concept of "transactional consistency" (only within MULTI/EXEC block or Lua script, but test/execute with WATCH/MULTI/EXC or Lua script).

Further, by allowing async execution of Lua scripts, any time writes can be interleaved between two Lua scripts (or arbitrary commands between Lua writes), you now have to not only send out information about the scripts to be executed to slaves and the AOF, but you also have to include information about the timing of script context switches. If the design requires that only read-only calls be performed during async execution raising an error on write, or upon a write call, Redis automatically switches script execution to atomic, then there will be confusion when using async scripts.

Are these necessarily deal breakers? No. I'm just trying to point out the future destination if this path is taken, and that something as "simple" as offering async execution of Lua scripts in Redis could have some ugly consequences on the code, documentation, and user support side of things. What makes me raise my eyebrow and stay in this discussion is that the stated problems that motivate async Lua script execution, can all be addressed with one or more read slaves - which Redis already supports.

Regards,

 - Josiah

[Felix Gallo]

Lua is already pretty persnickety inside redis.  It's pretty clear that Salvatore doesn't intend for it to be used as a full featured scripting language so much as a way to atomically execute some limited logic as a group.   The behavior of lua in slaves and clusters and in the presence of long running loops and so forth is a very tough problem that I think he's rightly punted on.

Fortunately it's easy to write async code to modify redis while continuing operation; just do it in the application layer.

F.