Dependent transactions

[Benjamin Polidore]

As far as I can tell, you cannot do dependent transactions (ie, use the results of one redis statement in another inside of MULTI).  I have been using the following approach for doing dependent transactions to make sure that redis always has a durable, roll-backable copy of the state even if my application has to do work in between redis calls and then dies.  I'm not sure if this is the best way, though, so I would love some feedback.  I have simplified this a bit (I'm using strings instead of hashes in my example), but you can get the picture.

Goal

Move $100 from account A to account B only if account A has at least $100.  Do this in such a way that is atomic-- ie, he can't transfer the same $100 to B and C at the same time.

The issue is that if you move the $100 to B and that leaves A negative, you have to move it back.  This fact has to be checked in an application, but if the application dies right after it moved the money from A to B, you won't know how A's balance went negative without some kind of transaction log.

Solution

transactionID = INCR user:a:nextTransactionId //an application generated guid could be used here instead. 

MULTI

  newBalanceA = DECRBY user:a:balance 100

  SET user:a:opentransaction:transactionID -100

EXEC

if newBalanceA < 0  //this check is in my application

   MULTI

     newBalanceA = INCRBY user:a:balance 100 //put the money back; doesn't have enough

     DEL user:a:opentransaction:transactionID

   EXEC

else

  MULTI

    newBalanceB = INCRBY user:b:balance 100 //give the money to B

    DEL user:a:opentransaction:transactionID

  EXEC

Then if the application dies after I take the money out of A, but before I either give it to B or or return it to A given that he doesn't have enough money, I could look at the open transactions and undo anything in them. 

A couple of obvious questions that people might have about this implementation:

• Why don't I just take it from A and give it to B in a MULTI
• Well, then B could spend it before I had confirmed that A actually had $100
• Why not just check if A has $100, then give it to B inside a MULTI
• Because in the time between the check and the execution of the multi, A could spend the money again

Anyway, I'm not asking for a feature, as this methodology works ok even if it requires an extra round trip to the (very fast) server, but it could potentially be simplified if you could "script" inside a multi.  something like:

MULTI

  $a balance = GET user:a:balance

  if $a > 100

    $newBalanceA = DECRBY user:a:balance 100

    $newBalanceB = INCRBY user:b:balance 100

  else

    $err = 'insufficient funds'

EXEC

Any feedback would be great.

Thanks,

bp

  

[Josiah Carlson]

Good workaround for those who have to deal with this currently.

The Lua scripting that will be shipped with the forthcoming Redis
scripting branch (whenever that happens) will make this even easier.

- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/HEbIeVHF1VsJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
>

[Javier Guerra Giraldez]

On Wed, Jun 22, 2011 at 4:39 PM, Benjamin Polidore <poli...@gmail.com> wrote:
> Move $100 from account A to account B only if account A has at least $100.
>  Do this in such a way that is atomic-- ie, he can't transfer the same $100
> to B and C at the same time.

what about using WATCH? ie:

WATCH balanceA
GET balanceA
if balanceA >= 100 --- check on the client app
MULTI
DECRBY balanceA 100
INCRBY balanceB 100
EXEC
else
UNWATCH

--
Javier

[Josiah Carlson]

Any key with high volume updates where race conditions are
possible/likely will run into fairly common failures on the client
side. Dealing with race conditions the same way as you deal with "out
of money" issues is convenient.

Your multi/exec solution is also good assuming low volumes of update contention.

- Josiah

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.

[Benjamin Polidore]

What if I have multiple clients sharing the same redis database?

[Benjamin Polidore]

There shouldn't be too many updates on this particular key since there will be many, many clients, but a single client's transfer of funds has to be fully transactional, and I can't think of a better way to do it in redis.

[Benjamin Polidore]

so with the lua scripting branch, you'll be able to do something like what I put in my pseudo code, and tell redis to do all commands transactionally? that would be great.

[Josiah Carlson]

Yes.

- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/redis-db/-/-MYmhV_8e7IJ.

[David Yu]

On Thu, Jun 23, 2011 at 6:18 AM, Benjamin Polidore <poli...@gmail.com> wrote:

so with the lua scripting branch, you'll be able to do something like what I put in my pseudo code, and tell redis to do all commands transactionally? that would be great.

Yep.

With a traditional transactional db, usually the writes are incremental as conditions are met and the whole transaction can be aborted/rolled back when the very last condition is not met.

With redis, you don't write until the last condition is met, which makes your lua function atomic.

This is atleast what I'm doing atm.  If anyone has better other techniques in doing this with redis, I'd be interested to know.

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/-MYmhV_8e7IJ.

To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--
When the cat is away, the mouse is alone.
- David Yu

[Javier Guerra Giraldez]

On Wed, Jun 22, 2011 at 7:12 PM, David Yu <david....@gmail.com> wrote:
> With redis, you don't write until the last condition is met, which makes
> your lua function atomic.

i think the real reason why Lua operations are atomic is because
they're a _single_ operation, and Redis is single threaded.

--
Javier

[David Yu]

If you compare redis to voltdb (both in-memory and both have no locks/etc), the latter supports full transactions where you can rollback any changes you made.  With redis, you can't ... w/c is why you have to make sure each condition is met before you write.

 

--
Javier

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Javier Guerra Giraldez]

On Thu, Jun 23, 2011 at 6:44 AM, David Yu <david....@gmail.com> wrote:
> On Thu, Jun 23, 2011 at 7:37 PM, Javier Guerra Giraldez <jav...@guerrag.com>
> wrote:
>>
>> On Wed, Jun 22, 2011 at 7:12 PM, David Yu <david....@gmail.com> wrote:
>> > With redis, you don't write until the last condition is met, which makes
>> > your lua function atomic.
>>
>> i think the real reason why Lua operations are atomic is because
>> they're a _single_ operation, and Redis is single threaded.
>
> If you compare redis to voltdb (both in-memory and both have no locks/etc),
> the latter supports full transactions where you can rollback any changes you
> made.  With redis, you can't ... w/c is why you have to make sure each
> condition is met before you write.

not sure if i get you.... do you mean that Lua commands are wrapped in
an implicit MULTI...EXEC?

what happens if, for example, i have a Lua function that:

1:- write to a Redis Hash
2:- dies of a Lua error (division by zero, call to a nil value, whatever)
3:- writes to a second Hash

obviously step 3 wouldn't be performed, but what about step 1's write
operation? would it be performed before the error? or are every
modification delayed until Lua returns and applied only if there
wasn't an error?

--
Javier

[David Yu]

On Thu, Jun 23, 2011 at 11:35 PM, Javier Guerra Giraldez <jav...@guerrag.com> wrote:

On Thu, Jun 23, 2011 at 6:44 AM, David Yu <david....@gmail.com> wrote:
> On Thu, Jun 23, 2011 at 7:37 PM, Javier Guerra Giraldez <jav...@guerrag.com>
> wrote:
>>
>> On Wed, Jun 22, 2011 at 7:12 PM, David Yu <david....@gmail.com> wrote:
>> > With redis, you don't write until the last condition is met, which makes
>> > your lua function atomic.
>>
>> i think the real reason why Lua operations are atomic is because
>> they're a _single_ operation, and Redis is single threaded.
>
> If you compare redis to voltdb (both in-memory and both have no locks/etc),
> the latter supports full transactions where you can rollback any changes you
> made.  With redis, you can't ... w/c is why you have to make sure each
> condition is met before you write.

not sure if i get you.... do you mean that Lua commands are wrapped in
an implicit MULTI...EXEC?

what happens if, for example, i have a Lua function that:

1:- write to a Redis Hash
2:- dies of a Lua error (division by zero, call to a nil value, whatever)
3:- writes to a second Hash

obviously step 3 wouldn't be performed, but what about step 1's write
operation?

It will go through because you already committed to writing the data.

You now have an inconsistent db.

The simple solution is to perform 2 iterations.

First pass will be to check if all conditions are met.

Second pass is to apply(write) it.

You wont need rollback semantics because the first iteration ensures 100% that the writes will be successful.

Note also you need to check programming errors on lua since that can also make your db inconsistent.

There's no room for error because you cannot rollback.

Also see http://nosql.mypopescu.com/post/568248288/redis-multi-exec-discard-and-transactions

would it be performed before the error? or  are every
modification delayed until Lua returns and applied only if there
wasn't an error?

--

Javier

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Javier Guerra Giraldez]

On Thu, Jun 23, 2011 at 10:51 AM, David Yu <david....@gmail.com> wrote:
> On Thu, Jun 23, 2011 at 11:35 PM, Javier Guerra Giraldez
> <jav...@guerrag.com> wrote:
>> 1:- write to a Redis Hash
>> 2:- dies of a Lua error (division by zero, call to a nil value, whatever)
>> 3:- writes to a second Hash
>>
>> obviously step 3 wouldn't be performed, but what about step 1's write
>> operation?
>
> It will go through because you already committed to writing the data.
> You now have an inconsistent db.

great, that was i assumed. and (IMHO) in line with the principle of
least surprise.

then:

A) Lua commands are atomic because of Redis single-threadness, not
because of check-before-commit.

B) database consistency is responsibility of the user, with or
without Lua scripts.

--
Javier

[Benjamin Polidore]

 A) Lua commands are atomic because of Redis single-threadness, not
because of check-before-commit.

This doesn't seem right. Sounds like lua scripts are synchronized, not atomic since half of your script can commit even if other half fails.

Javier

[Javier Guerra Giraldez]

On Thu, Jun 23, 2011 at 11:42 AM, Benjamin Polidore <poli...@gmail.com> wrote:
>  A) Lua commands are atomic because of Redis single-threadness, not
> because of check-before-commit.
> This doesn't seem right. Sounds like lua scripts are synchronized, not
> atomic since half of your script can commit even if other half fails.

doh!

so, the correct term is synchronized? or more like 'sequential' (or
sequentialized) ?

--
Javier