Lua script execution time - Redis DB

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Lua script execution time
Received-SPF: pass (google.com: domain of josiah.carl...@gmail.com designates 209.85.220.46 as permitted sender) client-ip=209.85.220.46;
MIME-Version: 1.0
In-Reply-To: <CADcB0yjDPr1zPeFWfKrtDh5ZN2PrCHKZReA12Cwjifo-KLg...@mail.gmail.com>
References: <cccd0756-dd1f-4bd4-b414-31afcec6c996@googlegroups.com>
	<CAH7wyv3LecgXNehKHZHiAZoW4AJx0N4MFR_Jf0cjyS4DVYw...@mail.gmail.com>
	<CAF95VAxXOHFqwZ+Vfv5zZJE+PfgrWz31+LA+JF2H=L0N33t...@mail.gmail.com>
	<CAF95VAxNsqNNJ2U3dcZbvg8V7U4RzpMhsMQofutqMCSgHrH...@mail.gmail.com>
	<35061c0a-3b56-4d3b-932e-38a341cd742e@googlegroups.com>
	<b39be74f-b708-4449-8692-08a379b33f2a@googlegroups.com>
	<24c16b5d-4c48-4e43-99b1-bc2c5814020c@googlegroups.com>
	<CADcB0yg7=8sTdgCNYV22PKGMxMopk1HP4x_PgHcYwk8FALg...@mail.gmail.com>
	<ec27ab26-997c-44b7-86de-42f4f5f35871@googlegroups.com>
	<CADcB0yhz70wwMSm=4RPqkYQ8pc0hpGJawbwe2GLATsdi5x3...@mail.gmail.com>
	<CAJr4V0BFW6vbLGyEipcz1aT7S-48Ppd-1tW2QibdjT7oDJu...@mail.gmail.com>
	<CADcB0yjDPr1zPeFWfKrtDh5ZN2PrCHKZReA12Cwjifo-KLg...@mail.gmail.com>
Date: Mon, 5 Nov 2012 16:47:42 -0800
Message-ID: <CADcB0yh5QF2CPfi397pRKwgvWpfH+J1CP9gb2mK8QqDcdbn...@mail.gmail.com>
Subject: Re: Lua script execution time
From: Josiah Carlson <josiah.carl...@gmail.com>
To: ar...@weisberg.ws, redis-db@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Sorry, bad script. Don't execute that.

A better script is:

redis.call('psetex', KEYS[1], ARGV[1], ARGV[2])
local i =3D 0
while redis.call('get', KEYS[1]) and i < 10000 do
    i =3D i + 1
end
redis.call('set', KEYS[1], i)
return i

Which at least includes a fall-through if someone is running it
multiple times, and it actually works :P

 - Josiah

On Mon, Nov 5, 2012 at 4:41 PM, Josiah Carlson <josiah.carl...@gmail.com> w=
rote:
> Re-adding the Redis mailing list.
>
> On Mon, Nov 5, 2012 at 3:36 PM, Ariel Weisberg <arielweisb...@gmail.com> =
wrote:
>> Hi,
>>
>> Looks like I didn't include the group in my comment?
>>
>> Ah, I just read the excellent documentation that describes how expiratio=
n is
>> implemented.
>>
>> The time returned from within a Lua script doesn't change right? As a us=
er I
>> would expect all keys to be expired before/after script execution and no=
t
>> during since time is "frozen".
>
> There is nothing that stops time from progressing inside Redis while a
> script is executing, and you don't need to call any time functions to
> tickle the edge-case.
>
> Imagine for a moment that you have a script that takes a variable
> amount of time to execute, based on the performance of a server. Maybe
> something related to a large union operation over a large number of
> sets or sorted sets. If you are using key expiration in the wrong way,
> you could end up with inconsistent data on your servers. As an
> example, you can see different results from a master and slave of
> different performance capabilities with the following:
>
> redis.call('psetex', KEYS[1], ARGV[1], ARGV[2])
> local i =3D 0
> while redis.call('get', KEYS[1]) ~=3D '' do
>     i =3D i + 1
> end
> redis.call('set', KEYS[1], i)
> return i
>
> Call the above with...
>
> EVAL ... 1 foo 1 bar
>
> Depending on the performance of your server, the destination key will
> have a different number.
>
> About the only advice I can give dealing with expirations of this
> nature is "don't do that".
>
> Then again, this does offer an opportunity to create timers based on
> Redis expiration, but that is a *really* bad idea.
>
>> If a script is running any keys you set cannot be expired when the scrip=
t
>> executes at a replica because the time of the script by definition prece=
des
>> the expiration time. If a key expires on access the script will never se=
e
>> the key and the result is the same at the replicas because the script ru=
ns
>> with the same timestamp everywhere.
>
> Time doesn't pause, so your premise is incorrect.
>
>> Or do the replicas not contain the same timestamp as the master?
>
> That too.
>
> Regards,
>  - Josiah
>
>> On Mon, Nov 5, 2012 at 4:01 PM, Josiah Carlson <josiah.carl...@gmail.com=
>
>> wrote:
>>>
>>> Expiration is typically executed on the master with DEL during random
>>> expiration or when a key that should be expired is accessed (the DEL
>>> is syndicated to slaves).
>>>
>>> In the middle of a script, you can't really inject a DEL (because you
>>> are executing a script and not individual commands), and injecting a
>>> delete before the script begins execution doesn't work because you can
>>> set a key with expiration.
>>>
>>>  - Josiah
>>>
>>> On Mon, Nov 5, 2012 at 11:51 AM, Ariel Weisberg <arielweisb...@gmail.co=
m>
>>> wrote:
>>> > Hi,
>>> >
>>> > You can make operations against keys with expiration deterministic by
>>> > having
>>> > the master manage key expiration.
>>> >
>>> > As long as all mutations are sequenced by the master and its replicat=
ion
>>> > stream you don't have to worry about asynchronous events like
>>> > expiration.
>>> > The tradeoff is that slaves have to maintain enough state and behavio=
r
>>> > to
>>> > correctly resume where the master leaves off which means two code pat=
hs
>>> > instead of one.
>>> >
>>> > Regards,
>>> > Ariel
>>> >
>>> >
>>> > On Monday, November 5, 2012 1:57:44 PM UTC-5, Josiah Carlson wrote:
>>> >>
>>> >> On Mon, Nov 5, 2012 at 10:03 AM, Pierre Chapuis
>>> >> <catwell...@catwell.info> wrote:
>>> >> > Le lundi 5 novembre 2012 17:42:21 UTC+1, pete_c a =E9crit :
>>> >> >>
>>> >> >> I believe the "scripts as pure functions" quote is taken from the
>>> >> >> Redis
>>> >> >> documentation on EVAL.
>>> >> >
>>> >> > Oh, that's true, I had not noticed. Of course everything that is s=
aid
>>> >> > in
>>> >> > the
>>> >> > documentation (and what Josiah said) is right. I would not use the
>>> >> > wording
>>> >> > "pure function" though: to me it means a function that has no side
>>> >> > effects,
>>> >> > meaning it could not write to Redis. A "pure function" should have=
,
>>> >> > among
>>> >> > other properties, idempotence. This is not true of Redis scripts
>>> >> > (which
>>> >> > as
>>> >> > Josiah explained can decide to write or not depending on the
>>> >> > existence
>>> >> > of a
>>> >> > key).
>>> >> >
>>> >> >  That being said it appears that there may be an actual problem wi=
th
>>> >> > expiration. What happens if you create key "A" with a TTL, then ru=
n a
>>> >> > script
>>> >> > that creates key "B" if "A" exists (which is "legal")?
>>> >>
>>> >> There is only a problem if you set the expiration to be arbitrarily
>>> >> low, and that expiration passes before script execution completes. B=
ut
>>> >> that's in interesting edge case. You may want to post a bug about it=
,
>>> >> though I suspect the answer will be "don't do that".
>>> >>
>>> >> > Even worse, couldn't you actually do that *without* scripting? For
>>> >> > instance
>>> >> > if you set a TTL on a Set and subsequently use SMOVE? I guess ther=
e's
>>> >> > some
>>> >> > kind of protection against that in the replication mechanism?
>>> >>
>>> >> This is another one of those "don't do that" situations.
>>> >>
>>> >>  - Josiah
>>
>>
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
