Re: Lua script execution time

[Yiftach Shoolman]

redis-cli --latency -h `host` -p `port`

More info here

On Mon, Nov 5, 2012 at 2:04 PM, Peter Choo <pete...@gmail.com> wrote:

Due to Redis not allowing write commands after TIME has been called,  How can I benchmark the execution time of a small lua script?

Is there a function I can call to ask how long it took to run the last script?  I am not interested in the data transfer time, just how long it took to run in Redis.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/G6XxpMoamdMJ.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--

Yiftach Shoolman
+972-54-7634621

[Marc Gravell]

OK, this is ***really*** nasty: I **strongly** suggest you don't use this, because it suggests that PTTL is incorrectly not being flagged as non-deterministic (unless I'm missing something - Salvatore might be able to advise more):

redis.call('setex', KEYS[1], 60, 'timing')

for i = 1,100000 do

    redis.call('set', 'ignore','abc')

end

local timeTaken = 60000 - redis.call('pttl', KEYS[1])

redis.call('del', KEYS[1])

return timeTaken

this is **really** ugly; note here that KEYS[1] is just a nasty guid I used for a scratch key, and `ignore` is just something to make the time detectable (that isn't meant to be illustrative).

Seriously, don't do this!

I would probably just time it at the client (which includes bandwidth costs), or issue a TIME before and after the EVAL/EVALSHA (which may include timings from other connections), and call it "close enough".

But I would consider the fact that the above *works* to be a bug... 

Marc

Regards,

Marc

[Marc Gravell]

Hmmm.... maybe calling it a bug is a bit strong - it is really a special case of violating the "scripts as pure functions" guidance - however, since every key could in theory have an expiration, I'm not quite sure how we can guarantee a pure function unless we never use lua to talk to a key that has an expiry.

Intriguing.

Marc

--
Regards,

Marc

[pete_c]

Thanks for the reply, while I believe this is a usable solution (however un-recommended you say it is), I don't think it will be accurate enough for my purposes, I believe the script will run in less than a millisecond, but I would like to know how much less.  I should have specified that above, apologies.

[pete_c]

Just a note, I'm looking for a response time preferably in microseconds.  I believe my script will be running in less than a millisecond, but need to know how much less.  Thanks all!

[M. Edward (Ed) Borasky]

1. If your Redis server is a guest virtual machine, clocks inside it
may not have the precision you desire nor necessarily be synchronized
closely with NTP. Most likely you're going to be getting better
numbers if you time the operation from outside the server.

2. My definition of response time is that seen by an *external* user.
So it includes network time from the moment you send the last byte of
the request to the moment you receive the last byte of the response.
The capacity planning community loves to argue about this definition,
so you may see other start and end times defined.

3. In theory the slowlog with 'slowlog-log-slower-than' set to zero
will capture the execution time of every command to a data structure
inside the Redis server, which your Redis client can retrieve using
the SLOWLOG GET operation. However, as I noted in 1 above, the numbers
may be bogus inside a virtual machine.

I'm experimenting with the slowlog command at the moment, and I'll be
testing both how much overhead it adds and how accurate it is inside a
kvm and a VirtualBox guest.

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/redis-db/-/5meidHy81s8J.

>
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.



--

Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://znmeb.github.com/Computational-Journalism-Publishers-Workbench/

How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?

[Pierre Chapuis]

Le lundi 5 novembre 2012 13:36:23 UTC+1, Marc Gravell a écrit :

Hmmm.... maybe calling it a bug is a bit strong - it is really a special case of violating the "scripts as pure functions" guidance - however, since every key could in theory have an expiration, I'm not quite sure how we can guarantee a pure function unless we never use lua to talk to a key that has an expiry.

What guidance are you talking about? Scripts were never meant to be pure functions.

The main caveat when using scripts is that if you want them to work with Redis Cluster (someday, when it actually exists) you should not touch keys that are not in KEYS (eg. put keys in ARGS, hardcode them in the script, build them in the script, fetch them from a set...).

[pete_c]

I believe the "scripts as pure functions" quote is taken from the Redis documentation on EVAL.  I'm fairly new to Redis and Lua myself, so I might be breaking rules somewhere regarding the KEYS/ARGV principle.  But seeing as the fields I need to access will always be the same, I don't see a problem with this, and currently our need for Lua is very small.  

@M Edward Borasky:

thanks for the clarification on the virtual server response times.  My curiosity for this question is that while the script is running, no other operations can be performed in Redis.  I want to know how long it takes on average to run this script, so we can get a figure for how many polls can be made per second, taking in to account future scalability.

Thanks for your responses.

[Josiah Carlson]

During AOF and slave sync, the Lua script itself and the arguments are
executed, not the commands that were executed as part of the script.
So in order for the result to be the same, it must essentially execute
in a non-deterministic fashion, assuming identical inputs.

If the function relies on timing, legitimate random numbers, etc.,
then it may not execute correctly.

How that plays out in practice is that you can decide to write based
on the existence/non-existence of a key (because that will be
consistent as you replay an AOF), but you can't decide what to write
based on internally generated time (it would lead to inconsistency
when replaying an AOF, which is why it was disabled, and why random
actually returns a consistent sequence of data).

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/redis-db/-/bBSPTYYKRHMJ.

[Marc Gravell]

Just to confirm, I was indeed referring to the section heading in the "EVAL" documentation. I should have been more specific. The key point it makes there is about the script being deterministic (hence the pre-seeded "random", and the tweaking of sort-orders) - I was simply noting that deterministic is tricky if the keys you are working on are on the cusp of expiring.

It would probably work ok if time was suspended (in terms of TTL / PTTL) during the script, and the exec time was broadcast during the AOF/slave - but the script above demonstrates that time is *not* suspended, therefore it cannot be fully deterministic.

Marc

---

From: pete_c
Sent: 05/11/2012 16:42
To: redi...@googlegroups.com
Subject: Re: Lua script execution time

I believe the "scripts as pure functions" quote is taken from the Redis documentation on EVAL.  I'm fairly new to Redis and Lua myself, so I might be breaking rules somewhere regarding the KEYS/ARGV principle.  But seeing as the fields I need to access will always be the same, I don't see a problem with this, and currently our need for Lua is very small.  

@M Edward Borasky:

thanks for the clarification on the virtual server response times.  My curiosity for this question is that while the script is running, no other operations can be performed in Redis.  I want to know how long it takes on average to run this script, so we can get a figure for how many polls can be made per second, taking in to account future scalability.

Thanks for your responses.

On Monday, 5 November 2012 16:20:50 UTC, Pierre Chapuis wrote:

Le lundi 5 novembre 2012 13:36:23 UTC+1, Marc Gravell a écrit :

Hmmm.... maybe calling it a bug is a bit strong - it is really a special case of violating the "scripts as pure functions" guidance - however, since every key could in theory have an expiration, I'm not quite sure how we can guarantee a pure function unless we never use lua to talk to a key that has an expiry.

What guidance are you talking about? Scripts were never meant to be pure functions.

The main caveat when using scripts is that if you want them to work with Redis Cluster (someday, when it actually exists) you should not touch keys that are not in KEYS (eg. put keys in ARGS, hardcode them in the script, build them in the script, fetch them from a set...).

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/ArhPTjYdbzoJ.

[Javier Guerra Giraldez]

On Mon, Nov 5, 2012 at 12:48 PM, Josiah Carlson
<josiah....@gmail.com> wrote:
> During AOF and slave sync, the Lua script itself and the arguments are
> executed, not the commands that were executed as part of the script.

I think Redis is the only system i know that does this. while a
perfectly valid choice, it's not what some expect.

sometimes i wish it was possible to do the other way for some scripts.
maybe a setting, or a commant at the start of the script would switch
the AOF/replication to a "results, not script" method. of course, it
would be practical only for those scripts that generate a bound number
of commands.

--
Javier

[Pierre Chapuis]

Le lundi 5 novembre 2012 17:42:21 UTC+1, pete_c a écrit :

I believe the "scripts as pure functions" quote is taken from the Redis documentation on EVAL.

Oh, that's true, I had not noticed. Of course everything that is said in the documentation (and what Josiah said) is right. I would not use the wording "pure function" though: to me it means a function that has no side effects, meaning it could not write to Redis. A "pure function" should have, among other properties, idempotence. This is not true of Redis scripts (which as Josiah explained can decide to write or not depending on the existence of a key).

 That being said it appears that there may be an actual problem with expiration. What happens if you create key "A" with a TTL, then run a script that creates key "B" if "A" exists (which is "legal")?

Even worse, couldn't you actually do that *without* scripting? For instance if you set a TTL on a Set and subsequently use SMOVE? I guess there's some kind of protection against that in the replication mechanism?

[Josiah Carlson]

On Mon, Nov 5, 2012 at 10:03 AM, Pierre Chapuis
<catwell...@catwell.info> wrote:
> Le lundi 5 novembre 2012 17:42:21 UTC+1, pete_c a écrit :
>>
>> I believe the "scripts as pure functions" quote is taken from the Redis
>> documentation on EVAL.
>
> Oh, that's true, I had not noticed. Of course everything that is said in the
> documentation (and what Josiah said) is right. I would not use the wording
> "pure function" though: to me it means a function that has no side effects,
> meaning it could not write to Redis. A "pure function" should have, among
> other properties, idempotence. This is not true of Redis scripts (which as
> Josiah explained can decide to write or not depending on the existence of a
> key).
>
> That being said it appears that there may be an actual problem with
> expiration. What happens if you create key "A" with a TTL, then run a script
> that creates key "B" if "A" exists (which is "legal")?

There is only a problem if you set the expiration to be arbitrarily
low, and that expiration passes before script execution completes. But
that's in interesting edge case. You may want to post a bug about it,
though I suspect the answer will be "don't do that".

> Even worse, couldn't you actually do that *without* scripting? For instance
> if you set a TTL on a Set and subsequently use SMOVE? I guess there's some
> kind of protection against that in the replication mechanism?

This is another one of those "don't do that" situations.

- Josiah

[Marc Gravell]

My understanding is that the replication includes the timestamp, so is robust for regular operations. It is trickier if, as the documentation suggests, EVAL just sends the outer command - that will have the start time of the script, but unless I'm missing something the operations after that won't have reliable times (and could depend on CPU performance). This is an edge case, but could get really messy. I'm genuinely intrigued as to whether this is an actual problem, or an invented one :)

---

From: Pierre Chapuis
Sent: 05/11/2012 18:03

To: redi...@googlegroups.com
Subject: Re: Lua script execution time

Le lundi 5 novembre 2012 17:42:21 UTC+1, pete_c a écrit :

I believe the "scripts as pure functions" quote is taken from the Redis documentation on EVAL.

Oh, that's true, I had not noticed. Of course everything that is said in the documentation (and what Josiah said) is right. I would not use the wording "pure function" though: to me it means a function that has no side effects, meaning it could not write to Redis. A "pure function" should have, among other properties, idempotence. This is not true of Redis scripts (which as Josiah explained can decide to write or not depending on the existence of a key).

 That being said it appears that there may be an actual problem with expiration. What happens if you create key "A" with a TTL, then run a script that creates key "B" if "A" exists (which is "legal")?

Even worse, couldn't you actually do that *without* scripting? For instance if you set a TTL on a Set and subsequently use SMOVE? I guess there's some kind of protection against that in the replication mechanism?

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/imWvOg_Wg4wJ.

[Josiah Carlson]

Re-adding the Redis mailing list.

On Mon, Nov 5, 2012 at 3:36 PM, Ariel Weisberg <arielw...@gmail.com> wrote:
> Hi,
>
> Looks like I didn't include the group in my comment?
>
> Ah, I just read the excellent documentation that describes how expiration is
> implemented.
>
> The time returned from within a Lua script doesn't change right? As a user I
> would expect all keys to be expired before/after script execution and not
> during since time is "frozen".

There is nothing that stops time from progressing inside Redis while a
script is executing, and you don't need to call any time functions to
tickle the edge-case.

Imagine for a moment that you have a script that takes a variable
amount of time to execute, based on the performance of a server. Maybe
something related to a large union operation over a large number of
sets or sorted sets. If you are using key expiration in the wrong way,
you could end up with inconsistent data on your servers. As an
example, you can see different results from a master and slave of
different performance capabilities with the following:

redis.call('psetex', KEYS[1], ARGV[1], ARGV[2])
local i = 0
while redis.call('get', KEYS[1]) ~= '' do
i = i + 1
end
redis.call('set', KEYS[1], i)
return i

Call the above with...

EVAL ... 1 foo 1 bar

Depending on the performance of your server, the destination key will
have a different number.

About the only advice I can give dealing with expirations of this
nature is "don't do that".

Then again, this does offer an opportunity to create timers based on
Redis expiration, but that is a *really* bad idea.

> If a script is running any keys you set cannot be expired when the script
> executes at a replica because the time of the script by definition precedes
> the expiration time. If a key expires on access the script will never see
> the key and the result is the same at the replicas because the script runs
> with the same timestamp everywhere.

Time doesn't pause, so your premise is incorrect.

> Or do the replicas not contain the same timestamp as the master?

That too.

Regards,
- Josiah

> On Mon, Nov 5, 2012 at 4:01 PM, Josiah Carlson <josiah....@gmail.com>
> wrote:
>>
>> Expiration is typically executed on the master with DEL during random
>> expiration or when a key that should be expired is accessed (the DEL
>> is syndicated to slaves).
>>
>> In the middle of a script, you can't really inject a DEL (because you
>> are executing a script and not individual commands), and injecting a
>> delete before the script begins execution doesn't work because you can
>> set a key with expiration.
>>
>> - Josiah
>>
>> On Mon, Nov 5, 2012 at 11:51 AM, Ariel Weisberg <arielw...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > You can make operations against keys with expiration deterministic by
>> > having
>> > the master manage key expiration.
>> >
>> > As long as all mutations are sequenced by the master and its replication
>> > stream you don't have to worry about asynchronous events like
>> > expiration.
>> > The tradeoff is that slaves have to maintain enough state and behavior
>> > to
>> > correctly resume where the master leaves off which means two code paths
>> > instead of one.
>> >
>> > Regards,
>> > Ariel

[Josiah Carlson]

Sorry, bad script. Don't execute that.

A better script is:

redis.call('psetex', KEYS[1], ARGV[1], ARGV[2])
local i = 0

while redis.call('get', KEYS[1]) and i < 10000 do

i = i + 1
end
redis.call('set', KEYS[1], i)
return i

Which at least includes a fall-through if someone is running it
multiple times, and it actually works :P

- Josiah

[Ariel Weisberg]

Sending again to copy the list. Also changed some grammar.

Hi,

Does it have to be that way? Scripts are supposed to be atomic WRT to other operations. Why not make them atomic WRT to expiration. It seems like an unnecessary caveat to keep around long term.

There is no technical reason you can't make expiration deterministic when used with scripting. If you abstract away the clock implementation you can have a separate one that is logical time (set via replication, or only updated between scripts and commands) and another for wall clock time that is used for logging and performance measurements. Then code that needs logical time can go to the right clock and you can update the clock at the replica before executing every statement that depends on the logical time at the master.

Regards,

Ariel

[Pierre Chapuis]

Le lundi 5 novembre 2012 20:23:02 UTC+1, Marc Gravell a écrit :

My understanding is that the replication includes the timestamp, so is robust for regular operations. It is trickier if, as the documentation suggests, EVAL just sends the outer command - that will have the start time of the script, but unless I'm missing something the operations after that won't have reliable times (and could depend on CPU performance). This is an edge case, but could get really messy. I'm genuinely intrigued as to whether this is an actual problem, or an invented one :)

If it were an actual problem I would have reported it as a bug already ;)

That being said the "SMOVE on an expiring Set" case worried me. The same thing could happen with other commands that read a key and write to another such as RPOPLPUSH, or even for SETNX.

The thing that (afaik) fixes that - and that I had forgotten yesterday - is that slaves do not actually expire keys, they wait for a DEL from the master (see https://github.com/antirez/redis/issues/187).

Now that I have remembered that I am worried about anything that changes expiration times in scripts and how that is replicated... I have never actually tried to do that in practice though so that worry is theoretical!

[pete_c]

Hi all, and thanks for your replies, it's certainly cleared up a lot in my mind.  I think for what I require (I was using TIME purely to benchmark, and not to affect the running of the script/output in any way, other than to let me know how long it's been running for serverside.) SETEX and PTTL will be the closest I can get.  

If only there was a UTTL...

[Josiah Carlson]

There may be a bug.

1. Start up a telnet client, and connect to Redis.
2. Inside that telnet client, send the following:
"*1
$4
SYNC
"
Without the quotes, and using \r\n line terminators (the default for telnet).
3. Once it is done syncing, go ahead and run the second script that I showed.
4. Note that a DEL is not sent to your slave telnet?

- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/redis-db/-/2di8MbxIxDkJ.