Re: getrange, bitmap and lua

[Josiah Carlson]

index = 1
while index <= string.len(data) do
index = string.find(data, "[^\0]", index)
if index != -1 then
# do something with this location
end
index = index + 1
end

That should repeatedly find non-nulls in the string, though my Lua is rusty.

Regards,
- Josiah

On Wed, Aug 22, 2012 at 6:48 PM, K <ka...@viki.com> wrote:
> After doing a bitop, I'll looking to get the index of all set members. My
> initial approach was to GETRANGE(key, 0, -1) and let the client loop through
> the result looking for set bits (in node). I decided to try doing the filter
> in LUA and thus only returning set index. I was surprised that doing it
> server side was slower. I *think* part of the problem is that GETRANGE's
> response is treated as a string (whereas with node I can manipulate the
> buffer directly). This is the code I have:
>
> local ids = {}
> local data = redis.call('getrange', key, 0, -1)
> for i = 1, #data do
> if data:sub(i,i) == "\u0001" then
> table.insert(ids, i)
> end
> end
> return ids
>
>
> I realize this can be optimized by checking a subset with and skipping where
> BITCOUNT = 0...but I'm still curious if there's a faster way to do this in
> LUA?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/8c_rl_uSlxwJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

[K]

Thanks it worked and sped things up considerably.

Anyone using this example will want to declare index as local.

K

[Josiah Carlson]

Alternatively, you could use the gmatch() function, which offers an
iterator (which may reduce per-item overhead). But I'm not sure if it
gives you an offset into the string.

- Josiah

> https://groups.google.com/d/msg/redis-db/-/TiI70vnPDREJ.

[Chris Love]

Josiah

Would you also recommend a local import of string.find and string.len?

local find = string.find -- my syntax may be off

[Josiah Carlson]

That does seem like an improvement, yes.

Note that I didn't intend my suggestion to be the ultimate final
answer (I didn't even verify that it was correct Lua syntax, and 99%
of my Lua experience is actually around modifying pre-existing World
of Warcraft add-ons 6+ years ago). But if you find other performance
enhancements, feel free to attribute them to my genius ;)

Regards,
- Josiah

> https://groups.google.com/d/msg/redis-db/-/PfNQDXKtdGwJ.

[K]

Sorry to resurrect  an old-ish thread, but I got to working on this project again, and I no longer think that the initial version (nor the follow up) ever really worked. They work fine if you have sparse data, but as soon as multiple bits are set within the same byte it falls apart.

Seems unfortunate that Redis is bit-aware but LUA is not. It certainly limits what one can do. Here's code that reproduces the issue. Note that bitcount works as expected (returning 3):

redis.setbit('test', 0, 1)

redis.setbit('test', 1, 1)

redis.setbit('test', 2, 0)

redis.setbit('test', 3, 1)

script = <<eos

  local ids = {}

  local data = redis.call('getrange', KEYS[1], 0, -1)

  for i = 1, #data do

    if data:sub(i,i) == '\1' then

      table.insert(ids, i)

    end

  end

  return ids

eos

p redis.eval(script, ['test'])

[Josiah Carlson]

Lua is a turing-complete programming language, you can make Lua as
bit-aware as desired.

More specifically; multiply offsets by 8 to get the base bit address,
then convert the byte into 8 bits via a sequence of modulos,
subtractions, and divisions (Lua doesn't seem to have bitwise & or >>
operators, but maybe that's in a library?).

Regards,
- Josiah

> https://groups.google.com/d/msg/redis-db/-/gW80_qwPqdAJ.

[M. Edward (Ed) Borasky]

http://lua-users.org/wiki/BitwiseOperators

Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://j.mp/QCsXOr

How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?

[Josiah Carlson]

Perfect :)

- Josiah

[M. Edward (Ed) Borasky]

Looks like there are bitops in Lua 5.2 but the Lua in Redis 2.6 is 5.1.

On Mon, Sep 17, 2012 at 10:49 PM, Josiah Carlson