Analytics scenario

[teleo]

Hi,

I need to implement memory-efficient time series, very similar to redis-timeseries project, with the following difference:

I would like my keys to contain only samples, and have these samples in the form of 64-bit integers. For example, a 2-hour string with 30 second step would be 240x8 bytes long. Because this is a plain array with well known structure, I would not need timestamps or separators. 

The sample in a given step is the sum of partial samples measured by multiple processes. In other words, I have multiple instances of the same service, each aware of a slice of the total activity, and only by summing up the per-instance samples do I get the overall value. 

For example, consider 10 application servers counting requests per second. Only by aggregating them does one get the total number of requests per second. 

Ideally, I would need each instance to run a command that would do something like this (note the last argument):

INCR <key> <delta> <offset-in-key>

Perhaps a Lua alternative can be devised, but I wonder if the same performance concerns that pushed Salvatore to implement BITOP and BITCOUNT as full Redis commands are not applicable here?

An alternative would be to store samples per instance, but that would increase memory footprint 10 times.

How would you approach this?

Thanks a lot for your help.

Cheers,

Teleo

[Josiah Carlson]

I would use Lua. If you don't expect your counters for that time
period to exceed 99,999,999 then you don't even need to binary pack;
just parse/write back decimal representations.

There is also an opportunity in that situation to pass multiple
counters to be updated at once.

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/bvNdOO0vzh4J.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

[MattK]

What makes 99,999,999 the limit? Its not a binary limit - string length?

> redis-db+unsubscribe@googlegroups.com.

[Josiah Carlson]

A 64 bit field is 8 bytes. If you are using only decimal digits, that
gives you a range of 0...99,999,999 .

Regards,
- Josiah

>> > redis-db+u...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/redis-db?hl=en.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/redis-db/-/bsV9U1XfOSkJ.

>
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to

> redis-db+u...@googlegroups.com.

[Javier Guerra Giraldez]

On Mon, May 21, 2012 at 2:52 PM, Josiah Carlson
<josiah....@gmail.com> wrote:
> A 64 bit field is 8 bytes. If you are using only decimal digits, that
> gives you a range of 0...99,999,999 .

IOW, it's not a limit, just that below that, binary numbers offer no
size advantage.

--
Javier

[teleo]

What if I wanted full 64-bit numbers, either integers or floating point?

[Josiah Carlson]

Then you would need to encode/decode the pieces in Lua. Redis 2.6 does
include support for lua_struct, so you should be able to pack/unpack
these kinds of integers.

Regards,
- Josiah

On Wed, May 23, 2012 at 12:22 AM, teleo <lev....@gmail.com> wrote:
> What if I wanted full 64-bit numbers, either integers or floating point?
>

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/redis-db/-/dW75jnKQJJ8J.

[highpointe]

Here is my SS: 259 71 2451