[ANN] Voldemort Sets 0.0.1

[Coda Hale]

Hey all,

I finally uploaded my project, voldemort-sets, to GitHub:

<http://github.com/codahale/voldemort-sets>

It's a set of view transformation classes which allow you to maintain
sets of numbers in Voldemort:

client.get("ages"); // null
client.put("ages", "ADD 34, 62, 30");
client.get("ages"); // "34,62,30"
client.put("ages", "REM 62");
client.get("ages"); // "34,30"

It supports 64-bit integers (stored using Avro's variable-length
zig-zag encoding) and 64-bit floating point numbers (stored as 8-byte
blocks), stored in unordered, sorted, and reverse sorted flavors.

Right now it's got great test coverage but zero feedback from
production environments, so look before you leap if you want to use it
with real data. Suggestions are always welcome; suggestions in the
form of patches are the easiest to take. ;)

Enjoy!

--
Coda Hale
http://codahale.com

[OG]

Interesting. Thanks for sharing it. What kind of application are you
using this in?

Otis

[Jay Kreps]

This is very cool. I would like to think about how this could be
generalized a bit. I assume this is specific to integers, but I think
it could be serialization agnostic (as long as the serialization
format supports a list type).

Something like

client.get(ages)
client.put("ages", ImmutableList.of(1,2,3,4))
client.put("ages", new Remove(4)) // [1,2,3]
client.put("ages", new Append(1,3,4))

The real problem is that it does not support a generic sublist
operation since get does not include additional parameters to pass in
a range. This is a problem I would like to figure out how to fix.

-Jay

> --
>
> You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> To post to this group, send email to project-...@googlegroups.com.
> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
>
>
>

[Bruce Ritchie]

> The real problem is that it does not support a generic sublist
> operation since get does not include additional parameters to pass in
> a range. This is a problem I would like to figure out how to fix.

Can I assume that by sublist you really mean a filter operation ? i.e.
not just index based but rather some like a VoldemortFilter (applied
on the server perferrably) that would restrict the returned results.
I've actually been thinking about that in a general sense outside of
this interesting set api to be used to get a list of keys/values. A
generalized solution that would cover both keys and values would be a
nice addition..

[Jay Kreps]

I am less concerned with a generalized filter (which probably requires
custom code as a view), then with a sublist operation to do pagination
(e.g. give me the first 50 values only).

-Jay

[Coda Hale]

On Mon, Dec 21, 2009 at 11:19 AM, Jay Kreps <jay....@gmail.com> wrote:
> I am less concerned with a generalized filter (which probably requires
> custom code as a view), then with a sublist operation to do pagination
> (e.g. give me the first 50 values only).

I've actually been thinking about the same thing.

One way of doing so would require a custom RoutingStrategy which would
hash on just the key, not the options (e.g., "lists:51827?count=30"
would get routed to the nodes responsible for "lists:51827"). After
that it's just a matter of creating a view or storage engine capable
of parsing those options and acting.

Right now the RoutingStrategy classes you can configure are fixed, but
loosening that up seems like a pretty straight-forward change.

[Coda Hale]

On Sun, Dec 20, 2009 at 7:58 PM, OG <otis.gos...@gmail.com> wrote:
> Interesting.  Thanks for sharing it.  What kind of application are you
> using this in?

This isn't extracted from anything; it just seemed like an interesting problem.