[erlang-questions] Can we ask this question to a distributed hash ?

[Eric Ho]

"Is item X in our distributed hash ?" where the distributed hash exists as a gigantic hash in the RAMs of a bunch of machines ? If Erlang can answer this, how can I write Erlang programs to answer this question ? How can I load this distributed hash with my contents ?  Obviously, each machine can only load a piece of the total contents... Thanks. -eric

[Jim McCoy]

2008/8/9 Eric Ho <drwho1020...@yahoo.com>:

> "Is item X in our distributed hash ?"
> where the distributed hash exists as a gigantic hash in the RAMs of a bunch
> of machines ?

Do you want to answer the question with an absolute yes or no, or is a
probabalistic answer good enough?

If the latter, then you need to do some research into "bloom filters".
If your dht supports deletion then you will probably want a counted
bloom filter, and if the dht is very large or network bandwidth is
constrained you will want to examine compressed bloom filters as well.

Jim
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions

[atomly]

[Jim McCoy <jim....@gmail.com>]

> Do you want to answer the question with an absolute yes or no, or is a
> probabalistic answer good enough?
>
> If the latter, then you need to do some research into "bloom filters".
> If your dht supports deletion then you will probably want a counted
> bloom filter, and if the dht is very large or network bandwidth is
> constrained you will want to examine compressed bloom filters as well.

And if you're looking for an implementation of a Bloom Filter in Erlang,
you can check out this one:

http://code.google.com/p/bloomerl/

--
:: atomly ::

[ ato...@atomly.com : www.atomly.com ...
[ atomiq records : new york city : +1.917.442.9450 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...

[Eric Ho]

I'm sorry... Actually, I meant not a dht, but more like a general distributed memory -- i.e. there may be duplicates in it and multiple actors (on different machines) can insert stuff into this memory.  This distributed memory can be configured to be either RAM-only or RAM-and-disk, across all machines. Thanks. -eric --- On Mon, 8/11/08, Jim McCoy <jim....@gmail.com> wrote:

[atomly]

Well, in that case, I think you'll want to look into Merkle trees:

http://en.wikipedia.org/wiki/Merkle_tree

I believe there is also an Erlang version somewhere on Google Code.

[Eric Ho <drwho1020...@yahoo.com>]

> _______________________________________________

> erlang-questions mailing list
> erlang-q...@erlang.org
> http://www.erlang.org/mailman/listinfo/erlang-questions

--
:: atomly ::

[ ato...@atomly.com : www.atomly.com ...
[ atomiq records : new york city : +1.917.442.9450 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...

[Justin Sheehy]

On 8/11/08 8:44 PM, "atomly" <atoml...@atomly.com> wrote:

> Merkle trees:
>
> http://en.wikipedia.org/wiki/Merkle_tree
>
> I believe there is also an Erlang version somewhere on Google Code.

There is indeed:

http://code.google.com/p/distributerl/

(and a minor improvement/bug-fix release will be going up this week)

-Justin

[Eric Ho]

I'm wondering if there is something equivalent to HyperTable in Erlang.

[atomly]

[Eric Ho <drwho1020...@yahoo.com>]
> I'm wondering if there is something equivalent to [1]HyperTable in
> Erlang.

Well, what features of HyperTable are you looking for, exactly? There
are a lot of similar projects and I imagine you could even integrate
with HyperTable from Erlang.

Try checking out Kai/Dynomite (Amazon Dynamo clones in Erlang), CouchDB,
bigdata or even Mnesia.

--
:: atomly ::

[ ato...@atomly.com : www.atomly.com ...
[ atomiq records : new york city : +1.917.442.9450 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...