[ann] gossipmonger - gossip protocol node for real-time peer-to-peer state distribution with pluggable storage and transport mechanisms

[Tristan Slominski]

Hello,

I'm happy to announce the initial release of Gossipmonger (https://github.com/tristanls/gossipmonger). 

Gossipmonger is an implementation of the Scuttlebutt gossip protocol for real-time peer-to-peer state distribution (and liveness monitoring). Additionally, it has pluggable storage and transport mechanisms (currently it ships with in-memory storage and TCP transport), so it is open to be implemented in the browser as well as server-side.

Cheers,

Tristan

[Kevin Swiber]

Tristan,

At a quick glance, the API looks great. I'll likely play around with this next week.

Can you provide some insight as to why dominictarr's scuttlebutt didn't suit your needs/style (if that's the case)? https://github.com/dominictarr/scuttlebutt

Thanks,

Kevin

Sent from my iPhone

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Floby]

Same as Kevin,

I'm currently using Scuttlebutt and it's working great for my use case. Can you point out the major difference or enhancements ?

[Tristan Slominski]

Hey Kevin and Floby,

Heh.. I have a hard time of thinking of a short answer to this, so please be patient.

I wanted to write an open source gossipmonger for a while now. My initial implementation of gossip was closed source and at the time dominictarr's version didn't exist. That's not an argument for primacy or anything, I just want to highlight that I've been living with this implementation approach in my head for a long time, so my intuition has been indoctrinated in the approach demonstrated in gossipmonger. I was excited to see dominictarr's work (and substack's, as well as others in the community) as they shared their vision for peer-to-peer designs. So, when I needed an implementation for my current project, naturally I took a look at scuttlebutt, but overriding any reasons I can highlight below, I already had something that I knew worked, so I didn't really have too much incentive to learn something else that also worked :D. I'm as lazy as the next human.

Having said that, I also don't know your use cases, so it's difficult to compare anything in the dark. All I can offer are some initial observations for what felt different in scuttlebutt compared to the gossipmonger implementation that I was familiar with for my particular use case.

My use case/criteria for gossip are:

1. Promoting global awareness of peer-state. That is, given peer1, peer2, and peer3, only peer1 writes peer1 state, only peer2 writes peer2 state, and only peer3 writes peer3 state. However, I want peer2 to know the state of both peer1 and peer3, and so on. 

2. Liveness determination (which is pretty much "phi accrual failure detection").

3. Asynchronous communication. Specifically, I mean that I don't want to force a request-reply pattern of interaction. Even more specifically :), I don't want to force a connection-oriented pattern of interaction. You could also call this a "push-only" requirement. Part of the reason for this is that my initial gossip implementation was implemented over a messaging service (initial prototype was actually over pubnub (http://www.pubnub.com/), although I moved away from that later, which ties into the next point).

4. Transport independence. That is, I wanted to be able to have an API that I could implement using messages in a bottle if I wanted to.

So...

"Scuttlebutt is always duplex. Scuttlebutt does a handshake..." (https://github.com/dominictarr/scuttlebutt#gotchas). That doesn't meet my criteria #3. This also seems to me that it won't work over UDP. UDP implementation is hinted at in Introduction section of "Efficient Reconciliation and Flow Control for Anti-Entropy Protocols" (http://www.cs.cornell.edu/home/rvr/papers/flowgossip.pdf): "Gossip protocols are designed to be non-invasive and have predictable performance, and for this a designer has to fix not only the gossip rate per participant but also the maximum size of gossip messages (e.g., maximum UDP packet size)."

Another factor...

"i.each(self.history(sources), function (data) {d._data(data)})" (https://github.com/dominictarr/scuttlebutt/blob/8217ec7f96091838be3b56122d16176ba2b63fa6/index.js#L150). This indicates to me that the entire history since the last timestamp is sent over the wire (I may be wrong in this, but it is the impression that I got). Taking look at section 3.2, Scuttlebutt Reconciliation, of "Efficient Reconciliation and Flow Control

for Anti-Entropy Protocols" (http://www.cs.cornell.edu/home/rvr/papers/flowgossip.pdf) the authors would call this "precise reconciliation", which can result in sending updates that are unnecessary between the peers, "scuttlebutt reconciliation", on the other hand, converges more slowly, but "leaves room in the gossip message for deltas from other participants." Another consideration in that section is how to pick which deltas are sent in a gossip message. The authors outline "scuttle-breadth" and "scuttle-depth" approaches and say that "scuttle-depth" seemed to work better. I haven't been able to figure out where in the source code of scuttlebutt these things come into play (gossipmonger uses the "scuttle-depth" approach). To be fair, there may be subtleties in the implementation that I'm not seeing, but when in doubt I already had my implementation that met my criteria :). You could say I time-boxed my analysis of the source code.

To summarize, for my four criteria, #2 was missing (or external), #3 seemed to not be satisfied, and #4 seemed to not be satisfied based on #3 not being satisfied. 3 out of 4 were missing, which was enough for me to open source an implementation I already had on hand.

I hope this helps, and if my analysis was incorrect, then great, I'd love to have someone walk me through how my use case could be implemented in scuttlebutt. Scuttlebutt is great, and it had much more eyeballs for much longer on it. To be fair, I didn't ask dominictarr about any of this, but this goes again back to the fact that I already had an implementation, so it was easier/faster to release it. 

Cheers,

Tristan

[Floby]

Thanks for taking the time to write a detailed reply. It's much appreciated.

My point of view is that I'm not aware of all the details of the science under the hood and I'm pretty happy this way for now. Dominic's implementation provides a very small API surface which is in turn very simple to learn (instanciate it, pipe it, use it). Extending his implemntation is also much simpler and requires less knowledge of the implementation details (only 2 methods to override). I haven't taken a closer look at your module yet but it already seems to me that I have to learn a whole lot of new concepts that I don't really need to know for a basic use.

My primary use case at the moment is synchronizing client side models (in a browser) accross multiple users. I mostly had to combine scuttlebutt, shoe and mux-demux. As you see, I'm only dealing with connected transports (websockets, http) exposed as streams. The stream approach to scuttlebutt makes it really easy to compose with other modules in my opinion.

I'll make sure to give a try to your implementation for another project of mine =)

[Tristan Slominski]

Since I've been asked to make the comparison a few times, it's become clearer what I think the difference between scuttlebutt and gossipmonger is. It actually starts to feel like comparing apples to oranges. 

scuttlebutt: "peer-to-peer replicatable data structure", two (maybe more?) scuttlebutts connect, negotiate the differences, and synchronize

gossipmonger: endpoint that uses connection-less gossip protocol to find other gossipmongers, track whether they are live or not, and use "scuttlebutt reconciliation" strategy to disseminate information to other gossipmongers. 

Perhaps the above comparison is shorter and more useful :)

There are other projects I'm working on that use gossipmonger as a component. When those become easier to consume, they might better illustrate the reason for gossipmonger's existence.

Cheers!

[Kevin Swiber]

Thanks, Tristan. This is helpful. I'll find time to give gossipmonger a test drive this week.  Looks great.

Sent from my iPhone

--

[Tristan Slominski]

Great :)

It's not easy to consume, but if you're feeling brave, here's one use of gossipmonger by a coordinator module that decides which machines to replicate data writes to (it's sort of like dynamo but without consistent hashing for this particular use case): https://github.com/tristanls/alldata/blob/2580caa36c7ec11fe1b28a2e57485dd5dc7fec61/scripts/alldata-http-http-leveldb.js#L172

Whenever gossipmonger detects a new peer or a dead peer it notifies the coordinator so that it isn't trying to replicate to dead nodes. Only "transport" and "location" keys are ever gossiped because that's all the information dissemination necessary for a coordinator (in this use case). 

[Floby]

I just realised now you also wrote `discover` which totally fits in the picture you're describing.

For me the most notable difference in use is that gossipmonger is not connected whereas scuttlebutt is. That's why scuttlebutt can use the stream interface and gossipmonger cannot.

[Tristan Slominski]

Yes, that is the big difference in the data transfer portion.

The connection allows scuttlebutt to guarantee in order delivery, which makes for a simpler implementation if you're transferring lots of data. TCP acks will acknowledge every update in the sequence, so many can be sent "at once" and in order. Whereas with UDP style connection-less gossip, only successive gossip messages serve as acks, which I assume tend to be more spread out in time when compared to packets in a TCP connection (gossip interval vs. interval between TCP acks).

This seems to be the reason why Cassandra went from UDP gossip to TCP, according to this thread: http://mail-archives.apache.org/mod_mbox/cassandra-user/201105.mbox/%3CBANLkTim+XmO3zsh...@mail.gmail.com%3E

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/UH7iCUy9eJU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.