Message passing architecture - why re-invent the wheel?

[Ben Langfeld]

I wonder if the Diaspora team have a response to the concerns raised
here:
http://news.ycombinator.com/item?id=1697266

I shall follow with a blog post soon with more details.

[Antoine Lyset]

Link's dead ?

[Johan Brinch]

On Fri, Sep 17, 2010 at 11:40, Antoine Lyset <antoin...@gmail.com> wrote:
> Link's dead ?

It's working here.

--
Johan Brinch,
Dept. of Computer Science,
University of Copenhagen

[shadowfirebird]

Working here too.

Unless I have misunderstood -- which is entirely possible -- they are
using an existing standard: OStatus. http://ostatus.org/about

On Sep 17, 11:06 am, Johan Brinch <zer...@diku.dk> wrote:

[Ori Pekelman]

Really unclear. They clearly are inspired by it, I can see a lot of
common stuff, but I have yet to find any documentation on this;
Furthermore oStatus seems only to support a subset of the
functionality of Diaspora. So I would truly love a clarification on
this. If they are in effect following the oStatus proposition I would
like to know what are their choices on this; oStatus is very new and
is broad enough to support many different types of implementations.

I further believe there are some issues with the oStatus design for
use cases other then "Distributed Twitter". There seems to be a lot of
GUIds going around, a lot of personal information spill.

[Hedgehog]

Hi,
I notice Ben Langfeld mentions Erlang as one (the?) 'obvious option'
to try and resolve horizontal scaling issues.
Some may argue zeromq (www.zeromq.org) is worth considering, since it
is a message passing solution that has _many_ language bindings
(including Erlang).
Anyway, the zeromq devs do seem to acknowledge Erlang, but try make
messaging done: Right. Once. Easier.
:)

If you do draw up a blog addressing messaging issues it would be
interesting to expose any ZeroMQ shortcomings that make it
inappropriate to use in Diaspora?

HTH?

[Hedgehog]

On Sep 21, 7:54 am, Hedgehog <hedgehogshia...@gmail.com> wrote:
> On Sep 17, 6:55 pm, Ben Langfeld <b...@langfeld.co.uk> wrote:
>
> > I wonder if the Diaspora team have a response to the concerns raised
> > here:http://news.ycombinator.com/item?id=1697266
>
> > I shall follow with a blog post soon with more details.
>

In case anyone wants to track this issue/quasi-feature-request I've
opened this issue:
http://github.com/diaspora/diaspora/issues/issue/287
hopefully it'll be tagged as a feature-request.

Since this area is full of wet paint, it'd be handy to point to blog
posts, etc. addressing relevant issues.

Best wishes.

[Martin Ammermüller]

Am 20.09.2010 23:54, schrieb Hedgehog:
> If you do draw up a blog addressing messaging issues it would be
> interesting to expose any ZeroMQ shortcomings that make it
> inappropriate to use in Diaspora?
>

Just from glancing over their FAQ: no security and rate limiting maxes
out CPU. The latter would become important if it's really a goal to
enable people running diaspora pods in their homes. Most people have an
ansynchronous connection with much smaller bandwidth upstream, so rate
limiting (think bittorrent for example) would become a handy feature. I
also doubt that there would be much dedictated machines (which would
make using max CPU less of an issue) for diaspora and more home servers,
that will most likely have other tasks assigned to them, in use.

Just my 0.02�
Martin

[Raphael Sofaer]

Hello,

While I appreciate the desire for discussion and improvement, I'm trying to keep the github tracker more specific than things like a general dissatisfaction with our message passing technology.  I think that email is a better place to discuss anyway, since the text isn't restricted to github's small column.

Anyhow, I sort of don't understand this thread.  We haven't made a 'choice of message passing technology' in any meaningful sense, we could rip out that segment of the codebase and replace it pretty rapidly.  All we are doing is using EventMachine's non-blocking http request class to queue up and send out POSTs over HTTP.  If we reached a point where XMPP was helpful, we could start using it.  I'd rather not write an XMPP schema or similar doc until we are more sure of what we need.

ZeroMQ seems like a cool technology, but we aren't having scaling problems or anything like that right now.  If we do need more speed, we could certainly look at it.

We actually implemented OStatus for public posts a while back, but took it out of Diaspora until we had a real place in the UI for it.  That will be back in fairly soon.

Thanks for being part of this design process,

Raphael

[shadowfirebird]

We're pretty much all saying that we're worried that you haven't made
that choice.

We can't do anything significant to help you until you do. And we
really, really want to help.

The message passing bit is the fundamental underpinning of the whole
concept. Even if you are doing agile development, this is something
that you need to design right at stage one.

On Sep 22, 6:21 am, Raphael Sofaer <raph...@joindiaspora.com> wrote:
> Hello,
>
> While I appreciate the desire for discussion and improvement, I'm trying to
> keep the github tracker more specific than things like a general
> dissatisfaction with our message passing technology.  I think that email is
> a better place to discuss anyway, since the text isn't restricted to
> github's small column.
>
> Anyhow, I sort of don't understand this thread.  We haven't made a 'choice
> of message passing technology' in any meaningful sense, we could rip out
> that segment of the codebase and replace it pretty rapidly.  All we are
> doing is using EventMachine's non-blocking http request class to queue up
> and send out POSTs over HTTP.  If we reached a point where XMPP was helpful,
> we could start using it.  I'd rather not write an XMPP schema or similar doc
> until we are more sure of what we need.
>
> ZeroMQ seems like a cool technology, but we aren't having scaling problems
> or anything like that right now.  If we do need more speed, we could
> certainly look at it.
>
> We actually implemented OStatus for public posts a while back, but took it
> out of Diaspora until we had a real place in the UI for it.  That will be
> back in fairly soon.
>
> Thanks for being part of this design process,
> Raphael
>

[Simon B.]

Shadow, it seems they do not want to "make a decision" and stick with
it, but rather change the tool (hammer for scredriver) if the problem
changes from a nail to a screw. By keeping message passing/protocol
defined by and *only by* the latest version of the source code at
github, we even help the project avoid getting split up. Also, "alpha"
means everything *will* change before the final release, so no point
in asking for "decisions" yet... :-)

If what you're really after is to get enough info to make a third
party GUI (iPhone/Android app?) then I suggest you get your (or your
subordinate programmer's) hands dirty with the source code and build a
public API that can be stable enought for you to program your app
towards?

[shadowfirebird]

I'm not after a decision and I don't have an ulterior motive. I'm
just after some evidence that this product will actually be designed,
rather than hacked together any old way -- because I want it to be
good.

And I'm certainly not the only person on this list to say that we need
a basic design for how diaspora instances communicate with each other
before we do much more coding.

I get that the team are into Agile development, and I'm okay with
that. But even in agile development you have to have some idea of how
the product will fulfil the goals of the project.

What if it turns out, for example, that the current webserver-based
implementation is painting the product into a horrible corner? If we
knew how instances communicated, we would whether that was true now,
before too much time has been wasted writing code that would have to
be scrapped...

[wenjie han]

think everyone has some vested interest incl. the core team
i too look to have some synergy with some web apps I intend to build
like many,,,we can just go with the incumbent Facebook

this project gathers a lot of funds and attention by waving the Open Source banner - fine by me
it only really means that the application will eventually release under an OSS licence
no OSS project (nor OSS defines) has to please everyone or be designed in commons

i think few successful projects can trace back to some point
an elaborate detailed plan was presented, and followed to the T, all concepts proven with scientific precision, mass consensus reached, nothing repeated and without huge design errors/mistakes

BUT i think how the community is engaged (and it will be evident in a few weeks)
will determine who/how many will
- stick around to contribute pre-alpha etc...
- stand around to leach benefits, e.g. host provider for seeds...
- wait to come back when it's beta
- see if it will even be well-received  i.e. when the water's warm

---

From: shadowfirebird <shadowf...@gmail.com>
To: diaspora-dev <diaspo...@googlegroups.com>
Sent: Thu, September 23, 2010 9:10:41 PM
Subject: Re: Message passing architecture - why re-invent the wheel?

[PeterH]

Seems to me that any viable distributed social network will need
facilities defined in:
- message passing.
- authentication, primarily a seed or that seeds user authentication
their id to another seed.
- a means of querying another seed, likely built on top of the above.

I'm disappointed that the above protocols aren't worked out and
DOCUMENTED yet for diaspora. Everything else needed for a social
network builds off of these.

First, make a framework that works. Then make a pretty wrapper.

[shadowfirebird]

I would settle for "worked out". If they're into Agile, then they may
decide that the best way to document them is to write the code. I've
not got a problem with that.

I've not done a lot of Agile, nor have I done any Open Source
development. But what I have done is twenty years of consultancy and
coding for other people. I've seen development done more different
ways than I can count. I've seen development fail more times than I
can count.

And the one common aspect to most of the fails was this: no-one
bothered to work out what the product was for, or how it was going to
work, before it started.

Of course, this one project could be the exception...

[Simon B.]

> And the one common aspect to most of the fails was this: no-one
> bothered to work out what the product was for, or how it was going to
> work, before it started.

Actually I've seen one PDF with quite good explanations, as well as
some videos. Do you all recognize this, or should I help to find the
links to those? I re-read the PDF just yesterday and thought I noticed
that quite some of the ideas from there had already been changed in
how the need should be implemented. The most current documents
"Roadmap" and Tracker at Pivotal are *very* recommended reading. Also
while I agree that the "Diaspora four" or "NYU four" should try to
communicate more, it could also be easily argued that they better
focus on code contributions to get the project forward, and we
bystanders should really just get some bloggeer into writing a good
journalistic analysis of what's going on and talk about the current
plans forward. Good resources are the IRC chat logs (I find something
interesting there from each of the days) and probably it's possible to
interview the guys on IRC as well.
On the other hand, some ppl donated money in order to get priority
access to information and to get to influence the project. (So pay up
or shut up, at least a little, on demands? "Tagga ner" som det sägs på
svenska)

Still; it seems to me that the functionality has some effect on how
message passing can be done. It also seems that an API to access your
own seed is more important than setting the communications protocol in
stone. Even the database is still in flux and those already running
"publish" sites are running the risk of having to delete everything
and start over before the first beta is released.

[shadowfirebird]

Not aware of any PDF, but I would certainly like to read it.

Found the Mozilla talk to be pretty much devoid of any strong ideas as
to how the software would work. Likewise the tracker, etc. Not
trying to insult the team - they've now said, on this list, that they
don't have a finished design. And I applaud them for saying so.

> On the other hand, some ppl donated money in order to get priority
> access to information and to get to influence the project. (So pay up
> or shut up, at least a little, on demands? "Tagga ner" som det sägs på
> svenska)

Well, I for one am not making any demands! Just expressing my
opinion. Presumably they want us to do that, or this list would not
be here...

> Still; it seems to me that the functionality has some effect on how
> message passing can be done. It also seems that an API to access your
> own seed is more important than setting the communications protocol in
> stone. Even the database is still in flux and those already running
> "publish" sites are running the risk of having to delete everything
> and start over before the first beta is released.

I'm not suggesting that anything should be set in stone; design should
never be set in stone. But they should have enough design to guide the
development process.

For example, we should be able to answer the following, at least
vaguely:

* How do seeds communicate with each other, and users to seeds? Are we
using end-to-end encryption, or are we trusting the integrity of the
software to preserve the data?

* Are we worried about seeds that are run with hacked code to
intercept data? What, broadly, do we want to do about it?

* How do users know that they are actually talking to the person they
think they are? Are we using a web of trust to authenticate users, or
something else?

[Simon B.]

> * How do seeds communicate with each other, and users to seeds? Are we
> using end-to-end encryption, or are we trusting the integrity of the
> software to preserve the data?

Just what this thread is (was) about :)

> * How do users know that they are actually talking to the person they
> think they are?  Are we using a web of trust to authenticate users, or
> something else?

By using existing, "tried and tested", infrastructure based on public
key cryptography and keyservers.
(It would be great to make a wiki-page to explain this -- any takers
on gathering wikipedia-links etc and writing a first draft?)

> * Are we worried about seeds that are run with hacked code to
> intercept data? What, broadly, do we want to do about it?

Public key cryptography. Read up on it, it's more interesting than
sliced bread :)

The PDF I want to recommend: http://www.joindiaspora.com/images/Diaspora_ISOC_Presentation.pdf

[shadowfirebird]

I agree with all your answers, by the way. But it's the team that
have to make the call on these things. They've started to - Raphael
ahs just posted that they are against end-to-end encryption for the
moment. So, perhaps we are inspiring some designing...

[Ben Langfeld]

To be clear, I didn't propose Erlang as a solution. Erlang is just a language, but it's more or less the best there is for async message passing, and is certainly going to scale better than EventMachine in a Thin.

The real meat of the issue lies in the protocol, not the implementation. The implementation is actually irrelevant.

XMPP is a protocol solution that has already had a lot of this work done in writing a specification. There are some things missing from the XEPs that Diaspora would need. But it's much easier to add than it is to start from scratch.

I havn't had a chance to write a blog post about this yet, but there's a huge amount of material out there already. OneSocialWeb have already done a lot of work on this. These are the things that are going to need to be done before Diaspora stands a chance of working as a project. You can't just swap and change the core protocol of a federated system in the project's github repo.

This stuff has to be planned and defined as a structure that evolves in a controlled way. A 4 month hackathon is not the right way to do that. You have $200k. Offer someone who worked on Wave a chunk for some advice and do this properly.

Regards,
Ben Langfeld

[shadowfirebird]

Agree totally. Protocol more important than implementation; design
before you code.

On Sep 27, 11:06 am, Ben Langfeld <b...@langfeld.co.uk> wrote:
> To be clear, I didn't propose Erlang as a solution. Erlang is just a
> language, but it's more or less the best there is for async message passing,
> and is certainly going to scale better than EventMachine in a Thin.
>
> The real meat of the issue lies in the protocol, not the implementation. The
> implementation is actually irrelevant.
>
> XMPP is a protocol solution that has already had a lot of this work done in
> writing a specification. There are some things missing from the XEPs that
> Diaspora would need. But it's much easier to add than it is to start from
> scratch.
>
> I havn't had a chance to write a blog post about this yet, but there's a
> huge amount of material out there already. OneSocialWeb have already done a
> lot of work on this. These are the things that are going to need to be done
> before Diaspora stands a chance of working as a project. You can't just swap
> and change the core protocol of a federated system in the project's github
> repo.
>
> This stuff has to be planned and defined as a structure that evolves in a
> controlled way. A 4 month hackathon is not the right way to do that. You
> have $200k. Offer someone who worked on Wave a chunk for some advice and do
> this properly.
>
> Regards,
> Ben Langfeld
>
> On Sat, Sep 25, 2010 at 11:28 PM, shadowfirebird

> <shadowfireb...@gmail.com>wrote:

[Dan White]

On 27/09/10�11:06�+0100, Ben Langfeld wrote:
>To be clear, I didn't propose Erlang as a solution. Erlang is just a
>language, but it's more or less the best there is for async message passing,
>and is certainly going to scale better than EventMachine in a Thin.
>
>The real meat of the issue lies in the protocol, not the implementation. The
>implementation is actually irrelevant.
>
>XMPP is a protocol solution that has already had a lot of this work done in
>writing a specification. There are some things missing from the XEPs that
>Diaspora would need. But it's much easier to add than it is to start from
>scratch.
>
>I havn't had a chance to write a blog post about this yet, but there's a
>huge amount of material out there already. OneSocialWeb have already done a
>lot of work on this. These are the things that are going to need to be done
>before Diaspora stands a chance of working as a project. You can't just swap
>and change the core protocol of a federated system in the project's github
>repo.
>
>This stuff has to be planned and defined as a structure that evolves in a
>controlled way. A 4 month hackathon is not the right way to do that. You
>have $200k. Offer someone who worked on Wave a chunk for some advice and do
>this properly.

The Wave federation protocol has some interesting ideas, and seems to solve
many of the same issues Diaspora is struggling with. However, Wave seems to
be based on a server to server model. I think it'd be a good starting point
for a Diaspora protocol.

--
Dan White

[Ben Langfeld]

Since the seed runs on a server somewhere, it doesn't strike me as a problem that the protocol is s2s, but rather the only sensible correct way to work.

Regards,
Ben Langfeld