A proposal to integrate new components

[Arndt Schnabel]

tl;dr: Flowhub + IPFS + Ethereum = Decentralized Microservice Awesomeness

It may take a while to understand the specifications and implications of these systems. All of them are open source, community oriented and promote decentralisation.

First, there is IPFS (ipfs.io). It has two components:

1. libp2p (https://github.com/ipfs/specs/tree/master/libp2p)

Libp2p provides a peer to peer identity system with a public key infrastructure (your node address is the hash of your public key) and protocol agnostic communication between nodes.

2. IPFS (https://github.com/ipfs/ipfs)

IPFS could be described as a combination of Git, Bittorrent, Kademlia and self certified file systems. It is build on top of libp2p and allows you to distribute and receive content addressed files over a peer to peer network.
In contrast to the current client-server model of the internet, where files are addressed by their location, here they are addressed by their hash. The network is queried for that hash and returns which nodes host it. With this information, you can stream your file from dozens of nodes at once. There is no "offline" anymore, only various degrees of connectivity.
While IPFS is built on immutable datastructures, it also provides a mechanism to publish mutable references to immutable files under your node address, thereby authenticating their origin.

There will be another component that incentivises reliable, untrusted, distributed storage, called Filecoin, but this does not exist yet.

Both components are still in development, but very thought out. Implementations in multiple languages (Go, Javascript, Python) exist.

There exist more libraries that provide the same functionality as libp2p, but they are not protocol agnostic. The multihash, multiprotocol and multicodec definitions make it extremely flexible.

3. Ethereum (https://ethereum.org/)

The third component addresses service registration, discovery, reputation and payment. While one can use IPFS to make data publicly accessible to the entire world at virtually no cost, most services will require to be paid.
Real world services, hardware services and software services - basically everything that a digital ontology can offer (e.g. https://www.dvs.tu-darmstadt.de/staff/aherzog/a3me/Classification_A3ME.png) could be accessed.

Some of you seem to be familiar with cryptocurrencies. You might have heard about payment channels or the "Lightning network" proposal. While the LN for Bitcoin is still a work in progress, payment channels on the Ethereum blockchain already work: https://github.com/obscuren/whisper-payment-channel. This schema allows one to make arbitrary micropayments and smart contract interactions (verifiable computation) off-chain. This could also include decentralised service registration and -discovery, removing the necessity of lookups in a centralised database, as Flowhub currently provides.

Ethereum is running in this very second: https://stats.ethdev.com/ and continues to become more flexible as well. From speaking with the respective developers I know that both projects might merge or at least share a significant codebase.
I'm not a maximalist for any of the mentioned systems (probably level 2 in the diagram, see the attachment), but think that they are currently the most elegant and efficient way to implement this.

4. Flowhub

Now, where does Flowhub enter the equation, why is it needed, when everything could be done locally? For two reasons, of which the first already exists:

1. It facilitates easy program creation. Even non-programmers can use it to interact with services.
2. It provides a trusted execution environment. The user does not have to worry about hosting reliability and server management.

The same use cases that exist apply, but now especially service interaction and task automatisation. Flowhub would be more flexible than, say, ifttt.com and zapier.com. This requires developing a service interface descriptor language, the implementation of specialised nodes (libp2p-connector, ethereum-payment, -contract interaction) and the creation of a service graph monitor.

As soon as we go meta and make the graph a service and couple it with a graphical user interface, anyone could create their own services.

5. Required components

noflo-ipfs - Interfacing with the default IPFS node ("ipfs")
https://github.com/ipfs/js-ipfs-api

noflo-web3 - Interfacing with the default Ethereum node ("geth")
https://github.com/ethereum/web3.js/tree/master

I started implementing noflo-ipfs, but am currently stuck, because adding files to the network requires giving a file path to the node - which would probably only work in a node environment.

6. Examples

For now, I'm assuming that the graphs are not used to implement the services themselves. Still, the following would be possible:
ifttt.com - style triggered interactions: "If the temperature sensor with the identity I published a value less then t, send me an SMS."
rundexter.com - style data flow processing: "Pipe my webcam stream to the NeuralTalk video descriptor service at identity I and pipe that to a local node doing voice synthesis"
automatic hosting: "If there is a new file in this folder, add it to IPFS, publish it under a name, contact a hosting service and schedule micropayments over a payment channel."

7. Enhancing components

Other possible improvements could be done in regards to the user interface.
Threenodes.js is great because it labels all inports and outports, as well as displays the current state of the node, if possible. The node state can even be modified (e.g. in a timer/function/color node).
http://idflood.github.io/ThreeNodes.js
Here is an example:
http://idflood.github.io/ThreeNodes.js/index_optimized.html#example/rotating_cube1.json
It reminds me of the Blender compositor, which has even more advanced functionality (node grouping etc.) - though that is already possible with FBP.


This is a moonshot and certainly crazy. But it might be worth it.

Please let me know what you think.

- Arndt

P.S. I'm 10+ years younger than I may sound, so please forgive my excitement and naivety

[Arndt Schnabel]

I forgot include any examples for what Ethereum integration would enable:

• Token Systems
• Financial derivatives and Stable-Value Currencies
• Identity and Reputation Systems
• Decentralized File Storage
• Decentralized Autonomous Organizations
• Savings wallets
• A decentralized data feed
• Smart multisignature escrow
• Prediction markets
• On-chain decentralized marketplaces

It's currently a very active space, a list of projects can be found here: http://dapps.ethercasts.com/

[Francisco León]

I think that the FBP model was made for this, it fits perfectly in the p2p descentralized world, where programs communicate via ad-hoc channels and they coordinate their actions from events  via message passing. By the way, I'm not 100% satisfied with NoFlo, behind scenes it emulates concurrency in Node which is single threaded and it relies on a callback approach for events communication. so I'm looking for a native implementation of the FBP in c++ that implements real multithreading concurrency, supports coroutines and being able to saving and restoring the program state allowing to execute version distributed versions of the program deterministically :  that could be the Holy Grail of Distributing Systems, a programming model made for a descentralized and concurrent world.

[Paul Morrison]

I am also leaning to a C[++] base, which I believe could support components written in multiple languages.  So far, I might suggest three possible (not mutually exclusive) directions to base further development on:

- CppFBP, a C++ implementation which uses Boost to handle its multithreading.  It has two modes: an interpretive mode, which scans off a network defined using the Stevens' notation; and a fixed format notation, which is compiled.  I have also built a Lua interface, so some components can actually be written using Lua - perhaps for more ad hoc requirements - https://github.com/jpaulm/cppfbp .  IIUC Boost knows what environment it is running in, and will use the appropriate multithreading services.

- Tom Young's DFD - see https://groups.google.com/forum/#!searchin/flow-based-programming/dfd/flow-based-programming/bzH1zb9WFdI/ms_MuX6oCwAJ and elsewhere

- Folding some ideas from *nix back into the mix - see several recent topics in The Google group

Has anyone else done work along these lines?

Regards,

Paul

On Sun, Jan 10, 2016 at 8:31 PM, Francisco León <super...@gmail.com> wrote:

I think that the FBP model was made for this, it fits perfectly in the p2p decentralized world, where programs communicate via ad-hoc channels and they coordinate their actions from events  via message passing. By the way, I'm not 100% satisfied with NoFlo, behind scenes it emulates concurrency in Node which is single threaded and it relies on a callback approach for events communication. so I'm looking for a native implementation of the FBP in c++ that implements real multithreading concurrency, supports coroutines and being able to saving and restoring the program state allowing to execute version distributed versions of the program deterministically :  that could be the Holy Grail of Distributing Systems, a programming model made for a decentralized and concurrent world.

[Arndt Schnabel]

As long as all implementations use the same protocol, we could apply optimisations locally when needed. WebAssembly could become an interesting part of this as well. I'm not sure if the FBP graph definition is the best imaginable interface for services, but it is the most versatile and powerful I found to date. And it fits perfectly with IPFS and Ethereum. For now, I will try to implement the APIs on top of FBP, then I will try to enhance service discovery, component sharing more on the lower level. Payments should probably stay on top, because there are so many possibilities to pay for services (per call, per byte, per byte*second, subscription based, token based etc.). This will probably need some kind of new 'multi-payment' protocol prefixes and definitions. This way, you could use a service with
<identity>/fbp/<graphid>/<componentid>/<port>

Where every component can publish a service description as an IPFS name (assuming every component has an identity, this way, they could be easily distributed over several nodes with the flomsg protocol). I also have to think about differences in Daemons/Services and how that affects the addressing. This is awesome.

NoFlo isn't perfect, although the work done so far is very impressive. Especially the UI needs a makeover: components should be able to describe a GUI in HTML (as in nodered), they should display their current state and the component and port names should be more visible. If anyone here is familiar with the Blender compositor, that would be something to aspire to: http://i.imgur.com/LBSEWh4.png

I hope the flowhub guys see that, when I first sketched out what a website should be like (subscription model, git integration) I came up with something very similar to flowhub. Integrating a trusted execution environment will be a challenge, but if done correctly a gold mine as well.

[Arndt Schnabel]

Have a look at http://urbit.org/. It's a challenge to understand, but a possible future direction.

[Paul Morrison]

This is very thought-provoking!  It occurred to me though that, as we move towards reusable components, we may run up against naming problems - we already have done so in a small way in the discussion about Collate (in https://groups.google.com/forum/#!topic/flow-based-programming/bzH1zb9WFdI ).  Basically, the universe of names is a hierarchy, so it's OK if the components involved are in different folders - if they are in the same folder, it may be a race to see who grabs a name first! 

Another issue is, if you have a big hierarchy, how do you walk the hierarchy to find the component you want?

On the other side, Wayne Stevens addressed this problem, and pointed out that in the real world this is not a problem - if you want to connect two materials together, everyone knows a few commonly used techniques; if they are not adequate, you ask the person in the hardware store; failing that, you go on the Internet!  I quote Wayne on p. 48 of the 2nd ed.: "The point is that effective reuse doesn’t require catalogues, although they can certainly help."

[Arndt Schnabel]

This touches two of the most important problems of decentralised systems: identity and trust. It is also exactly what Zooko's triangle predicts: https://en.wikipedia.org/wiki/Zooko's_triangle. In a decentralised system, names can be either human-meaningful or secure and unique. Some systems try achieve a tradeoff between these parameters, Urbit for example has a hierarchical system of names that expands when more users are added. In IPFS, on the other hand, addresses (names) are the hash of the public key of the keypair that represents an identity. The identity could represent anything: a computer, a process, a file, a database or even a reference to published data. Currently, IPFS automatically resolves DNS names to identities, such that users can insert a TXT record into DNS that references their identity. Another decentralised possibility is using smart contracts - in fact, an extremly simplified version of this can be done with two lines of code.

if not contract.storage[msg.data[0]]:
  contract.storage[msg.data[0]] = msg.data[1]


More requirements could be added easily, such as a minimal required deposit, a system for transferring control over names or verification of external credentials, for example github or social network accounts. The main problem occuring with the decentralised alternatives is trust. If the registry is not a generally accepted mapping and the contract is not a world wide default, the problem is only extended because the contract itself must be named. There are some organisations working on this problem, in the end it really comes down to the preferences of influential software developers.

[John Cowan]

Arndt Schnabel scripsit:

> This touches two of the most important problems of decentralised systems:
> identity and trust. It is also exactly what Zooko's triangle predicts:

There are two known methods of squaring Zooko's triangle up to Sybil attacks.
See the WP article <https://en.wikipedia.org/wiki/Zooko%27s_triangle>
and its references.

--
John Cowan http://www.ccil.org/~cowan co...@ccil.org
It's like if you meet an really old, really rich guy covered in liver
spots and breathing with an oxygen tank, and you say, "I want to be
rich, too, so I'm going to start walking with a cane and I'm going to
act crotchety and I'm going to get liver disease. --Wil Shipley

[Arndt Schnabel]

And Ethereum aspires to become the generalised consensus layer of the web that they imagined. (In fact, one Ether cryptocurrency unit is named 'szabo' :) )

To return to my main quest: I currently implement noflo-ipfs, due to an upstream bug (that will be solved with a merge with the main branch later), it still does not work, but I'm optimistic this happen in the next few days. If anyone is interested to contribute, I will create a public repo on github, I am certain most people here are way more experienced than I am with Javascript, CoffeeScript and NoFloJS - because I just discovered it.