ANN: GoFlow - Flow-based programming package for Go

[Vladimir Sibirov]

I'm glad to announce GoFlow package:

https://github.com/trustmaster/goflow

It provides a framework that lets you build Go applications in flow-based/dataflow/reactive programming style. A flow-based program is a graph of components which run in parallel and communicate by sending information packets through channels. You could surely achieve the same effect by writing plain Go code, but Flow-based programming (FBP) distinguishes 2 viewpoints in programming:

 - Graph view - what components the program consists of and how they are connected to each other. This is similar to DFD/SADT and is preferably represented by visual diagrams.

 - Component view - what inputs and outputs a component has and how it processes data. This viewpoint is focused on smaller entities which implement one specific function and the actual data processing code.

I'd describe major goals of Flow-based programming paradigm in general this way:

 * Concurrency by design, not by purpose. Operate with entities which work in parallel and share by communication by default.

 * Unified approach in software development from domain analysis and architecture to coding. Start with diagrams and data flows and continue the structural breakdown until you reach data processing algorithms.

 * Practices of hardware engineering mapped to software development world. Some say software could be as reliable as hardware if it was built in a similar way :)

 * Active use of visual diagrams. Human beings are likely to understand high-level things from diagrams easier.

Unfortunately GoFlow isn't capable of running visual diagrams yet. Currently this package does the following:

 * Turns your Go structs into FBP components and graphs.

 * Enables you to program components in event-driven way.

 * Lets you make networks consisting of process and other networks.

 * Provides a runtime that makes it work alltogether.

More information can be found on project page and wiki.

Looking forward to hearing from dataflow enthusiasts from the Go community.

Vladimir.

[Archos]

Is there any type of application where it could take any advantage if
were used that style based in flow?

[OmarShariffDontLikeIt]

This sounds very interesting. I've never heard of this approach
before.

Cheers,
Ben

On Feb 13, 5:48 pm, Vladimir Sibirov <trustmas...@kodigy.com> wrote:

[Brandon Peters]

Yes, this does sound very interesting, thanks for putting it together Vladimir.  I look forward to trying it out.

[Sebastien Binet]

Vladimir,

On Mon, Feb 13, 2012 at 6:48 PM, Vladimir Sibirov
<trust...@kodigy.com> wrote:
> I'm glad to announce GoFlow package:
> https://github.com/trustmaster/goflow

I had a quick during my way back home on the train: looks very nice
and I might test it at some point for my app.

one thing I am not sure GoFlow can handle (at least from my quick
eyeballing of the doc and code) is the following use case:

- a PointsProducer process produces Point{x,y} structs and send them
on its Out port,
- a XMeanConsumer takes the Point{} from its In port to calculate the
running mean of this stream of Points,
- a YMeanConsumer takes the Point{} from its In port to calculate the
running mean of this stream of Points,
- you want each of the {X,Y}MeanConsumer processes to consume the
'same' stream of Points.

how do you handle this w/o sprinkling too much plumbing in users' code ?
I suspect this should be handled by the Graph component but I couldn't
find any kind of "repetitor" nor "broadcaster"...

-s

[Vladimir Sibirov]

So far Flow-based programming has been used quite successfully in the
following application areas:

- Data analysis and ETL (e.g. Pentaho Kettle, Pypes).
- Transactional business applications such as banking software
(originally FBP was made for it).
- Digital signal processing is flow-based. But GoFlow in particular
is probably too high-level for DSP.
- Multimedia broadcasting (e.g. Kamaelia).
- Simulations (LabVIEW is flow-based).
- Networking (AMQP is a similar approach).

I'm personally experimenting with FBP in web development.

[Archos]

I went to say you that the model of your framework reminded me to a
web framework in Erlang based in flow, Webmachine[1]. Here you have
the diagram[2] about the flow of processing


[1]: http://wiki.basho.com/Webmachine.html
[2]: http://wiki.basho.com/Webmachine-Diagram.html

[Vladimir Sibirov]

Hi Sebastien,

This question deserves a detailed answer.

There is a rule in Flow-based programming defined by J. Paul Morrison: multiple outputs can be connected to a single input, the packets will arrive in a simple FIFO order; single input cannot be connected to multiple outputs, replicator components should be used in such case instead. He explains that in second case the expected behavior may be undetermined and lead to confusion between sender and receivers.

GoFlow isn't bundled with any generic components yet, but a replicator for your example would be as simple as this:

type PointReplicator struct {

flow.Component

In   <-chan Point

Out1 chan<- Point

Out2 chan<- Point

}

func (r *PointReplicator) OnIn(p Point) {

r.Out1 <- p

r.Out2 <- p

}

A generic replicator would be a more elegant solution, but it would have to operate on channels of interface{} type. This means that it has to be connected to channels of interface{} type too and components which connect to replicator would have to use type assertions. Which is not very elegant. The same problem exists for other generic components, e.g. packet counters.

Probably implementing special "broadcast" outports which allow multiple subscribers is a better idea. I'll think it over.

2012/2/14 Sebastien Binet <seb....@gmail.com>

[kortschak]

That looks very nice. I have a question about the example in the
README though - why include the time.Sleep call?

[Sebastien Binet]

Valdimir,

On Tue, 14 Feb 2012 14:05:21 +0400, Vladimir Sibirov <trust...@kodigy.com> wrote:
> A generic replicator would be a more elegant solution, but it would have to
> operate on channels of interface{} type. This means that it has to be
> connected to channels of interface{} type too and components which connect
> to replicator would have to use type assertions. Which is not very elegant.
> The same problem exists for other generic components, e.g. packet
> counters.

right. another usecase for generics, I suppose.
although, in your case, automatically generating stub components with
the correct number of in/out ports with the correct channel type, and
then automatically inserting them at the adequate network vertices,
shouldn't be a big issue.
(that's what I do in my pie-in-the-sky "framework". the automated code
generation part, that is.)

-s

--
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################

[Vladimir Sibirov]

Oh, yes, Erlang and FBP are well aware of each other and share some
principles. Thanks for the link, a flow-based web framework in Erlang
is interesting.

[Vladimir Sibirov]

It is because otherwise close(in) will terminate the application
immediately. I'd say it is a bug and needs better built-in
synchronization, see <https://github.com/trustmaster/goflow/issues/1>

[Vladimir Sibirov]

I'm not fond of code generators and try to use handy APIs whenever possible instead, but if there is no other way, I'll write code generators for generic components (indeed, this is mimicking a generics compiler). Thanks for the idea!

2012/2/14 Sebastien Binet <seb....@gmail.com>

[roger peppe]

i haven't had time to look at this package, but one superficial
comment from a moment's glance at the documentation:
it would be great if the documentation used the standard Go
style of comments - full sentences.

for instance instead of:

"Runs the network by starting all of its processes."

you'd write

"RunNet runs the network by starting all of its processes."

it makes everything read so much more nicely.

[spir]

On 02/14/2012 11:05 AM, Vladimir Sibirov wrote:
> Hi Sebastien,
>
> This question deserves a detailed answer.
>
> There is a rule in Flow-based programming defined by J. Paul Morrison:
> multiple outputs can be connected to a single input, the packets will
> arrive in a simple FIFO order; single input cannot be connected to
> multiple outputs,

Did you mean "single output cannot be connected to multiple intputs" ?

Automation programming looks somehow similar to FBP, I guess. Like GUI
programming, it is based on a cycle. The main difference is the things
that actually do events (actuators) or inform on them (sensors) are pure
hardware; thus, as expected, automation reaches very high levels of
reliability for extremely complex machines.
Typically, sections of programms read relevant inputs, process, set
outputs (i/o is updatedby the framework between cycle loops).

Sebastien's example would be directly written as is in automatics.
Inputs are just available, any part of the program reads them as needed.
The point of difference, i guess, may be that inputs and outputs are
indeed targeted, but to hardware components, not software (they are I/O
cards ;-), interfaces between hardware and software in both directions).
If I understand correctly, in FBP outputting info is targetting it to a
given receiver, so to say through a private port, thus also preventing
others to read it. Right? If yes, why? Why don't emitters just let info
available? (I'm sure there are good answers about program correction,
consistency, reliability and such...)

Denis

PS: I am rather sure for a while already that general-purpose
programming would highly benefit to study the "paradigm" of automation.
sure, one cannot make software be hardwere; but we can take inspiration
from what defines hardware and makes it different (read: incredibly
reliable compared to the state of the art in programming) to model a way
to design our components, and (even more important?) how they relate to
each other --I mean physically.

[Vladimir Sibirov]

Indeed, I meant "single output cannot be connected to multiple inputs".

> If I understand correctly, in FBP outputting info is targetting it to a given receiver, so to say through a private port, thus also preventing others to read it. Right? If yes, why? Why don't emitters just let info available? (I'm sure there are good answers about program correction, consistency, reliability and such...)

Components are generally reusable. You can treat them as hardware chips which have a set of input ports and output ports forming their interface. Knowing their interface you can wire them to other components making a bigger circuit. Which can have its own inputs and outputs and can be used in even bigger circuits. Just like hardware components in integrated circuits, software components have limitations on their input/output "load".

While FBP is closer to hardware design principles with its wiring, Actor model uses a different approach: an actor sends a message to the addresses known by the actor (including those which it has received in a message before). There is another interesting approach called "tuplespace" in Linda coordination language: senders emit messages to the tuplespace and receivers consume messages from there like piranhas. And there are Go channels which are originated from the late CSP (Communicating Sequential Processes). It is proved that Actor model can be represented by CSP and vice versa. GoFlow is an FBP implementation on top of CSP, which gives plenty of freedom if you really know what you're doing. Sorry for this digression.

Vladimir

2012/2/15 spir <denis...@gmail.com>

[kortschak]

It's race dependent then, because I tried the example without and it
worked fine. From what you say, this would depend on the network being
run.

thanks

[Vladimir Sibirov]

It was. I've already fixed it though.

2012/2/16 kortschak <dan.ko...@adelaide.edu.au>

[Maxim Pimenov]

Hello Vladimir!

I've run GoFlow under ThreadSanitizer (http://code.google.com/p/data-race-test).

Here is an excerpt from the report, see the full log attached:

WARNING: Possible data race during read of size 1 at 0x57529c: {{{

   T28 (L{}):

    #0  github.com/trustmaster/goflow.TestInitFinish ~/go/src/pkg/github.com/trustmaster/goflow/component_test.go:144

    #1  testing.tRunner ~/go/src/pkg/testing/testing.go:262

    #2  runtime.goexit ~/go/src/pkg/runtime/proc.c:266

  Concurrent write(s) happened at (OR AFTER) these points:

   T30 (L{}):

    #0  github.com/trustmaster/goflow..Finish ~/go/src/pkg/github.com/trustmaster/goflow/component_test.go:121

    #1  github.com/trustmaster/goflow.shutdownProc ~/go/src/pkg/github.com/trustmaster/goflow/component.go:184

    #2  github.com/trustmaster/goflow._func_002 ~/go/src/pkg/github.com/trustmaster/goflow/component.go:152

    #3  runtime.goexit ~/go/src/pkg/runtime/proc.c:266

}}}

Although it may look like a mere race in a test, the real reason for it lies deeper.

It seems that your usage of sync.WaitGroup is incorrect.

For those interested, the case boils down to this (not uncommon) misuse of

WaitGroup:

-- there are nodes whose children are also nodes, at least 1 goroutine per node

-- a node may start only when all its children have finished their jobs

-- sync.WaitGroup is used to account for this

-- instead of a single wg.Add(numChildren) in the parent, each

child registers itself with parent.wg.Add(1)

It does not provide the required synchronization: wg.Wait() in the parent

may proceed before a child registers itself as opposed to after it

registers itself and signals of its completion.

[Jim Whitehead II]

Indeed, when using a WaitGroup you should always increment the count
before you've spawned the goroutine in question to ensure correct
synchronization. Good catch!

- Jim

[Vladimir Sibirov]

Thanks for pointing at that, Maxim!

Could you provide the tsan options you used? I'm getting quite different results with lots of warnings about races in the Go runtime itself.

It doesn't seem that component_test.go is affected by the problem you described because it doesn't spawn any goroutines before calling wg.Add(), it spawns them afterwards. Though I agree that the problem persists for network_test.go and nested calls to RunNet() may lead to the effect described.

2012/2/22 Jim Whitehead II <jnwh...@gmail.com>

[Samuel Lampa]

Just wanted to chime in and add that I totally enjoy dabbling with the GoFlow library!

Also, for anyone looking for more example code, below are links to my very basic dabbling around with it:

* Started a repo for a (wish-to-become) "flow based bioinformatics library": 
  github.com/samuell/blow
 (only one "bioinformatics component" so far :P )
* Made a real dummy workflow, of four chained base complementers, running serially on an input file of DNA code (in "fasta" format), and printing it out on stdout in original format again:

  gist.github.com/samuell/6203010

What did strike me is how easy all of this is on the conceptual level (well also there's not much of a tech stack to it either), while still providing such an elegant and powerful modularization. Haven't really seen anything like that before. 

Flow based / FBP for sure has came to stay!

Cheers

// Samuel

[Vladimir Sibirov]

Hi Samuel,

Nice to hear that it makes sense!

I haven't announced it yet, but GoFlow now also supports loading network definitions in NoFlo JSON format which you can draw with the latest version of JPM's DrawFBP tool. I just need a few peaceful days to update the docs and polish the code a bit. And then we can proceed to getting onboard of NoFlo-UI.

Best regards,

Vladimir Sibirov

2013/12/13 Samuel Lampa <samuel...@gmail.com>

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/bJOf8cwHfy8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Samuel Lampa]

On 2013-12-13 08:07, Vladimir Sibirov wrote:

I haven't announced it yet, but GoFlow now also supports loading network definitions in NoFlo JSON format which you can draw with the latest version of JPM's DrawFBP tool. I just need a few peaceful days to update the docs and polish the code a bit. And then we can proceed to getting onboard of NoFlo-UI.

Aha, that is great news!

I didn't know that the DrawFBP tool will support it too! That will help a lot. Having a stand-alone tool for drawing networks as well, is very good thing for many use cases I think, so that is all great news!

Btw, must say the stuff up at the NoFlo-UI:s new home, flowhub.io looks totally super-slick, so that will be very exciting to see go live, and GoFlow support will be just the perfect backend IMO.

But how about compatibility with the component repositofy/ies in Flowhub? Will there need to be ported versions of NoFlo:s components in GoFlow too, or do you know if it will be possible to have custom component repositories?

Cheers
// Samuel

-- 
Developer at www.uppmax.uu.se/uppnex / www.farmbio.uu.se / rilpartner.se
G: http://google.com/+samuellampa
B: http://saml.rilspace.org
T: http://twitter.com/smllmp

[Aaron Schlesinger]

Vladmir - 

Very interesting! Is goflow similar to the reactive streams project (https://github.com/reactive-streams/reactive-streams/blob/v1.0.0.M3/README.md#specification)?

Aaron

[Vladimir Sibirov]

Hi Aaron,

The Reactive Streams look similar in some aspects, current GoFlow implementation is mostly reactive. But in general GoFlow is inspired by a different model. For more details see https://github.com/flowbased/flowbased.org/wiki

Best regards,

Vladimir

[Aaron Schlesinger]

Thanks Vladimir - I've added that wiki page to my reading list

[liew...@gmail.com]

Hello Vladimir,
Is there any further developement on goflow? I see that it has last updated since Nov 7, 2015.

Are there any further intergration in goflow with noflo-ui?