Go (language)

[Paul Tarvydas]

Concurrency is not parallelism (it's better)
http://concur.rspace.googlecode.com/hg/talk/concur.html#landing-slide

Simplicity is complicated
http://www.thedotpost.com/2015/11/rob-pike-simplicity-is-complicated

pt

[Paul Tarvydas]

In all likelihood, I've been using terminology in a confusing way, because I think in terms of events.  To clarify:

Events sent between components are asynchronous.  The events can be sent at any time and are not synchronized in any way.

The components are concurrent, because they can handle asynchronous events.

pt

[Paul Tarvydas]

My early impression of Go is that it is FBP.  With some added baubles.

Go treats channels as first-class objects.  Channels are bounded buffers.

Functions can be made concurrent by prefixing them with the keyword "go".

Functions can be turned into FBP components by passing the input channels and the output channels as arguments to the functions.

Selective receive is supported ("select" statement).

Here is my first attempt at writing Collate (page 91).  Unless I've made a major blunder, it looks pretty simple:

https://github.com/guitarvydas/go-fbp-collate/blob/master/go-fbp-collate.go

I (we) need to experiment with fan-in - can more than one component output to the same channel?

I haven't yet looked at Rust, which also supports channels.

pt

[Humberto Madeira]

Hi Paul,

It seems like the concepts of both ports and flow connectors are being conflated into the channel concept..

How do you isolate the component from the flow connections without the concept of ports?

Or am I missing something here?

Regards,

--Bert

[Paul Tarvydas]

My knee-jerk reaction was to make the parameters to a function be the "ports" and the caller (the parent uber-net) create the channels and pass them into the appropriate parameters.  This seems to work, other than maybe needing a nil-check on a parameter to allow for N.C.'s (No Connection, as in hardware design).

Tell me more about what you see the difference being...

thanks

pt

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Humberto Madeira]

Hi Paul,

>other than maybe needing a nil-check on a parameter to allow for N.C.'s

Precisely that, it seems to me that if any of in0, in1 or out are nil, then you will have some runtime problems.

You could maybe do a check of some sort, but this is one of the things you get for free (as in not having to code it every time) when you use ports.

Also, in classic FBP, you should be able to connect more than one connection to a specific input port.

I realize that you probably wouldn't want to do that in a collation example.  But it is quite common in the more general FBP case (I use it quite a lot when I do looping).

So how would you do that here without modifying the code of the component, or requiring another component to merge the streams?

And (although I use something different, for similar purposes) how would you handle the notion of array ports?

Best Regards,

--Bert

[Paul Tarvydas]

On 15-12-26 08:21 PM, Humberto Madeira wrote:

Hi Paul,

>other than maybe needing a nil-check on a parameter to allow for N.C.'s

Precisely that, it seems to me that if any of in0, in1 or out are nil, then you will have some runtime problems.

You could maybe do a check of some sort, but this is one of the things you get for free (as in not having to code it every time) when you use ports.

a) If you allow for N.C.'s, then a nil-check has to be done somewhere, it's never free.  The only thing, then is syntactic sugar.  This could be done by making Send() a function and hiding the nil-check there.

b) IMO, doing a nil-check in Send() is wrong.  The component should not know what it is connected to, nor if it is not connected.  Knowledge about the connection should only be kept in the parent.  This leads to a different solution - the /dev/null solution.  N.C. can be implemented by a channel pointing to a reader that simply dumps whatever comes in.  There might be other solutions that I haven't thought of yet.

Also, in classic FBP, you should be able to connect more than one connection to a specific input port.

Supported in Go.

And (although I use something different, for similar purposes) how would you handle the notion of array ports?

With arrays (slices) of channels?  I've never used array ports, so I might be missing something, here...

pt

[Humberto Madeira]

Hi Paul,

On Sunday, 27 December 2015 12:06:57 UTC-5, Paul Tarvydas wrote:

a) If you allow for N.C.'s, then a nil-check has to be done somewhere, it's never free.  The only thing, then is syntactic sugar.  This could be done by making Send() a function and hiding the nil-check there.

That should work, IMO.

 

b) IMO, doing a nil-check in Send() is wrong.  The component should not know what it is connected to, nor if it is not connected.  Knowledge about the connection should only be kept in the parent. 

 

+1 agree

 

This leads to a different solution - the /dev/null solution.  N.C. can be implemented by a channel pointing to a reader that simply dumps whatever comes in.  There might be other solutions that I haven't thought of yet.

I'm not completely sure what suggestion to offer here.  

My own framework doesn't implement a receive(), at least not in the body of a Task (roughly a component-equivalent).

This design difference means that if a data item is never sent to a Task, or if nothing is connected to one of the Inlets of the Task, then the performing method of the Task will never get called.

The connection attachment is not done directly to the Task, but to  a wrapper object that contains the Task (amongst other things).

In my own framework, data items come in to the task one at a time wrapped in a Context object that also contains a reference to the specific Inlet that the data object used to come in (again, amongst other things).

Data items leave the task by explicitly entering one (or more) of the Outlets.  Flow connections connect explicitly from Outlets to Inlets (or their equivalent). 

The main design intent was to keep as much  as possible of the mechanics of the flow framework out of the Task itself.

Translating these ideas to the code example, I would probably implement some sort of generic context object into which you could add Inlets and Outlets as needed by the component (typically in some sort of factory method or constructor), 

context.addInlet("in0")

context.addInlet("in1")

context.addOutlet("out")

then some sort of code to connect the channels to the Inlets and Outlets at configuration time 

context.getInlet("in0").connect(channel0)

context.getInlet("in1").connect(channel1)

context.getOutlet("out").connect(channel2)

and finally then pass the configuration into the component method at network startup time

component.start(context)

From inside the component method you could use something like

data = context.getInlet("in0").receive()

... do stuff

context.getOutlet("out").send(data)

In my own framework, since the task body gets called separately for each arrival, the code looks more or less like this

inputValue = context.getData()

inletName = context.getInletName()

... do stuff

context.getOutlet(OUTLET_OUT).enter(outputValue)

I'm not sure if this suggestion is workable in the classic FBP context, or if there is a better, or even, more canonical way to do it WRT classic FBP.

Perhaps someone else could offer an opinion, or additional alternatives?

Also, in classic FBP, you should be able to connect more than one connection to a specific input port.

Supported in Go.

Ok.  I missed that. (not familiar with Go)  An example might help.

And (although I use something different, for similar purposes) how would you handle the notion of array ports?

With arrays (slices) of channels?  I've never used array ports, so I might be missing something, here...

Sadly, I'm not that familiar with array ports either, but I believe (from what I have read about them) that they offer some sort of ability to add ports at flow configuration-time.

With my own framework, I can add additional inlets (or outlets) to a Task at flow-configuration time such that the task doesn't have to know about them statically ahead of time.

In the case of inlets, I use this feature to map the inlet to a join dimension.  I would imagine that array ports could be used for the same thing (perhaps someone can confirm?),

You could possibly use an array port called "dimension" where it could have some number of dimensions eg. dimension[0], dimension[1]... that sort of thing

I would also imagine that the main advantage of array ports is that you could have more than one array port, so you could identify the extra ports by some sort of named category, (such as "value", "count" etc)

eg value[0], value[1], count[0], count[1] The task would know about each array port "category" ahead of time, but would't need to know about the specific dimensions until it got configured.

I suppose if I were to extend my own framework in a similar direction as array ports, the equivalent would be map ports (where currently I only have one implicit map port)

Something like value[person], value[time], count[person], count[time] etc  

Not sure how I would mangle the current syntax to accommodate it, but it's a feature I'm likely to add when the time comes.

Without something like dynamic ports (plain, array, or map) you would likely have to hard-code any joining-type task to know the dimensions of the join in advance.

In my own case, I prefer as much as possible to configure this sort of thing, to avoid bringing in a programmer to create additional (essentially boilerplate) code,

or to require the framework to do code compilation of any sort

Regards,

--Bert

[Paul Morrison]

Various points - reading this Group is like drinking from a fire hose!

- I am quite willing to accept Rob Pike's definition of "concurrency", and it certainly seems as though "classical" FBP is "concurrent" by this definition.

- I sort of see how ports are mapped onto parameters, but how would Paul T's input parameters handle array ports? Something like (in <-chan[] string) - and it should read 'IP' instead of string... so
  (in <-chan[] IP) ?

- In all my FBP implementations, input ports (or input port array elements) correspond 1:1 to input connections - one output port cannot be connected to more than one input channel  (unlike NoFlo!)

- yes, array ports allow the array size to be determined at run time - in these same implementations, "openInputArray" constructs "inportArray" at run time, which can then have its length queried by the run code.  Typically, array ports allow components to support multiple input or output ports which basically have the same function - and since you have to say in the component attributes (where supported) whether a port is array-type or not, at configuration time I treat IN[0] and IN as essentially the same.  If the elements have different functions, I would use different ports and port names.

- this also allows the component to have other ports with different names (either array-type or not) - Collate has CTLFIELDS

- Paul T's Collate goroutine in his Go Collate example seems to be very different in a number of ways from the JavaFBP, C#FBP, JSFBP Collates (JSFBP is missing the logic to insert brackets in the output - my bad!),

      -   the Collate component should treatsall IN port array elements identically, and the number of elements may range from 1 to indefinite (the 1 case just does bracket insertion), determined at network
          configuration time
      -   Collate  assumes all input streams have been sorted - and by the same key values
      -   the two readers can use the same code - or at least the two readers should generate IPs (information packets) which are acceptable to Collate - I would have preferred to see that in the example
      -   the general logic seems to be wrong - at a high level, Collate should do the following:
           - it starts by receiving an IP from each input port array element and saving it in an IP array
           - initialize a field (call it 'hold') to high values 
           - repeat following:
               - for each input port element,
                     - do a receive and compare the key against 'hold'
                     - if key is low, set 'hold' to key, and save port element number
                     - if key is equal, save port element number
              - output IP saved in IP array [saved number]  (adding in bracket logic, based on this key and previous key)
              - receive IP from input port array element [saved number], saving it in IP array [saved number]
          - keep doing the above - until end of stream on all elements
      -  a useful side effect is that, if two records have the same key, the one from the lowest numbered array port element goes out (and is refreshed) first
      -  note also that Collate does not reference a particular element number explicitly
    
- the code in https://github.com/jpaulm/javafbp/blob/master/src/main/java/com/jpmorrsn/fbp/components/Collate.java  is a bit more complex than needed to illustrate this function, as Collate has to decode a list of control fields, but the key logic is in statements 77-114 .  "sendOutput" looks after inserting close and open brackets - as I said above, JSFBP seems to be missing this function - I have raised it as an issue!

- have either of you looked at Vladimir Sibirov's Goflow - https://github.com/trustmaster/goflow ?  Haven't looked into it, but my impression is that it hews closer to NoFlo than "classical" FBP... but I may be wrong!  Vladimir, any comments?

Bert, how can you say, "Sadly, I'm not that familiar with array ports either, but I believe (from what I have read about them) that they offer some sort of ability to add ports at flow configuration-time."?

a) that's not their purpose, but it wouldn't be too hard to add either kind of port at run-time - we just have to sacrifice the config time checking against component attributes, or have some kind of half-way house...
b) apparently, you have attended all the Toronto meetups and have never read the book...  Tsk!   ;-)

Finally, I will ask again: why do you all keep trying to implement FBP in various exotic languages, rather than just using JavaFBP or C#FBP (or CppFBP or JSFBP) for real applications and building up a body of knowledge which we can all benefit from?  This question does not apply to Paul T or Matt, as I know your reasons for choosing your particular languages!  

[Vladimir Sibirov]

Re: GoFlow vs. NoFlo vs. Classical FBP

Originally GoFlow was more reactive than classical but at some point I added support for classical FBP components:

https://github.com/trustmaster/goflow/blob/master/component_test.go#L400

You can actually mix classical and reactive components within the same network. As well as mix synchronous components with asynchronous, blocking semantics with non-blocking and what not.

To be honest, nowadays I think that GoFlow as it is tends to offer too many options and represent too many models at the same time (which was the goal of the original academic project). Having done some real world dataflow and FBP programming, these days I lean more towards using just classical FBP components alone and I would like to rewrite GoFlow one day using just the best practices learnt and throwing the options that nobody would ever need away.

Best regards,

Vladimir

[Paul Tarvydas]

On 15-12-29 12:38 PM, Paul Morrison wrote:
> Various points - reading this Group is like drinking from a fire hose!
>

:-)

> ...

> - Paul T's Collate goroutine in his Go Collate example seems to be
> very different in a number of ways from the JavaFBP, C#FBP, JSFBP
> Collates (JSFBP is missing the logic to insert brackets in the output
> - my bad!),
>

I'll reread this post a few times and improve the implementation, as
time permits :-). I've been assuming that the main reason for
discussing Collate is to test whether back-pressure is possible in
various implementations. My understanding of Collate has been clouded /
eroded by that consideration.

My main interest in this is to poke at Go, and maybe Rust, to see how
close they are coming to the concepts we discuss here. Failure is an
option :-).

I don't see (yet) any problem incorporating the concepts in this post,
e.g. Go channels can carry anything, structs, other channels, etc. I
believe that arrays (slices) can be defined at runtime and their sizes
can be queried with len() and cap().

We'll see...

pt

[John Cowan]

Paul Morrison scripsit:

> - I am quite willing to accept Rob Pike's definition of "concurrency", and
> it certainly seems as though "classical" FBP is "concurrent" by this
> definition.

Jugglers have only two hands (parallel execution engines) but can keep N
balls in the air (N-way concurrency). Indeed, you can juggle with just
one hand.

> - In all my FBP implementations, input ports (or input port array elements)
> correspond 1:1 to input connections - one output port cannot be connected
> to more than one input channel (unlike NoFlo!)

Go does not seem to have this restriction, but it is clear that if
multiple goroutines (processes) try to read from a channel, it is
non-deterministic which process gets which information packet. Usefully,
channels are first-class in Go, so for example a "reply channel" can be
created and sent to another goroutine, which can then use it to send
a reply without interfering with any other input channels the sending
goroutine may have.

--
John Cowan http://www.ccil.org/~cowan co...@ccil.org
Rather than making ill-conceived suggestions for improvement based on
uninformed guesses about established conventions in a field of study with
which familiarity is limited, it is sometimes better to stick to merely
observing the usage and listening to the explanations offered, inserting
only questions as needed to fill in gaps in understanding. --Peter Constable

[Paul Morrison]

Hi Paul,

Collate should test back pressure, but I tend to see it more as a good test for selective read.  Plus, I thought that, as you quoted a page number from the book, you were thinking in terms of the logic described there!  As the function of your Collate is so different from the Collate I describe, I would rather see it given a different name!  

Yep, maybe we should step up to the problem of naming wrt reusable components - it could get to be a big problem!  :-)  Remember when we only had 8 characters...?!

pt

[Alfredo Sistema]

My problem with understanding Collate is that I was missing the problem it actually solves. Providing a test that a Collate component has to pass should be good for posterity.

[Paul Morrison]

Alfredo and Paul T.,

Rereading p. 88, I see it's pretty brief!  It says: "If C is a Collate, then the output of A will be merged with the output of B according to key values – usually the key fields are specified to Collate by means of option IPs."  I probably should have said there that the assumption is that both input streams are sorted, and on the same keys.  However, it does say that on p. 91.

When you say "test case", what medium do you want it in?  The same page does show some sample output from a Collate - or would you prefer working code, in which case I could add Collate (and maybe other component) tests to the various *FBP Github directories.  I can add a bunch of "component test" networks in a separate directory.  Actually I like that idea - hopefully this is what you had in mind!

[Tom Young]

Hi Paul,

On Tue, Dec 29, 2015 at 12:38 PM, Paul Morrison
<paul.m...@rogers.com> wrote:
> Various points - reading this Group is like drinking from a fire hose!
>

Seems like only yesterday you were wondering why there were so few
posts to this group.

> ...

>
> Finally, I will ask again: why do you all keep trying to implement FBP in
> various exotic languages, rather than just using JavaFBP or C#FBP (or CppFBP
> or JSFBP) for real applications and building up a body of knowledge which we
> can all benefit from? This question does not apply to Paul T or Matt, as I
> know your reasons for choosing your particular languages!

I see the DFD as extending the use of *nix pipes, such that existing
filters and other executables can be quickly and easily connected
together into a larger network of executable components. As
mentioned below by Richard Stevens, this can be difficult without
something like the DFD to connect the plumbing. This approach permits
components to be written in any language, I'm aware of, with no
special FBP coding or API required. For example, the VDFD editor uses
TCL/TK, PERL, C, and GraphViz components in a cyclic DFD network.

C language components can make use of a DFD API to enable special
features such as named ports, but this in no way affects other
components.

The DFD is written in C for a number of reasons, but mainly because I
would not know how to create such a system
in any other language.

Cheers!

"...the shell syntax required to initiate a coprocess and connect its
input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

[Paul Morrison]

Alfredo, there is a test for Collate in JavaFBP at https://github.com/jpaulm/javafbp/blob/a0578d8e000b999b5fe0fdfaaa04b68afe777702/src/main/java/com/jpmorrsn/fbp/examples/networks/Update.java

This network generates collated output to the console, showing where the open and close brackets are inserted.  It runs fine under Eclipse.

If you could let me know how you would like to see that modified, I will try to enhance it accordingly.

Regards,

Paul

[Paul Morrison]

Hi Tom,

I've been seeing your closing quote by W. Richard Stephens for quite a while without understanding what he (or you) were getting at!  Not sure why he thinks the notation would be contorted...  I think the other Stevens' notation ( A OUT -> IN B OUT -> IN C ) is pretty straight-forward, but for me diagrammatic notations are definitely easier to grasp.

And I have been feeling for a while that C would be the best base for a language-agnostic FBP - I can't quite visualize it yet though!  Probably Java will be a problem...?   But somehow *nix is in there playing a role also!   Do you have any running code with Java?

Season's Greetings!

[Kenneth Kan]

- Paul T's Collate goroutine in his Go Collate example seems to be very different in a number of ways from the JavaFBP, C#FBP, JSFBP Collates (JSFBP is missing the logic to insert brackets in the output - my bad!),

      -   the Collate component should treatsall IN port array elements identically, and the number of elements may range from 1 to indefinite (the 1 case just does bracket insertion), determined at network
          configuration time
      -   Collate  assumes all input streams have been sorted - and by the same key values
      -   the two readers can use the same code - or at least the two readers should generate IPs (information packets) which are acceptable to Collate - I would have preferred to see that in the example
      -   the general logic seems to be wrong - at a high level, Collate should do the following:
           - it starts by receiving an IP from each input port array element and saving it in an IP array
           - initialize a field (call it 'hold') to high values 
           - repeat following:
               - for each input port element,
                     - do a receive and compare the key against 'hold'
                     - if key is low, set 'hold' to key, and save port element number
                     - if key is equal, save port element number
              - output IP saved in IP array [saved number]  (adding in bracket logic, based on this key and previous key)
              - receive IP from input port array element [saved number], saving it in IP array [saved number]
          - keep doing the above - until end of stream on all elements
      -  a useful side effect is that, if two records have the same key, the one from the lowest numbered array port element goes out (and is refreshed) first
      -  note also that Collate does not reference a particular element number explicitly
     

Hi Paul,

It's great to hear what you expect out of the Collate Challenge. I had always thought that the litmus test was to see if one could implement Collate in a particular (potentially) FBP framework. After having worked with Paul T on his reactive approach and looking into Matt Lai's tclfbp, I realized (maybe I'm still wrong) that the litmus test was not to test how a Collate is implemented, but whether it guarantees against overflows, given FBP classic's back pressure mechanism.

If a Collate is implemented as a black-box component and you feed it the same series of IPs in the same way (with array ports, etc) and get back the same output IPs, in the expected order, would that be enough to say this particular framework is FBP classic, even if in this framework there is no back-pressure? What if it provides back pressure in another way, like with Matt Lai's traffic cop pattern?

I'm a bit confused as to what the litmus test is about. Is it testing the implementation within the process, which doesn't make much sense given FBP is supposed to be a black-box paradigm? Is it testing whether there is back-pressure so things don't "blow up", in which case Matt's reactive FBP is actually FBP classic, I believe? Or is it simply testing whether given the same input IPs, though the order between streams is not guaranteed, you would get the same output IPs, which is what I think Aflredo meant by "the test"?

In any event, happy new year everyone. :)

[Tom Young]

Hi Paul,

In your example below; A, B, and C are simple filters, not the kind of
coprocesses Mr. Stevens had in mind, I think. A, B, and C are
simply concurrent, but there is no feedback. In the Linux bash shell
this would simply be A|B|C.

However, when you attempt (in Linux, say) to do something like A out
-> B in; B out -> A in, you get something like
mkfifo /tmp/fifo; A< /tmp/fifo |B > /tmp/fifo; rm /tmp/fifo
or worse. The DFD allows "(a A)->(b B)->(a)"; where A and B are the
executables and a and b are temporary process names. About as simple
as I can get it and allow for current shell features.

Diagrams are fine. Systems can be created and executed with the VDFD
without ever creating a DFD network file. A think a diagrammatic
notation of some sort is essential, though, if for no other reason
than to store diagrams efficiently.

Java should not be a problem for the DFD; don't think I have any Java
code though. If you send suitable working Java code I should be
able to refactor it into two or more Java coprocesses running under
the DFD.

*nix makes things easier for me.

Cheers,

Tom

"...the shell syntax required to initiate a coprocess and connect its
input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

[Tom Young]

Hi Again Paul,

Now I'm back from the GBA(Greater Boston area), I can answer your
question better below.

On Thu, Dec 31, 2015 at 4:45 PM, Paul Morrison <paul.m...@rogers.com> wrote:
> Hi Tom,
>
> I've been seeing your closing quote by W. Richard Stephens for quite a while
> without understanding what he (or you) were getting at! Not sure why he
> thinks the notation would be contorted... I think the other Stevens'
> notation ( A OUT -> IN B OUT -> IN C ) is pretty straight-forward, but for
> me diagrammatic notations are definitely easier to grasp.
>

Richard Stevens dedicates seven pages of his book [1993] (referenced
below) to coprocesses including examples in C. He cites pp. 65-66 of
Bolsky and Korn[1989] for details supporting his
"...coprocess..contorted.." assertion. His first example consists of
two processes where each gets its standard input from the other's
standard output.

Stevens states "A filter[a program reading standard input and writing
standard output] becomes a coprocess when the same program generates
its input and reads its output". I interpret 'program' to mean the
combined processes.

His book describes a solution to the problem of connecting a coprocess
to programs like 'awk' which buffer I/O, by using pseudo terminals
to avoid deadlock -- a major shell contortion. Pseudo terminals work
because *nix filter output is not buffered when connected to a
terminal.

I highly recommend this book as a reference for most things *nix,
especially for C/C++ language programmers.

By the way, there exists a 'coproc' bash shell function, which I have
never used. 'coproc' connects the standard input and output of
another process(command in shell speak) to the standard output and
input respectively of the current running shell. I assume I/O
buffering might be a problem; also there can be only one running
coprocess at a time-.

So what I am getting at is that the DFD avoids shell contortion.
--twy

[Paul Morrison]

Hi Ken,

Happy New Year to you too!

This is an interesting question!  I totally agree that, given the same input data, Collate should produce the same output every time!

It should also:
 - never crash
 - not deadlock (or rather the network that uses it)
 - be very general, with identical behaviour independent of whether there is only one input port element, or tons
 - other criteria...?

And yes, IMO, the infrastructure must support back pressure and selective receives...

Right now, my "standard" (most commonly used) Collate just has a list of key lengths as parameter list, and assumes the key fields are contiguous starting at the 1st byte.  All these could be parametrized, as could be the field compare logic (e.g. using a standard (installation-wide?) subroutine or callback).

I will be adding bracket creation to the collate.js on the Github for JSFBP, and putting up a test case - I may try a "mocha-like" compare...

Cheers,

Paul

[Paul Morrison]

Re: Collate

It just occurred to me that some readers might be wondering why this interest in Collate? 

Well, it goes back to the early days of FBP - early '70s, say.  In those days, the majority of programs were batch (file-to-file, no human interaction), and probably over half of those were what we called "Updates", in which a sorted transaction file was run against a master file (naturally sorted on the same key as the transaction file).  Updates typically had multiple grouping levels, totals, subtotals, control reports, and who knows what else.  This was strictly one-thing-at-a-time coding, and had no reusable functions available.  All the programmer got was a general flow chart, called the "Balance Line" technique, which was almost impossible to adapt the customer's specific needs.

Story (p. 91 of the 2nd ed.): I once had the dubious pleasure of having to modify an update program (not FBP, obviously!) whose author had written the client an explanation of why his enhancement request could not be satisfied, which started, “Owing to the limitations of data processing,...”! My clear recollection is that modifying that program (and, for a conventional program, it was really quite well-written) was only almost impossible!

When I started playing with FBP concepts, I suddenly realized that I could build a Collate, based on the Collate machine of Unit Record days, which would vastly simplify the coding of Updates!  All of a sudden, what had before been one of the most complex coding jobs became incredibly simple - by the use of one generalized FBP component!  That was a real "aha!" moment!  The two approaches are shown on slides 9 and 10 of http://www.jpaulmorrison.com/fbp/FBPnew.ppt .  In slide 9, I couldn't fit the words in, but the diamonds are obviously compares.

A boss of mine, who I guess fancied himself as a logician, once asked me, "But mustn't complexity be conserved - surely you are just moving it around" - to which I realized (later) that you may be trying to do a job with a totally inappropriate tool, such as, say, digging a ditch with a knife and fork.  It can be done, but will take forever, and won't be a great job at the end of it!

[John Cowan]

Paul Morrison scripsit:

> "But mustn't complexity be conserved - surely you are just moving it
> around" - to which I realized (later) that you may be trying to do a job
> with a totally inappropriate tool, such as, say, digging a ditch with a
> knife and fork. It can be done, but will take forever, and won't be a
> great job at the end of it!

Well, your boss was right, but moving complexity into a tool is useful if
you are going to do the job more than once, or someone is. A bulldozer
is far more complicated to make than a fork, but once you've spent
that complexity budget, you can make many ditches and a huge number of
other things as well. Hence the programmer's maxim: if a problem is
not interesting, generalize it until it is.

--
John Cowan http://www.ccil.org/~cowan co...@ccil.org

Eric Raymond is the Margaret Mead of the Open Source movement.
--Bruce Perens, a long time ago

[Ged Byrne]

Hi Paul,

The conservation of complexity?

Does an energy saving lightbulb that provides the same light for a fraction of the energy fail to conserve energy? No, it eliminates waste.

The traditional lightbulb produces both light and heat. The energy saving lightbulb produces just light, and does not waste energy producing heat.

Brooke's points out in No Silver Bullet that there are two types of complexity: essential and accidental.

The essential complexity is already exists, it is inherent to the problem. The accidental complexity we unintentionally create ourselves trying to solve the problem.

With FBP you eliminated much of the accidental complexity. You are not moving the complexity around, you are avoiding it's creation.

Regards,


Ged

--

[Paul Morrison]

Hi Ged,

That makes sense!  Quoting  p. 17 of the 2nd ed.:

... there is a complexity measure used quite widely in the industry called McCabe’s Cyclomatic complexity measure, which is based very directly on the number of binary decisions in a program. In comparison, in our work, we discovered that the amount of logic in conventional programming is in fact reducible, because much of the logic in conventional programming has to do with the synchronization of data, rather than with business logic. Since FBP eliminates the need for a lot of this synchronization logic, this means that FBP actually does reduce the amount of logic in programs.

Cheers,

Paul

[John Cowan]

Paul Morrison scripsit:

> Since
> FBP eliminates the need for a lot of this synchronization logic, this means
> that FBP actually does reduce the amount of logic in programs.

Rather, it displaces the logic into the framework. I didn't even try to
write the synch code in JavaFBP; I stole it from Doug Lea, one of the
best in the business. But that's the dozer analogy: build your dozer
out of hundreds (thousands?) of parts, and it makes digging any number of
ditches easy.

--
John Cowan http://www.ccil.org/~cowan co...@ccil.org

He that would foil me must use such weapons as I do, for I have not
fed my readers with straw, neither will I be confuted with stubble.
--Thomas Vaughan (1650)

[Ged Byrne]

The key is in the clean desperation of business and synchronisation logic.

When two problems are combined the number of possible states explodes exponentially.

By separating the two problems this combinatorial explosion is avoided.

[Paul Tarvydas]

On 16-01-07 12:42 PM, Ged Byrne wrote:
> The key is in the clean desperation of business and synchronisation
> logic.
>
> When two problems are combined the number of possible states explodes
> exponentially.
>

Yes. Harel's seminal paper on StateCharts provides a wonderful answer
(as long as one rips out the concurrency part).

pt

[Ged Byrne]

Interesting.  Google found this, is it the right paper?

http://www.inf.ed.ac.uk/teaching/courses/seoc/2005_2006/resources/statecharts.pdf

Also Obviously that should have read 'clean separation.'   (shakes fist at phone's autocorrect)

[Paul Tarvydas]

On 16-01-08 04:16 AM, Ged Byrne wrote:
> Interesting. Google found this, is it the right paper?
>
> http://www.inf.ed.ac.uk/teaching/courses/seoc/2005_2006/resources/statecharts.pdf
>
> Also Obviously that should have read 'clean separation.' (shakes fist
> at phone's autocorrect)

Yes, that's the paper, although I've got a copy without all of the
horizontal gray lines (ask me if you want that one).

There's a follow-up paper (a year later?) that discusses the
micro-semantics of this notation, proving that it is quite unworkable
:-). (UML adopted the StateChart notation, whole-hog, without noticing
its deficiencies :-).

The secret is to remove all of the concurrency from the StateChart
notation, and to move it "up" into FBP notation. A two-layer notation -
hierarchical FBP at the top level, showing layers of concurrent
components, then putting hierarchical state machines in the leaf nodes
(and/or code in the leaf nodes - e.g. Paul M's stuff or "for-select
loops" in Go https://talks.golang.org/2013/advconc.slide#24 (Sameer
Ajmani)).

The concurrency in StateCharts is hidden under-the-hood - state machines
can react to other state machines by simply naming them - the equivalent
of a "come-from" statement (as opposed to "go-to"), possibly even worse
than call/return :-). FBP entirely solves this problem.

pt

[Paul Morrison]

No gray lines when I view it!  I seem to be using Adobe Acrobat Reader DC.

I definitely feel that this discussion about complexity zeroes in on how the FBP paradigm shift changes the programming experience.

[Paul Morrison]

I have fixed the JSFBP collate.js component on Github - https://github.com/jpaulm/jsfbp/blob/master/components/collate.js - to generate bracket IPs under control of multiple contiguous key fields; also added a test case, which I believe Alfredo was interested in: https://github.com/jpaulm/jsfbp/blob/master/examples/update.js .

Regards,

Paul M.

[Paul Morrison]

Just a quick note to say I have added 2 Collate diagrams to https://github.com/jpaulm/jsfbp/blob/master/README.md (see "Update" Networks). The second one contains enough info to generate a JSFBP running network - if there is any interest in my building the generator...?

[Samuel Lampa]

On Wednesday, December 23, 2015 at 10:07:46 PM UTC+1, Paul Tarvydas wrote:

My early impression of Go is that it is FBP.  With some added baubles.

Go treats channels as first-class objects.  Channels are bounded buffers.

Functions can be made concurrent by prefixing them with the keyword "go".

Functions can be turned into FBP components by passing the input channels and the output channels as arguments to the functions.

If of interest, I have played with a model inspired by Vladimir Sibirov's GoFlow, but trying to avoid reflection as far as possible, where structs are used as processes and struct fields populated with channels are used as ports. This allows to assign the same channel to multiple outputs of upstream tasks, and other nice things. For example:

type CsvToExcelConverter struct {

    InCSVFile chan *FileTarget

    OutExcelFile chan *FileTarget

}

And a constructor:

func NewCsvToExcelConverter() *CsvToExcelConverter {

  return &CsvToExcelConverter{

    InCsvFile: make(chan *FileTarget),

    OutExcelFile: make(chan *FileTarget),

  }

}

One also gets a nice plain Go syntax for connecting ports together: Just assign them, so that they point to the same channel:

csvToExcelConverter.InCSVFile = someProcessThatCreatesTheCsvFile.OutCSVFile

someOtherTask.In = csvToExcelConverter.OutExcelFile

... etc ...

Then, the actual execution code is stored in a Run() method tied to the struct.

func (proc *CsvToExcelConverter) Run() {

  defer close(p.OutExcelFile)

  proc.OutExcelFile <- // TODO: Some code to read from proc.InCsvFile and convert it to Excel here

}

Note that go-keyword is not used inside this function. This can instead be done from outside, in the program's main function:

go someProcessThatCreatesTheCsvFile.Run()
go csvToExcelConverter.Run()

go someOtherTask.Run()

theFinalTaskInTheChain.Run()

(A technical note: If having a chain of connected routines like this, the last one has to not have the go-keyword, since the program's main o-routine has to drive the chain).

Sorry if the examples above are a bit sketchy. This post provides a much better and clearer code example though:

http://bionics.it/posts/how-i-would-like-to-write-golang

I have also documented the pattern more in a bit detail in these two posts on GopherAcademy:

https://blog.gopheracademy.com/composable-pipelines-pattern/
(A bit too long backgorund about generators, and a pre-mature version of the pattern)

https://blog.gopheracademy.com/advent-2015/composable-pipelines-improvements/
(a follow up to the first post, with some key improvements that made the pattern more usable)

The pattern is right now the basis for a scientific batch (and poormans-stream) workflow system we develop: http://scipipe.org/

We have yet to see how it works in a production setting, but based on my dozen of experiments (in the examples folder), it looks very promising.

I'm very much interested in feedback, and general experimentation and brainstorming around this though. There are obviously many things to solve here still.

Our pattern is still very simplistic and does only implement certain FBP features, but I agree that Go seems to make very much sense for FBP.

One thing that we need which seems easier in Go than one of the existing FBP systems, and that might depart a little form classical FBP, is that we want to fire up new processes in concurrent lightweight threads dynamically. This is because we want to use them for handling the launching of work on an external HPC resource manager, and then waiting until the job has finished before proceeding the state of the process. This we want to do for as many jobs simultaneously as possible, and thus a fixed number of processes is not acceptable. Also, since these lightweight "job manager processes" won't do actual processing work themselves (they pass it off to the HPC system), we can happily fire up tons of them without worrying about bogging down the CPU.

 

I haven't yet looked at Rust, which also supports channels.

I heard they dropped channels from the core language. Also, they don't have the automatic M:N multiplexing of M number of lightweight threads to N number of CPU cores, which Go has, which I really don't want to do without.

Best

// Samuel

[Samuel Lampa]

On Friday, April 8, 2016 at 1:47:28 PM UTC+2, Samuel Lampa wrote:

I'm very much interested in feedback, and general experimentation and brainstorming around this though. There are obviously many things to solve here still.

Oops, that sentence became too short: What I meant to write is that we are very interested in further experimentation and brainstorming regarding the intersection of Go and FBP.

// Samuel

[Samuel Lampa]

On Tuesday, December 29, 2015 at 6:38:40 PM UTC+1, Paul Morrison wrote:

Finally, I will ask again: why do you all keep trying to implement FBP in various exotic languages, rather than just using JavaFBP or C#FBP (or CppFBP or JSFBP) for real applications and building up a body of knowledge which we can all benefit from?  This question does not apply to Paul T or Matt, as I know your reasons for choosing your particular languages!  

Paul, I saw this question now which I think might have went unanswered, and just wanted to add my 5c:

In my case, it is because of the following features of Go, in descending order:

1. The automatic multiplexing of lightweight threads (go-routines) onto physical cores, as described in the second paragraph in this section: https://golang.org/doc/effective_go.html#goroutines , which allows to be much less worried about firing up even short-lived go-routines when needed (for example firing up thousands of simple "bookkeeping" go-routines that would just sit and keep track of a cluster jobs as it queues and runs on the HPC system). 

2. The ease of use of the language, as it includes a garbage collector, and is much more "script-like" and terse than C++, Java, C# ant the like. This seems to make it easier to get people in my field of biology (who are far better application experts than coding experts) to use it. Most people would even find anything else than python or perl too hard. Go seems to be at least one of the easiest "next step" into the world of compiled languages.

3. The fact that many if not most of the FBP primitives are baked in to the language (things like buffered, block-on-full channels), which might enable a far simpler FBP system to be written, meaning less maintenance burden.

4. The static linking, making it really easy to deploy whole programs to cloud computers, clusters etc, without dependencies.

5. The rich standard library.

Best

// Samuel