Wikipedia Merge Proposal

[Ged Byrne]

Hi all, 

Having allowed myself to get carried away with abstract topics, I thought it might be a good idea to focus on something practical.

Right now there is a proposal to merge Dataflow Programming and Flow Based Programming on Wikipedia.

We are told that the following is required to remove the merge proposal tag:

If both dataflow programming and flow-based programming are eligible to be separate articles, then each article should explain what the differences between both terms are.

Example:

• In the FBP article, there should be statements such as: "FBP should not be confused with dataflow programming. In dataflow programming, this and that is different." .........
• And in the dataflow programming article, there should be statements such as: "Dataflow programming should not be confused with FBP. In FBP, this and that is different."

http://en.wikipedia.org/wiki/Talk:Flow-based_programming#Merge_discussion

I would like to make this update to Wikipedia and I'd like to get agreement.  The first thing to agree is whether or not Flow Based Programming is distinct from Data Flow Programming.

I would say that the key difference is that FBP is COMPONENT BASED while Data Flow programming is DECLARATIVE.  This places the two approaches into entirely different paradigms.

The root article for Component Based Software provides the following history:

The idea that software should be componentized - built from prefabricated components - first became prominent with Douglas McIlroy's address at the NATO conference onsoftware engineering in Garmisch, Germany, 1968, titled Mass Produced Software Components.^[2] The conference set out to counter the so-called software crisis. McIlroy's subsequent inclusion of pipes and filters into the Unix operating system was the first implementation of an infrastructure for this idea.

Brad Cox of Stepstone largely defined the modern concept of a software component.^[3]He called them Software ICs and set out to create an infrastructure and market for these components by inventing the Objective-C programming language. (He summarizes this view in his book Object-Oriented Programming - An Evolutionary Approach 1986.)

http://en.wikipedia.org/wiki/Component-based_software_engineering#History

Flow based programming follows this time line.  In the book Paul states that Brad Cox  had come to beleive that the FBP concepts should be implemented on top of Objective-C.

Brad Cox, who is the inventor of Objective-C and one of the acknowledged gurus of OO, has come to feel that OO alone is not adequate for building large systems. He came to the conclusion that FBP concepts should be implemented on top of Objective-C, and then could be used as building blocks for applications. Using a hardware analogy, he refers to Objective-C as "gate-level", and FBP as "chip-level". He had in fact already started experimenting with processes and data flows independently when he found out about our work and contacted me.

http://www.jpaulmorrison.com/fbp/oops.shtml

If it could be possible to publish the correspondence between Cox and Morrison then this could serve as a primary source establishing FBP within the Component Based paradigm to be referenced from the Wikipedia article.

The other key difference between Dataflow and FBP is the trigger for an processor to begin operating.  In Dataflow an operation begins when it's input conditions are true.

Operations consist of "black boxes" with inputs and outputs, all of which are always explicitly defined. They run as soon as all of their inputs become valid, as opposed to when the program encounters them. Whereas a traditional program essentially consists of a series of statements saying "do this, now do this", a dataflow program is more like a series of workers on an assembly line, who will do their assigned task as soon as the materials arrive. This is why dataflow languages are inherently parallel; the operations have no hidden state to keep track of, and the operations are all "ready" at the same time.

Dataflow programs are generally represented very differently inside the computer as well. A traditional program is just what it seems, a series of instructions that run one after the other. A dataflow program might be implemented as a hash table instead, with uniquely identified inputs as the keys, used to look up pointers to the instructions. When any operation completes, the program scans down the list of operations until it finds the first operation where all of the inputs are currently valid, and runs it. When that operation finishes it will typically put data into one or more outputs, thereby making some other operation become valid.

http://en.wikipedia.org/wiki/Dataflow_programming

(emphasis added)

FBP is not implemented in this way.  It is implemented as a set of components connected by bounded queues.  The processors are always running, and will process data when it becomes present in the input queue.

Processes are connected by means of FIFO (first-in, first-out) queues or connections which can hold up to some maximum number of IPs (called a queue's "capacity"). Processes are connected by means of FIFO (first-in, first-out) queues or connections which can hold up to some maximum number of IPs (called a queue's "capacity"). 

http://www.jpaulmorrison.com/fbp/concepts.shtml

The reason for the similarities between the two concepts might be because they both make use of an assembly line metaphor:

a dataflow program is more like a series of workers on an 

assembly line, who will do their assigned task as soon as the materials arrive.

http://en.wikipedia.org/wiki/Dataflow_programming

However, the nature of that assembly line, or connection between processes, is different.  Dataflow uses a logical connection while FBP uses a physical connection.  

In Dataflow programming the connection between two processes is logical.  If Process A is connected to Process B it is because A leaves the data in a state that matches the input criteria for A.

In Flow Based Programming the connection between two processes is physical.  They are connected by a bounded FIFO queue.  Process A will be Connected to Process B by a connection.  When A pushes an IP to the connection it will be processed by B regardless of state.

What are your thoughts?

Regards, 

Ged

[Paul Morrison]

Hi Ged,

Great stuff!  I totally agree that we need to highlight the differences between dataflow and FBP, and hopefully put the merge proposal to rest.  I didn't realize that it was still on the books :-)  As you know, anyone can update Wikipedia, so by all means go ahead - at least as far as I am concerned.

I think you have really brought the differences to the fore - but I might also add that a part of the difference is between the idea of "valid input states" and IPs:   it seems to me (I may be wrong) that "valid input states" would tend to be simple values, whereas IPs are structured chunks of data with a defined lifetime.  Would you agree? 

--
http://jpaulmorrison.blogspot.com/

[Tom Young]

Hi Ged,

It is my understanding that FBP grew out of DataFlow Design  and that DataFlow Programming was entirely separate, hardware oriented, and not based on software components. 

So, I would argue that FBP is entirely separate from DataFlow programming, but not quite in the way stated.

        Cheers,

                 -Tom

--
"...the shell syntax required to initiate a coprocess and connect its input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

[Ged Byrne]

Hi Paul,

I've just remembered your comments on Linda, which sum up the differences: http://www.jpaulmorrison.com/fbp/concepts.shtml

The idea that IPs are mutable with a dynamic lifetime is important.

In Dataflow the application state is stored in a set of tuples.  If the tuple matches the input for a process it is consumed.  If the processes input is defined as destructive then the tuplie is removed from the state's tuple-space.  If it is non-destructive it remains.

When a process completes then it may add one or more tuples to the tuple space that become available for others to consume. 

In dataflow there exists no relationship between the tuples consumed as input and those produced as output.  Tuples are immutable and can only be created or consumed.

IPs are processed by a process but not consumed.  There exists a clear relationship between an IP that is received and sent by a process.

It is possible for a process to destroy one IP and create another but it is not mandatory.

Would you agree?

 

I think you have really brought the differences to the fore - but I might also add that a part of the difference is between the idea of "valid input states" and IPs:   it seems to me (I may be wrong) that "valid input states" would tend to be simple values, whereas IPs are structured chunks of data with a defined lifetime.  Would you agree? 

Regards, 

Ged 

[Tom Young]

Ged,

It seems to me that the  Wikipedia article titled "Dataflow Programming"  should have been titled
"Dataflow Design".   

  -Tom

On Tue, Apr 10, 2012 at 10:02 AM, Ged Byrne <ged....@gmail.com> wrote:

[Vladimir Sibirov]

Hi Ged,

I agree with you in general. There are a few notes though.

The boundaries of Dataflow programming itself are not distinct. For example, VHDL is included in the list but as far as I know VHDL it is component-based. Is COMPONENT BASED orthogonal to DECLARATIVE? I'm not sure, but if it is, then VHDL obviously belongs to FBP instead of Dataflow.

There is Dataflow in-large and Dataflow in-small. Same is true for FBP. Some people claim their specific model is Dataflow and vice versa Dataflow is their specific model. This results into a lot of misunderstanding and other people start thinking that Dataflow is that specific model. Or, the opposite, Dataflow is beyond that specific model, a wider class of models. It's hard for me to judge which Dataflow programming is the Dataflow programming: in-large or in-small. But it would be a lot more clear if everyone at wikipedia understood that there is such a difference.

It may be my fault that I often used the term "FBP" in-large (a wider class of models) rather than in-small (Paul's model). However, now that you have an image of all 4 actual terms, here are a few statements of mine:

1) "FBP in-large" may be identical to "Dataflow in-large" or may be a subset of it;

2) "FBP in-small" may be a subset of "Dataflow in-large";

3) "FBP in-small" is very different from "Dataflow in-small".

Now I assume that the actual discussion is "FBP in-small vs. Dataflow programming in-small". Then, the key differences in my opinion are:

1) FBP is component-based, Dataflow is declarative (agree with Ged).

2) FBP has 2 levels of abstraction: coordination/structure level (visual) which is a graph of components and connections and behavior level (textual) which is imperative code for component operations.

3) FBP uses bounded buffers for communication, Dataflow uses unbounded call chains or tuple spaces (another Ged's point).

4) FBP deals with bounded non-determinism, Dataflow deals with unbounded non-determinism.

Just my 2 cents.

Regards,

Vladimir

2012/4/10 Ged Byrne <ged....@gmail.com>

[Ged Byrne]

Vladimir,

Thanks for your notes, very useful.

I've been trying to get a strict definition of dataflow as well, and failed.  I'm trying to base the comparison on the Wikipedia content, since it seems to be the only source available.

It would make for an interesting project to try and map out this space properly.  I wish I were an academic so I could devote my time to it, but I'm not.

I'll try to put together some succinct to add to the article that takes into account all the views expressed here.

Regards, 

Ged

On 10 April 2012 15:45, Vladimir Sibirov <trust...@kodigy.com> wrote:

Hi Ged,

I agree with you in general. There are a few notes though.

The boundaries of Dataflow programming itself are not distinct. For example, VHDL is included in the list but as far as I know VHDL it is component-based. Is COMPONENT BASED orthogonal to DECLARATIVE? I'm not sure, but if it is, then VHDL obviously belongs to FBP instead of Dataflow.

[...]

[Ron Lewis]

For Wikipedia, probably good to quote sources to support the explanation of the differences. So, I suppose Paul's book is one good source. And a link to this discussion thread might be another source.

[Ged Byrne]

Vladmir,

Do you know where I can get more detail regarding the unbounded call chains approach to data flow?

Thanks,

Ged

On 10 April 2012 15:45, Vladimir Sibirov <trust...@kodigy.com> wrote:

[Vladimir Sibirov]

Sorry, this mostly applies to functional programming, not dataflow programming. In functional languages messages can be passed through a (theoretically) infinite sequence of recursive function calls. Some traces can be found in Oz though: http://en.wikipedia.org/wiki/Oz_(programming_language)#Dataflow_variables_and_declarative_concurrency

2012/4/10 Ged Byrne <ged....@gmail.com>

[Brad Cox]

Hi. I just joined, triggered by Morrison's email. If my comments to him shed any light, feel free to add them here (Paul).

The main thought in those comments is that FBP immediately struck me as sensible, practical, useful and essential to moving software ahead (card-level granules). So I struck out on that trail and never explored the other, except to read a few papers about Linda and wonder WTF? Never really got that at all.

[Ged Byrne]

Hi Brad,

Thanks for joining our discussion. Your opinion would be useful
regarding FBP as a Component Based Software Engineering approach.
Since wikipedia credits you as largely defining the modern concept of
a software component with your description of Software ICs you are
what they call a "established expert on the topic" and your comments
within this group will be acceptable as a self published source.

How do you feel that the ideas of Flow Based Programming contribute to
the field of software components. You mention "card-level granules."
Could you explain what that means?

Thanks,


Ged

[Brad Cox]

Thanks for the compliments!

Paul's paper, referenced earlier, is exactly what I mean. The Sequence and Filter components are two examples of card-level components that eat some slice of the elephant, in exactly the way that workstations on an assembly line tackle some well-defined part of building a card.

Card-level components are invariably composed of smaller level parts which I call chip-level and gate-level. Chip-level components are Objective-C style objects, which are then composed of even smaller gate-level components (C with Objective C, C++).

I've often described the difference as: C++ is a silicon fab line. Big, expensive, complicated but unsurpassed for buiilding gate-level pieces. Objective-C is a soldering iron; small, cheap, and unsurpassed for glueing together the result to build larger level granules.

Data-flow components continue the analogy one level higher. Ideal for building reliable parallel systems from lower-level parts.

I've always found it puzzling that this industry never bought in to the dichotomy, if not the terminology, used above. The discussion derails as soon as the word granularity comes up. To me (and real engineers) its just common sense.

--
Cell: 703-594-1883
Blog: http://bradjcox.blogspot.com
Web: http://virtualschool.edu
Manassas VA 20111

[Brad Cox]

I meant car not card, last word of first paragaph.

[Brad Cox]

I've always found it puzzling that this industry never bought in to the dichotomy, if not the terminology, used above. The discussion derails as soon as the word granularity comes up. To me (and real engineers) its just common sense.

Can't let this go at that. What really happens is that people actively resist or misunderstand the very notion of many levels of granularity, and steer the conversation in the direction of which language is better (completely orthogonal debate). A good example of that is the paper I responded to earlier (whose name I've forgotten). The idea there was, a long-time C++ programmer decided that functional languages are "better" (for parallelism among other things), and predicts the replacement of C++ by some functional alternative. 

The question is, why do software engineering minds slip so easily in a direction that completely misses the point, when hardware engineers consider granularity as plain old common sense. Everything is composed of something smaller. 

[Brad Cox]

I experimented awhile with another analogy in my blog: http://bradjcox.blogspot.com/2008/07/mud-brick-business.html.

This was triggered by the growth in humongous does-all SOA middleware (WS-Security, Metro, Axis2), trying to shift attention to the idea that SOA cities could be better made of bricks rather than armies of contractors building cities from mud and straw. SoaKit was meant to be that kit of bricks.

Must confess, it led to diddley squat, like every attempt to shift the debate away from my language is better than yours. Sigh.

[Paul Morrison]

On 10/04/2012 10:38 AM, Ged Byrne wrote:
> It is possible for a process to destroy one IP and create another but
> it is not mandatory.
>
> Would you agree?

Yes!

Now suppose I have incoming IPs, and we have a process which modifies a
particular field in each IP (e.g. the Assign component), is it better to
destroy the incoming IP and create a new one with the same structure as
the incoming one - or can we just modify the incoming one and send it
on? Or doesn't it matter? From my point of view, I find the
[im]mutability issue confusing :-) Does this have a bearing on the
dataflow vs. (?) FBP issue?

If each process is a separate microprocessor, does this change the answer?

[Dan]

Ged Byrne: “I would say that the key difference is that FBP is COMPONENT BASED while Data Flow programming is DECLARATIVE”

This is not correct. We can not compare “component based” with “declarative”. These are two different aspects but you can compare “declarative” with “imperative” for instance. “Component based” means: the system is based on components. It highlights the aspect of composition and nothing more.

FBP is declarative and component based in the same time. This is the essence.

I don't see any difference between Data Flow Programming and Flow Based Programming. I have said this before: unfortunately the programmers associate different meanings to these common words: flow, data, components.

The eventual differences are related to implementations. By implementation I mean how the declarative program (FBP or DataFlow, whatever) is translated / compiled into the executable program for a specific platform / microprocessor.

I am really against the idea that the parameters / data / IPs represent an aspect that makes the difference between FBP and DataFlow. We have had this discussion before. An IP is mutable. You can follow this simple rule: what moves (IP in our case) can be changed but in a single place at one time. It can change again in another place at another time. “Place” is a component. The components that modify the IPs (other components) don't move.

The order is strictly given by the connections: TIME is ORDER and ORDER is SPACE. Voilà: space-time continuum
:-). The sequence (time order we are used with in the imperative programming) in FBP or DataFlow is given by ordering the processing of the IPs in space, moving the IPs from one place (component) to another.

I have said before that in my opinion an IP could be a component. The components are first-class citizens. In fact it is the only way a “polymorphic aspect” (as it is defined in OOP) could manifest.

Vladimir Sibirov: “FBP has 2 levels of abstraction: coordination/structure level (visual) which is a graph of components and connections and behavior level (textual) which is imperative code for component operations”.

This is interesting. At first glance this assertions makes one to think that the border between “declarative” and “imperative” in FBP is made by components' membrane: what is outside is “declarative” and what is inside should be imperative at some more basic level. This could be misleading.

It is true that a FBP program could be just that (i.e. more coordination and less processing/computation) but a FBP program could be declarative all the way down to the level with the finest granularity i.e. atomic components (basic operations).

In the end all the FBP (declarative) program should be translated / compiled into an executable. The result of this translation could be very different depending on the target microprocessor or the number of processors available. The FBP program should be stretched / map and partitioned in the translation process (compilation) on a single or a limited number of threads. A thread is an ordered set of microprocessor instructions that feed the microprocessor. The executable remains a program that respects the same data flow principles but this time the code is data (bytecode) that feeds a single (or a limited set of) real microprocessor(s). In other words you start with a network of interconnected virtual processors (components) and end up with a single or limited number of real processor(s). This translation / compilation is not trivial but it could be done all the way from the FBP declarative program (source code) down to the executable.

Vladimir Sibirov: “FBP uses bounded buffers for communication, Dataflow uses unbounded call chains or tuple spaces”

This is the implementation detail I was talking about. I don't see it as an important aspect that makes a paradigm difference. How do you implement the following diamond shaped diagram?

Result z = D(B(A(a).b), C((A(a).c));

All the components / processors (A,B,C and D) can be executed in parallel but x and y need synchronization or at least a mechanism that assures they are triggered by the same IP that originated them i.e. the input of A that is a. If all these components are on the same computer / process / thread everything seems to be simple but imagine that A, B, C and D are complex processes that executes on different computers. Streams can break, the order is an issue. The FBP program needs to be resilient to errors. a,b,c,x,y and z could be streams but a specific x element is linked with a specific y element. Is FBP and DataFlow different with this respect? How do you assure the consistency in this case?

By the way, is out there anybody that can explain to us what a MONAD is, explaining it in plain English and with a little bit of FBP flavor? Of course I have read some documentation about monads in functional languages but I am eager to know how a FBP programmer thinks about monads.

Regards,

Dan

[Vladimir Sibirov]

Hi Dan,

Your message is precise illustration of understanding FBP in-large and it proves the statement that FBP in-large and Dataflow programming in-large is the same thing. As I said before, this doesn't apply to the same terms used in-small.

Probably the main question that should be answered is: should we bind the term "FBP" specifically to the model described by JPM in his book and implemented in JavaFBP/C#FBP or should we rather use it to refer to a wider class of concurrent component-based systems? The second question is: should we use the term "Dataflow design" for such a wider class of systems instead of "FBP" then?

Regards,

Vladimir

2012/4/11 Dan <dpol...@gmail.com>

[Ged Byrne]

Hi Dan,

I'm sure we would both agree if we both shared the same understanding of what Dataflow was.  Unfortunately we don't, since we there isn't a single authority we can share.  Vladmir's concept of in-the-large and in-the-small is useful here.

Given a lack of authoritative definitions, we can only look at the specific problem: should the Dataflow programming and Flow Based Programming articles on Wikipedia be merged?

I'm not talking about anything in-the-large here, I'm talking about the specific definitions provided by Wikipedia.  Given that specific definition I believe that my statement are true, since I have supported them with quotations from that source.

Can you agree on these points, purely within the context of Wikipedia?

 COMPONENT BASED is orthogonal to either DECLARATIVE or IMPERATIVE.  Flow Based Programming is in the category of COMPONENT BASED and this is correct.  Components can be implemented in either a declarative or imperative style, it does not matter.  

You say that FBP is declarative, and I disagree with these.  Do you consider JavaFBP to be declarative?  It is implemented in Java without extensions, and Java is an imperative language.  I do not see how it could be considere declarative.  Do you consider it possible to write declarative code in a imperative language?

Wikipedia classifies Dataflow Programming as DECLARATIVE.  Flow Based Programming is not exclusively declarative and therefore does not belong in this category.

Therefore if Flow Based Programming was to be merged with Dataflow Programming the classification must be changed, otherwise it will be inaccurate.  Flow Based Programming would be incorrectly classified.

If the two are the same, then one of the possible must be true.  They cannot all be true:

1. JavaFBP is not an example of Flow Based Programming.  Flow Based Programming is DECLARATIVE and JavaFBP is IMPERATIVE.

2. Dataflow Programming is not DECLARATIVE and it has been incorrectly classified in Wikipedia.

If Flow Based Programming and Dataflow Programming are to be considered synonymous then he above contradictions will need to be resolved.  Either the Dataflow Programming article must be reclassified or our understanding be revised to no longer include JavaFBP.

Is there a flaw in my logic here?  

Regards, 

Ged

[Raoul Duke]

On Wed, Apr 11, 2012 at 1:22 PM, Ged Byrne <ged....@gmail.com> wrote:
> You say that FBP is declarative, and I disagree with these.  Do you consider JavaFBP to be declarative?  It is implemented in Java without extensions, and Java is an imperative language.  I do not see how it could be considere declarative.  Do you consider it possible to write declarative code in a imperative language?

of course it is.

everything ends up being implemented in C anyway, at the end of the
day, under the covers ;-)

[Vladimir Sibirov]

I think Dan wanted to say that at the component graph level FBP programs are declarative. And according to Wikipedia's definition of http://en.wikipedia.org/wiki/Declarative_programming, he is right. Though, FBP isn't restricted to declarative programming alone.

2012/4/12 Raoul Duke <rao...@gmail.com>

[Paul Morrison]

On 11/04/2012 2:08 PM, Dan wrote:

All the components / processors (A,B,C and D) can be executed in parallel but x and y need synchronization or at least a mechanism that assures they are triggered by the same IP that originated them i.e. the input of A that is a. If all these components are on the same computer / process / thread everything seems to be simple...

Hi Dan,

Just wondered why you picked this particular topology for your example - this is a perfect example of a deadlock-prone network.  I would say it is anything but simple: if you can guarantee that A emits the same number of IPs to B and C, and that they in turn emit the same number of IPs on the connections to D, then you avoid a deadlock.  You have to pay attention to this even when you are sharing memory.  I confess I don't see what it means to say that you can run that network in one process or thread - unless you are using the arrows to mean something different from standard FBP usage.  I certainly agree the situation becomes even more complex if the arrows are connections between machines, but it's pretty complex even on one machine...

[Ged Byrne]

Raoul,

I know you have a smiley, but I'm going to take you literally.

Could you point to the properties of JavaFBP that make it declarative.  Each line is a command that instructs the JVM to create new object instances.  It declares component objects and links them together with connections that are bounded queues.  All of this is specifying exactly what is to be done, not the computation that is to be achieved.  The code has side effects and does not clearly correspond to computational logic.

JavaFBP is written in Java that compiles into Bytecode that is translated into machine code by the Hotspot VM.  No C is generated at any point.

Regards, 

Ged

[Raoul Duke]

On Wed, Apr 11, 2012 at 1:50 PM, Ged Byrne <ged....@gmail.com> wrote:
> Could you point to the properties of JavaFBP that make it declarative.

it all depends on what level you are talking about. compilers all the
way down. except for the interpreters. oh but those are mostly jits
now anyway. oy veh, the humanity.

> JavaFBP is written in Java that compiles into Bytecode that is translated
> into machine code by the Hotspot VM.  No C is generated at any point.

but hotspot is written in C, or at least used to be.

one can pretty much layer any turing complete language paradim on top
of or underneath any other turing complete language paradigm.

if i as the end user am using
http://en.wikipedia.org/wiki/DataRush_Technology or
https://github.com/trifork/erjang/wiki or http://clojure.org/ or
http://openquark.org/Welcome.html or http://shenlanguage.org/ running
on ABCL on a JVM, what am i doing?

sincerely.

[Ged Byrne]

Roaul,

How is the language used to implement the JVM in any way relevant?

You are saying that Java is a declarative language because Clojure compiles to the JVM?

This is just getting silly.

Regards, 

Ged

[Ged Byrne]

Vladimir,

But the code does not declare the component graph, it describes an imperative list of instructions that constructs the component graph. None of the code in JavaFBP or Java code that uses it is declarative.  It is all imperative.  

If the DrawFBP graph was to be translated at runtime, then that could be declarative.  The Java code generated by JavaFBP is always imperative.

Regards, 

Ged

[Raoul Duke]

On Wed, Apr 11, 2012 at 2:04 PM, Ged Byrne <ged....@gmail.com> wrote:
> You are saying that Java is a declarative language because Clojure compiles
> to the JVM?

it is quite right that Java itself doesn't change.

but it is not quite right to say that the layer on top cannot be
called anything other than imperative.

sincerely.

[Brad Cox]

Could you point to the properties of JavaFBP that make it declarative.  Each line is a command that instructs the JVM to create new object instances.  It declares component objects and links them together with connections that are bounded queues.  All of this is specifying exactly what is to be done, not the computation that is to be achieved.  The code has side effects and does not clearly correspond to computational logic.

JavaFBP is written in Java that compiles into Bytecode that is translated into machine code by the Hotspot VM.  No C is generated at any point.

It is be useful to take the term "component-based" dead-seriously. What's inside a component is immaterial to its users; its an impenetrable box. Real ones only expose knobs and dials (configuration settings) and its inputs and outputs. Seems to me that the notation used to configure it (the "each line is a command" above) doesn't take away from the declarative nature of the pipes between them. BTW I prefer to think of connecting the pipes as separate from configuring the box's internals. Then there's "the code" in your last line; it isn't clear if that means the configuration code or the code used to implement the box's internals, which in my world is some lower-level language like Java or C.

I've not looked at JavaFBP lately and am not sure if Paul stuck to my programming-as-configuring closed boxes model. Suspect so knowing his proclivities. And mine. ;)

Of course there's a slippery slope there; those configuration settings could easily grow into a full-featured programming language, which would undercut the very advantages of componentization. In my opinion of course.

[John Cowan]

Ged Byrne scripsit:

> Could you point to the properties of JavaFBP that make it declarative.

JavaFBP mini-language has both declarative ("These are the components,
this is how they are wired up") and imperative ("Create these components,
wire them up like this") readings. This is normal for code written
in a functional language, or as in this case a functional subset of a
non-functional language.

Since the mini-language is just Java, it's perfectly possible to create
components on the fly, or change the writing on the fly. But normally
FBP programmers don't, which is why the mini-language is effectively
functional/declarative.

(The comment below is chosen randomly, but it's apropos.)

--
Man has no body distinct from his soul, John Cowan
for that called body is a portion of the soul co...@ccil.org
discerned by the five senses, http://www.ccil.org/~cowan
the chief inlets of the soul in this age. --William Blake

[Ged Byrne]

John,

So all FBP programs are declarative, regardless of language?

The same is true of any program that constructs a user interface, such as a swing program?

The distinction between imperative and declarative programmer is therefore meaningless?

[John Cowan]

Ged Byrne scripsit:

> So all FBP programs are declarative, regardless of language?

That depends on the design of the FBP package. In JavaFBP and C#FBP,
the *normal* use of the mini-language features is functional no
mutations, no reassignment of variables), so both imperative and
declarative readings exist.

> The same is true of any program that constructs a user interface, such
> as a swing program?

It could be. I'm not familiar with the details of using Swing.

> The distinction between imperative and declarative programmer is
> therefore meaningless?

There are programs that have only imperative readings, not declarative
ones. Anything involving assignment (as opposed to initialization),
or that involves changing the state of an object (as opposed to just
setting it up), has only an imperative reading. If a text has only a
declarative reading, it's markup rather than code.

--
I marvel at the creature: so secret and John Cowan
so sly as he is, to come sporting in the pool co...@ccil.org
before our very window. Does he think that http://www.ccil.org/~cowan
Men sleep without watch all night?

[Paul Morrison]

On 11/04/2012 5:14 PM, Brad Cox wrote:

I've not looked at JavaFBP lately and am not sure if Paul stuck to my programming-as-configuring closed boxes model. Suspect so knowing his proclivities. And mine. ;)

I think I did - actually John Cowan did (the author of JavaFBP before I started tinkering with it)...  

The very first FBP implementation, AMPS, used mainframe Assembler macros to specify the network connections, which _felt_declarative.  However, the macros expanded to executable code, which of course had to be executed before the network was actually run.   JavaFBP and C#FBP essentially do the same thing.   DFDM, the 2nd mainframe implementation, used Wayne Stevens' notation (alluded to before) which is purely declarative, and had to be _processed_ to generate the network connections. 

The problem is that using executable code directly to generate the connections allows the user to add in additional statements which need to be run prior to network execution - this is not necessarily a bad thing if used sparingly.   DFDM did not have that problem (or feature) - but of course, for the occasional situation where it would have been useful this capability was not available!   I have to say that "declarative" for me is a matter of intent, rather than implementation technology. :-)

[Brad Cox]

I'm using two separate steps for configuring a flow-based system for filtering twitter feeds, in which pipes convey streams of JSON records.

The configuring step for a throw-out-uninteresting-tweets component involves providing some JSON that specifies what fields to search in, and for what. The JSON comes from a form in an AJAX app responsible for configuring, assembling and viewing the results of a filter pipeline.

The connecting step is separate, and originates from drag and drop actions to "draw" connections between components. I'm a bit puzzled as to why assembler macros were needed to do what I think of as (and implement as) workplace.connect(upstreamComponentID, downstreamComponentID).

The execution step is a start button that kicks the pipeline into operating. Typically tweets flow from a ReadTwitter component, thru a pipeline of filter-not-interesting-tweets (discard not-fish) and map-json-record components (trim fins and scales) and eventually wind up routed to a WriteMongo component that saves them for later review. 

That's just an example; there are more components (Map, Reduce, Fork, Join, etc) that can be configured and assembled to do almost anything that can be expressed as a stream of JSON records.

Its just as straightforward as I said. No semantic minefields in it at all, like the difference between imperative and declarative. Just JSON records flowing through pipes and components no more complicated than you might find at Lowes, each with a few knobs to twiddle until you get what you want.

Java SynchronousQueue handles the concurrency. I've also tinkered briefly with a Node.js version that uses the Node.js notion of call-backs in lieu of threads. I reverted to threads to coexist with legacy stuff like Twitter and MongoDB. 

[John Cowan]

Brad Cox scripsit:

> The connecting step is separate, and originates from drag and drop actions
> to "draw" connections between components. I'm a bit puzzled as to why
> assembler macros were needed to do what I think of as (and implement as)
> workplace.connect(upstreamComponentID, downstreamComponentID).

The whole AMPS system was written in (System 360) assembler. The macros
made the mini-language more readable.

--
Normally I can handle panic attacks on my own; John Cowan <co...@ccil.org>
but panic is, at the moment, a way of life. http://www.ccil.org/~cowan
--Joseph Zitt

[Paul Morrison]

On 11/04/2012 7:40 PM, Brad Cox wrote:
> The connecting step is separate, and originates from drag and drop
> actions to "draw" connections between components. I'm a bit puzzled as
> to why assembler macros were needed to do what I think of as (and
> implement as) workplace.connect(upstreamComponentID,
> downstreamComponentID).

Hi Brad,

Maybe I wasn't clear - AMPS was written using mainframe Assembler
language and macros - no high level languages supported at all.
Remember we are talking about the late '60s, early '70s! The (what we
would now call FBP) networks were defined using a few different macros,
the most heavily used one being the one that specified a process and the
connections that attached to it. As you would expect, this macro
generated a process control block and zero or more connection control
blocks. The last macro in the program essentially said "run it". HTH !

--
http://jpaulmorrison.blogspot.com/

[Paul Morrison]

On 11/04/2012 8:56 PM, John Cowan wrote:
> Brad Cox scripsit:
>
>> The connecting step is separate, and originates from drag and drop actions
>> to "draw" connections between components. I'm a bit puzzled as to why
>> assembler macros were needed to do what I think of as (and implement as)
>> workplace.connect(upstreamComponentID, downstreamComponentID).
> The whole AMPS system was written in (System 360) assembler. The macros
> made the mini-language more readable.
>

Thanks, John! I should have said that!

[Dan]

Hi Ged,

Ged: "COMPONENT BASED is orthogonal to either DECLARATIVE or IMPERATIVE.  Flow Based Programming is in the category of COMPONENT BASED and this is correct.  Components can be implemented in either a declarative or imperative style, it does not matter. "

I fully agree: component based is orthogonal to either declarative or imperative and the components could be implemented in either style.

Ged: "You say that FBP is declarative, and I disagree with these.  Do you consider JavaFBP to be declarative?  It is implemented in Java without extensions, and Java is an imperative language.  I do not see how it could be considere declarative.  Do you consider it possible to write declarative code in a imperative language?"
JavaFBP is not declarative indeed. FBP is declarative but it leaves the freedom to build components at a lower level of finer granularity in imperative style but I say that FBP promotes declarative style all the way down to the atomic level. That's why I am saying FBP is declarative. As long as you implement subcomponents in imperative style (nobody stops you to do that) it means that part is not FBP. The beauty of FBP is the fact it allows that and you can mix them because FBP is doing the coordination. So it is not a contradiction at all. The main question was: what is FBP part from a system that has also components built in imperative style? The answer is: not all of it, that's why FBP remains declarative. The fact you can mix them it doesn't make FBP more imperative.

DECLARATIVE versus IMPERATIVE is decided by WHO is doing the action. Take for instance a coffee machine versus a person that is doing a coffee manually. In the first case you have a machine with compartments, pipes, raw materials (coffee, sugar, water) etc. All the subcomponents exists and work at once, in parallel. If you are about to make a model of that machine then you describe it in a declarative manner like the FBP network of components. In the second case you have a list of instructions that a person follows in order to make a coffee. This is the difference between declarative and imperative. In declarative style you describe the real world and the system behind it brings the building materials (the matter). In imperative style you have a list of instructions for a single entity that does them all. For some it looks the same but it isn't. We know that in the end everything ends up in a microprocessor so we translate all the declarative code to a list of instructions (and data) but a programmer should describe the real worlds he or she is trying to simulate. After all this what a computer programmer should do. This is the logic of a computer program.
I have to admit that the difference between declarative and imperative is not obvious at all. I have asked programmers to explain to me exactly what it is and many of them failed. I hope I was just a little bit better :-).

All the best,
Dan

[Brad Cox]

Ahhh... ;)

Ask me to show you the spare byte I keep in case I need a little more. About the size of a hard-cover book. A rack with 8 tubes from an IBM604.  

[Ged Byrne]

All,

Clearly this isn't working.  I'm a strong believer that the right questions will get the right answers.  Here we are looking for clear, precise answers with unambiguous meaning.  We should be able to compile some clear, easy to understand statements that everybody can agree with.

Since this isn't happening I think the problem must be with the question.  "Are Dataflow Programming and Flow Based Programming the same thing?" Instead of a clear answer it leads to the question "What exactly do you mean by Dataflow and Flow Based Programming."  Vladimir has given the best answer, with regards to "in-the-large" and "in-the-small" responses.

What, then, is the question that we should be asking?

Here are some candidates:

• Is FBP a programming paradigm, like OO or Dataflow?
• Is FBP an architectural style, like Rest or SOA?
• Is Flow Based Programming based on a single paradigm or multiple paradigm?

I'm beginning to form the opinion that FBP is an architectural style that layers applications in terms of granularity rather than function.  Different programming paradigms are applied at different layers.

I'm not sure if "architectural style" is right, though.  It appears to be the "Uniform Pipe and Filter" style described as a Dataflow architectural style by fielding: http://www.ics.uci.edu/~fielding/pubs/dissertation/net_arch_styles.htm#sec_3_2

What are your thoughts?

Regards, 

Ged

[Brad Cox]

That's a promising idea. 

Its not a new programming paradigm, more of a way to get computers to do what we want without programming as normally defined. By pushing the programming job off on component builders so we can do something more productive ;)

The architectural style idea seems closest although I'm not 100% with that. In fact SoaKit arose out of the notion that SOA was a larger-yet granularity in which SOA objects were like cities, composed of SoaKit (FBP) objects like buildings, composed of Java objects like bricks, composed of even lower (card- and gate-level) objects like cement and gravel. SoaKit was a way of building secure SOA objects (buildings) out of a kit of WS-Security data flow components. Maybe that dog will hunt.

Re single or multiple paradigms, maybe, but that'll open a whole new debate about what you mean by "paradigm". If it means integration style, there are function calls for gate-level objects, message-passing for chip-level, synchronous queues for FBP objects, SOAP messages for SOA objects, etc. 

[Brad Cox]

Suggestion: we've been looking at this from the back side of the glass, with only computers in the frame and no people. As I said in my first book that OO is really a new way for *people* to work together, and DFP is just a (new kind of) objects one level higher that I described in that book.

Can't speak for FBP vs DFP since I've only focused on one. But I choose that one because it allowed *people* (component producers and consumers) to cooperate (share each other's work) with fewer hinderances than the alternatives we've had in the past. 

In particular, unlike message-passing and function calls where the callee shares the callers control thread, a data flow component encapsulates its own thread, which simplifies the producer-consumer interface by one notch.

On Thu, Apr 12, 2012 at 7:36 AM, Ged Byrne <ged....@gmail.com> wrote:

[Ged Byrne]

Brad,

Perhaps we are talking about Construction Styles.  I quick google led to the following blog post about construction styles.  It looks to be a good analogy:

Building construction has evolved since time began as shelter is one of our basic needs. We need to protect ourselves from the heat, cold, rain and other weather elements. What has happened over time is that we built our homes based on local building techniques and availability of materials. Maybe we started with stone or pieces of wood and that grew into buildings of stone and mortar or later timberframes. That was the method for a long period of time. 

http://www.thebuildingblox.com/2012/03/construction-styles-and-methods-change.html

Perhaps "paradigm" is too abstract an idea.  Instead we can talk about Architectural Styles, since fielding does provide us with an authoritative definition of Dataflow from that perspective.

This gives us Flow Based Programming as a Construction Style that complements the Pipe and Filter Style of Dataflow Architecture.  It describes the standard for the component blocks and the connections used to configure them.

Regards, 

Ged

[Brad Cox]

On Thu, Apr 12, 2012 at 8:11 AM, Ged Byrne <ged....@gmail.com> wrote:

This gives us Flow Based Programming as a Construction Style that complements the Pipe and Filter Style of Dataflow Architecture.  It describes the standard for the component blocks and the connections used to configure them.

Yes. But instead of complements (another debate bait), I suggest "extends the Unix pipe/filter architectural style to support streams of any data type instead of just streams of bytes." I'd have written Unix/Fielding, but I've not read his views on that.

[Ged Byrne]

So Flow Based Programming is a style of software construction that  extends the Unix pipe/filter architectural style to support streams of any data type instead of just streams of bytes.  It provides a standard for the component blocks and the connections used to configure them.

Does anybody have any objections?

[Vladimir Sibirov]

Seems like we finally got somewhere :)

2012/4/12 Ged Byrne <ged....@gmail.com>

[Paul Morrison]

Could we add something to the effect that the network topology can also be more complex - multiple inputs and outputs, loops, and hierarchies (subnets)...?

[Paul Morrison]

Dan's coffee machine analogy makes me think that there are two ways of specifying how to assemble a piece of furniture:  you can either express the instructions procedurally - i.e.

- take piece 'a', attach it to piece 'b' using tab 1 on 'a' and slot 1 on 'b',
- now take piece 'c', and attach it to piece 'b' using tab 1 on 'c', and slot 2 on 'b', etc.

Many people find that irritating - and it means that it is hard to recover if something unexpected happens.

A better way (at least for me) would  be to put up an exploded diagram of the parts with dotted lines (perhaps) showing how tabs and slots relate to each other.  You could also make suggestions about the order of assembly, but often it won't be necessary.  In my book I talk about recipes and following driving directions - the other main procedural activities in people's lives (Chap. 28 in the 2nd edition).

So for me, FBP is declarative _at the network level_, irrespective of the implementation technique, and procedural within components - is this a problem?!

--
http://jpaulmorrison.blogspot.com/

[Brad Cox]

No objections, perhaps improvements, "a standard for connecting component blocks" via streams of any data type.

Maybe too far, perhaps mention that the data types can contain object references so streams contents can be lists or trees, unlike Unix byte streams.

[Ron Lewis]

On Thursday, April 12, 2012 7:36:33 AM UTC-4, Ged Byrne wrote:

We should be able to compile some clear, easy to understand statements that everybody can agree with.

One thing that might be somewhat unique about Flow Based Programming (FBP) is that FBP requires some other paradigm besides FBP. This thought is discussed in this topic: Flow Based Programming as a Fractal Approach to Software Development (link:  https://groups.google.com/d/topic/flow-based-programming/hRfyKqv3QtQ/discussion).

This FBP paradigm not standing alone would seem to me to be rare (although I know very few paradigms). I think OO would claim to do everything by itself, Would Data Flow Programming claim to do it all? I think Functional Programming (FP), claims to almost do it all, but needs monads or something to handle what it does not (such as basic I/O ! ).

This thing about FBP not standing alone in no way reduces its benefits. Seems like the few paradigms of which I know all fall apart when the system size becomes very large. FBP can be used starting at a very low level all the way to the highest level. The lowest level is not in FBP. The lowest level might be written in assembler, C, Python, or any language. And the lowest level might be OO, Component-Based, or anything, even Spaghetti. ("Spaghetti" meaning some horrible paradigm or non-paradigm, or whatever.)

The greatest benefits from FBP probably result when FBP is used from a very low level on up to the system level. Also, FBP could be used to include legacy systems and programs with new programs.

That is what I have been understanding.

So, (among other differences could FBP being meant to be applied in conjunction with one or more other paradigms be something that uniquely distinguishes it from Data Flow Programming?

[Ged Byrne]

Hi Paul,

I think this idea of granularity, with different principles and ideas being applicable at different levels or granularity, is the key here.

The flow network graph is declarative in nature.  The method for defining that graph could be a parser or instructions constructing the nodes and connections.  Either way, a graph is built that describes the behaviour.

The nodes of the graph are components that may be developed in an imperative style.  They contain a list of instructions describing how the IPs consumed from the input ports can will be processed. They could also be declared using a functional style.  It doesn't matter  as long as they meet they interface standards described by FBP.

Regards, 

Ged

[John Cowan]

Brad Cox scripsit:

> Maybe too far, perhaps mention that the data types can contain object
> references so streams contents can be lists or trees, unlike Unix byte
> streams.

This is not such a contrast as JPM traditionally makes it out to be.
It's true that at the OS level, pipes are byte streams, as are files.
However, most tools used in the Unix pipes and filters architecture treat
both pipes and files as string IPs delimited by newline characters (or
NUL characters in a few cases), or even as records with multiple fields
separated by a TAB character (usually changeable).

--
You are a child of the universe no less John Cowan
than the trees and all other acyclic http://www.ccil.org/~cowan
graphs; you have a right to be here. co...@ccil.org
--DeXiderata by Sean McGrath

[Dan]

Ged,

Short questions of yours, short answers of mine:

Is FBP a programming paradigm, like OO or Dataflow?

Definitely yes.

Is FBP an architectural style, like Rest or SOA?

Yes, it is.

Is Flow Based Programming based on a single paradigm or multiple paradigm?

It is based on multiple paradigms.
I would like to have several paradigms in the name but it will become too long, something like CAMFOP: Component Aspect Model Flow Oriented Programming. I am just kidding.
But have you noticed that in FBP all the other important aspects (some I have enumerated above) are coming naturally? There is a say: "Misfortunes never come alone". The same is true for the opposite. In nature a good mutation brought a chain of benefits pushing the living organism on the next level of evolution. The same is happening with the software paradigms. For instance, to implement AOP in imperative programming is a mess (check the current documentation available on the net). In FBP it comes naturally. The same with modeling and others.

Regards,
Dan

[Brad Cox]

Unix bytestreams don't work for lists or trees; anything that contains references/pointers. JSON or XML DOM trees, for example. Not talking about ability to parse XML text to trees, but passing the trees between components directly, without conversions.

[Paul Morrison]

On 12/04/2012 11:10 AM, Ged Byrne wrote:
> Hi Paul,
>
> I think this idea of granularity, with different principles and ideas
> being applicable at different levels or granularity, is the key here.
>
> The flow network graph is declarative in nature. The method for
> defining that graph could be a parser or instructions constructing the
> nodes and connections. Either way, a graph is built that describes
> the behaviour.
>
> The nodes of the graph are components that may be developed in an
> imperative style. They contain a list of instructions describing how
> the IPs consumed from the input ports can will be processed. They
> could also be declared using a functional style. It doesn't matter
> as long as they meet they interface standards described by FBP.
>
> Regards,
>
>
>
> Ged
>
>

Absolutely - I agree totally! And I think Ron is absolutely right in
saying "FBP can be used starting at a very low level all the way to the

[Dan]

Ron: "One thing that might be somewhat unique about Flow Based Programming (FBP) is that FBP requires some other paradigm besides FBP. [...] The lowest level is not in FBP"
Ged: "I think this idea of granularity, with different principles and ideas being applicable at different levels or granularity, is the key here."
This is interesting. I have asked myself: why a paradigm is good at one level and is bad on another level (e.g. high vs. low level, fine vs. coarse granularity). We have several answers to this but I want to highlight that I do not think that "FBP is not for lowest level". I think FBP can describe all the levels, all the way from high level (coarse granularity) to the lowest (finest granularity). Nothing stops us to have a software program fully FBP down to the atomic components (like basic arithmetic or logic operations).
It is true that all this "description" (FBP program) should be translated for a specific machine that has a different structure then our program. The software components are not mapping one to one to the hardware components. What we do in fact is to simulate our network of components on the hardware available.
This translation process (i.e compilation) is a pattern recognition system (exactly as it is defined in the literature). We need to flatten the multidimensional structure of the FBP program (our network of components) to the dimension of the hardware available (single or a limited set of microprocessors).
The process of translation (compilation) is gradual, changing the "declarative source code" into a more flatten structure with less dimensions until it reach a single thread if we have just one microprocessor. In the process of translation we keep the principles of data flow and in the end we have just one real component (the microprocessor) and a lot of data in the flow. The data is the original components but in another stage (interpretation stage).
As I said before: this process is not trivial but it can be done. Our problem is we are afraid to think all the way down to the machine code (bytecode). That's why we use for the lower level an imperative language that we have available.
Regards,
Dan

[Paul Morrison]

On 12/04/2012 12:13 PM, Dan wrote:
> For instance, to implement AOP in imperative programming is a mess
> (check the current documentation available on the net). In FBP it
> comes naturally.

Hi Dan,

Interesting post! Is your AOP Aspect-Oriented Programming or
Attribute-Oriented Programming (see https://en.wikipedia.org/wiki/Aop
)? If the former, could you maybe give us an example, and show how it
could be handled in FBP?

Thanks in advance,

Paul

[Brad Cox]

Agree, with the proviso that the fractal notion (FBP all the way down) encourages the "one language could do it all  (if we could only agree on the right one). Its only thinkable because we use hardware to floor out the recursion and don't recognize that the hardware itself is a component that is composed of smaller components, with radically different binding mechanisms to join them (the card-, chip-, gate-level and beyond) I've mentioned before.

That's true even of that tree in the yard. The trunk/branches/twig seems fractal until you realize that those are bound via lignin/cellulose; i.e. cellular-level bindings, and there are many other very different bindings beyond that, at the molecular, atomic and nuclear levels to name just a few.

[Paul Morrison]

On 12/04/2012 12:42 PM, Dan wrote:
> Our problem is we are afraid to think all the way down to the machine
> code (bytecode). That's why we use for the lower level an imperative
> language that we have available.

One dimension that is implicit in a number of these discussions is the
one that runs from theoretical to pragmatic - I guess we are all at
different places on this axis. And please no debates about who is more
pragmatic than someone else! I can certainly visualize many different
ways of specifying components, but my orientation has always been to
build, say, a banking system that performs decently on the chosen
hardware. As I said in my book, you _can_ build a half-adder using FBP,
but it would really only be of interest as a very slow simulation! I'll
sit back and wait for the brickbats :-)

[Dan]

Paul: "Just wondered why you picked this particular topology for your example - this is a perfect example of a deadlock-prone network.  I would say it is anything but simple: if you can guarantee that A emits the same number of IPs to B and C, and that they in turn emit the same number of IPs on the connections to D, then you avoid a deadlock."

I have picked this particular topology because it is very representative, suggestive and quite common. It shows a divergent process (the output of A) and a convergent process (the input of D). Even if A,B,C and D are totally independent components (processors) in fact the data that is coming to the input of D is not because we have to make sure it was originated by the same input a. In an imperative programming (single threaded) this is trivial because you write it as:
r = A(a); //r is a structure that contains two parts a and b
x = B(r.b);
y = C(r.c);
z = D(x,y);
In this case (single threaded) nobody can intervene in the chain of processes so you are sure that x and y are the right ones. In our case (FBP program) we can not be sure and even if I guarantee that A emits the same number of IPs to B and C this is error prone especially when we try to deploy the components A,B,C and D to different locations (threads, processes or different computers).
This force me to add extra information to the IPs in order to make sure they are part of the proper "package".
The problem here is: the COMPONENTS are INDEPENDENT but the IPs are NOT. I have to make sure they flow convergently in a proper manner (i.e arrive in sync). This is a matter of locality.
This situations is very common; you encounter it all over the places. A network of components has divergent and convergent flows. The convergent flow is a problem in FBP in general.
How do you solve it?

Regards,
Dan

On Wednesday, April 11, 2012 11:50:40 PM UTC+3, Paul Morrison wrote:

On 11/04/2012 2:08 PM, Dan wrote:

All the components / processors (A,B,C and D) can be executed in parallel but x and y need synchronization or at least a mechanism that assures they are triggered by the same IP that originated them i.e. the input of A that is a. If all these components are on the same computer / process / thread everything seems to be simple...

Hi Dan,

Just wondered why you picked this particular topology for your example - this is a perfect example of a deadlock-prone network.  I would say it is anything but simple: if you can guarantee that A emits the same number of IPs to B and C, and that they in turn emit the same number of IPs on the connections to D, then you avoid a deadlock.  You have to pay attention to this even when you are sharing memory.  I confess I don't see what it means to say that you can run that network in one process or thread - unless you are using the arrows to mean something different from standard FBP usage.  I certainly agree the situation becomes even more complex if the arrows are connections between machines, but it's pretty complex even on one machine...

[Brad Cox]

Recently read about some guy who set out to build a toaster from raw ore. Actually managed to get fragments of one working after some years.

[John Cowan]

Brad Cox scripsit:

> Unix bytestreams don't work for lists or trees; anything that contains
> references/pointers. JSON or XML DOM trees, for example. Not talking
> about ability to parse XML text to trees, but passing the trees
> between components directly, without conversions.

Well, it's true that when you are no longer sharing memory, you need
some kind of transducer. But if you want something very close to pure
data representations, consider ASN.1 BER or, more flexibly, Protocol
Buffers. Such things will always be needed by FBP programs that require
distribution beyond the boundaries of a single shared memory.

--
John Cowan co...@ccil.org http://www.ccil.org/~cowan
C'est la` pourtant que se livre le sens du dire, de ce que, s'y conjuguant
le nyania qui bruit des sexes en compagnie, il supplee a ce qu'entre eux,
de rapport nyait pas. --Jacques Lacan, "L'Etourdit"

[Brad Cox]

Yep. That's why ESBs distinguish between BCs (Binding components) and ordinary internal components. BCs are the transducers; the edge components.

[Paul Morrison]

On 12/04/2012 1:06 PM, Dan wrote:
> A network of components has divergent and convergent flows. The
> convergent flow is a problem in FBP in general.
> How do you solve it?

I have a whole chapter on deadlocks. In practice we didn't need
convergent-divergent flows very much, and, when we did, the safest
solution was to write one arm of the pattern onto a file. You can see a
couple of examples in http://www.jpaulmorrison.com/fbp/image010.jpg ,
where the files are buried inside the composite components labelled
INFQUEUE (meaning "infinite queue"). INFQUEUE is shown in
http://www.jpaulmorrison.com/fbp/InfiniteQueue.jpg . In the former
diagram a file is split into 3 parts, one part is sorted, and the others
then recombined with the sorted subset using Collate (MAXCCOLX).

FBP deadlocks (as opposed to transaction deadlocks) are always a design
problem.

[Dan]

Yes, AOP stands for Aspect-Oriented Programming.
AOP is very related to Model-Oriented Programming and all these paradigms can be implemented along with FBP with less effort comparing to the current implementations that are using imperative languages.
An aspect is a part of a model. My view is that in FBP a component could be a model and an instance in the same time. Well, FBP name here is not relevant but modeling is a very important aspect that we need to implement in any modern language. For instance you can find "modeling" aspect in C++ as class or template but a class or a template can not be an instance in the same time.
The problem with aspects is the fact they are cross-cutting concerns. An aspect is a model or part of a model. It means that you can find it (i.e. the same aspect) in different places but it crosses with others. It's a construction problem.
A programmer wish is to separate the concerns (aspects) and define them in different places (containers). Any documentation about AOP gives the famous example of creating file logs. Logging is a very pervasive process because it dirties the components' code. You don't want to end up with components that contain logging code that is specific to a platform. That's why you want to isolate the logging code to a single place.
What can you do for this in FBP? Well, you build a component that contains the logging code that is injected where ever the criteria fit. This is related to Inversion of Control.
But building a FBP component is part of the compilation process. The compilation process is a FBP program per se. The programmer should take the benefit to use "compiler components" at run time for what he or she would do anyway in an imperative language (e.g. instances used as templates in games applications).
AOP is a construction process in FBP where the final component is the result of a building process that is a FBP program by itself.
If you want to change a single aspect you don't have to change the code all over because you don't have to call logging functions in each component. You don't write the code to call anybody in FBP anyway but also you don't define the logging calling code inside any component.
It means the code is recompiled each time you change an aspect? Well, it does. Anybody that says it doesn't is wrong. In C++ you can implement AOP and IoC (Inversion of Control) without recompilation of the C++ program but in reality you moved the problem at runtime where you are forced to "interpret" your own specific structures or your specific "high level scripts".
Conclusion: AOP in FBP (and in any language) is related to the construction process. We have not discussed too much about modeling and construction process in FBP in this group.
My post is becoming longer and longer so I will stop here for the moment.

Regards,
Dan

[Paul Morrison]

On 12/04/2012 1:42 PM, Dan wrote:
> It means the code is recompiled each time you change an aspect? Well,
> it does.

Thanks Dan, that's very thought-provoking! And, if I understand you
correctly, you are right that we can take advantage of system knowledge
to generate different networks for different environments - maybe even
semi-automatically. An example of this I alluded to earlier in this
thread might be to adjust the number of parallel paths in the network
based on the number of processors. And some adjustments of the network
could even be made at run time - carefully! I talk a bit about this in
my book in Chap. 7 Composite Components). Maybe someone will run with
this!

[Dan]

Yes, Paul. It is exactly what I had in mind.
(Paul: "we can take advantage of system knowledge

to generate different networks for different environments - maybe even

semi-automatically" and "adjust the number of parallel paths in the network

based on the number of processors.   And some adjustments of the network

could even be made at run time - carefully!").
This is so different comparing to virtual machine solution for instance. Virtual machines are just another bottleneck (a kind of Von Neumann bottleneck). With the ideas above we can take advantage of translating the same FBP program for different platforms.
But what is different comparing to the classical imperative languages? We have compilers for many platforms. Well, in FBP we know all the possible parallel paths, we can map these one in so many ways on the real hardware. This is such a valuable information that you don't have in imperative languages. Moreover in FBP you can do memory optimizations that you just can not do in imperative languages.
And, as Paul said, we can do some adjustment even at run time, carefully of course :).
Dan

[John Cowan]

Dan scripsit:

> But what is different comparing to the classical imperative languages? We

> have compilers for many platforms. Well, *in FBP we know all the possible
> parallel paths*, we can map these one in so many ways on the real hardware.

Up to a point. Unless we have deep knowledge of all the components and which
ones are stateful, we don't know how much actual parallelism (as opposed to
concurrency) we can get.

--
John Cowan co...@ccil.org http://ccil.org/~cowan
The penguin geeks is happy / As under the waves they lark
The closed-source geeks ain't happy / They sad cause they in the dark
But geeks in the dark is lucky / They in for a worser treat
One day when the Borg go belly-up / Guess who wind up on the street.

[Dan]

You are right, we need deep knowledge about components in order to have the freedom to map them on the real hardware in any way we want. If we don't have this knowledge about some components then it means we can not break them down and transform (compile/translate) them into a thread. In this case these components are just compact black boxes. We can not do anything to avoid this. If something is a black box we just can not make it white but we can try to bleach it a little bit :).
Also you say: "Unless we have deep knowledge of all the components and which
ones are stateful, we don't know how much actual parallelism (as opposed to
concurrency) we can get."
We don't need to know which ones are stateful in order to translate them properly and execute them concurrently (some say in parallel). Unfortunately, a component that is a black box (unbreakable) is already compiled otherwise I would have known more about its internals. I don't need referential transparency for that.
I do agree with you that we don't know how much actual parallelism we can get because we don't know much about the black boxes and whether they are executed concurrently or not. We don't have control over the black boxes. That is why a black box is not a big technical advantage because it is inflexible, untranformable.
In fact, when we deal with black boxes we are dealing more with linking process and less with compilation/translation.
We need to know which components (functional units) are referentialy transparent when we are about to reuse them in one way or another.
Parallel computing and concurrent computing is fuzzy for many people. Some highlights the differences but looking into the dictionary you will find overlaps. Wikipedia says:
"Parallel computing is a form of computation in which many calculations are carried out simultaneously, operating on the principle that large problems can often be divided into smaller ones, which are then solved concurrently ("in parallel").
Concurrent computing is a form of computing in which programs are designed as collections of interacting computational processes that may be executed in parallel."
After reading these definitions it is even more confusing about what exactly "parallel vs. concurrent" means.
Regards,
Dan

[John Cowan]

Dan scripsit:

> Also you say: "Unless we have deep knowledge of all the components and

> * which ones are stateful*, we don't know how much actual parallelism

> (as opposed to concurrency) we can get." We don't need to know which
> ones are stateful in order to translate them properly and execute them
> concurrently (some say in parallel).

FBP components are executed concurrently, but not in general in parallel. See
http://recycledknowledge.blogspot.com/2012/04/concurrency-vs-parallelism.html
for my explanation of the difference.

--
Your worships will perhaps be thinking John Cowan
that it is an easy thing to blow up a dog? http://www.ccil.org/~cowan
[Or] to write a book?
--Don Quixote, Introduction co...@ccil.org