Event-driven or message-driven

[Paul Morrison]

This discussion was started by Herman Van Goolen, in a personal note to me.  He and Ken Kan have agreed to open this discussion up to the full Google Group.  There had been quite a bit of back and forth before we decided to do this, so this post is quite long!  Sorry!   Unfortunately, this has to be a single post, as IIUC posts to the Group cannot be modified after the fact. (I have dropped the closing greetings, to save space.)

Note from Herman Van Goolen:

A major part of the last discussions of the Google Group on FBP seems to be related to a ‘wrong’ execution sequence of components. Ged seems to be stating that this is the result of the fact that FBP is message-driven. In my opinion it is the result of the implementation of the FBP driver, which may be a result of some missing elements of the FBP concepts to make it suitable for the world outside the business transaction processing style of the 1980’s.

When I presented my personal view of [an implementation of FBP being developed within IBM Canada] to Wayne Stevens [the late IBM architect who spent years trying to get FBP accepted within IBM] I proposed a priority system to guide the execution sequence. In my opinion this was meaningful because (1) this could be used to optimize the execution time and (2) to be able to cope with unknown new possibilities in this changing environment.

In fact, the driver is selecting the next component to be (re)started based on a list of waiting ports. When there are several components that could be (re)started, the sequence of the ports in this list will influence the result of this selection. As such the implementation of the driver can influence the execution sequence. The execution sequence may (seem to) be unpredictable to the programmer.

In order to give the programmer the possibility to have a better control over the execution sequence, I proposed a priority to be assigned to the receiving ports. The highest priority would be an immediate execution whenever an IP arrives at the port.

The implementation of NoFlo corresponds with the assignment of this highest priority to all input ports. Of course this kind of default priority will not result in all circumstances to an optimal execution sequence. But it has the major advantage that it (often) reduces the task of the driver to an absolute minimum. And I got the impression that in the implementation of NoFlo performance has played a very important role. Maybe Henri Bergius even had (/has?) the idea that he can build a flow-based system without using a driver, or maybe only within some specific structures.

It would be possible to implement an FBP driver using the same default priority. (Causing problems and/or restrictions.) This shows that there is no substantial difference in execution sequence between a message-driven approach and an event-driven approach. The only advantage of the NoFlo implementation may be due to the use of the callback interface as a common function to its ‘driver’ mechanism and other (I/O) events.

In order to make FBP more flexible, I would suggest to allow the programmer to assign a “bypass the search mechanism of the driver” attribute to all  the input ports that can benefit [from] it.

Response from Paul Morrison

Nice to hear from you!  I must admit you have lost me a bit here - I didn't even get the idea you describe in your first line out of the discussion - but the topic is getting very long and I may have missed it.  Perhaps you could clip the relevant sections and send them to me (us).

I will have to spend some time thinking about this, but I seem to remember Ken saying that NoFlo does a chain of calls (?)  from left to right until it hits a network leaf.  I may be wrong, so I am copying him on this note.

JavaFBP and C#FBP processes already have a priority attribute, which I think would address your concern wrt the "classical' multithreaded (multitasking?) FBP implementations, so it would be very interesting to play with that to see if that can be used to affect responsiveness. I know you described your priority as applying to ports, so perhaps we could the port priority to dynamically change the process priority - sounds complex, but I can sort of visualize it!   It may be a better fit with "green thread" implementations, as port priority and/or process priority can be used to determine the order in which processes are added to the (FBP) "future events" queue.

I also don't know whether this would do much to solve the problems Ged has brought up - although I feel that solutions along the lines I proposed should address a lot of them - with no changes to FBP implementations.  My test case is extremely fast! [I am referring to http://www.jpaulmorrison.com/graphicsstuff/ClientServerMultiplex.png ]

I'm also not sure what search mechanism you are referring to - all the code for my implementations is up on Github.  Also I am afraid I don't know anything about NoFlo implementation - I don't even know if it has a driver!

Ken, any comments?

Response from Ken Kan:

Hi Herman!

First off, I think Paul is right that you should post this to the forum! I'm continually amazed by the collective knowledge of the forum.

And for Paul's specific questions, yes, each message that gets sent from the point of origin (say a JSON object returned from the server) is passed through the graph synchronously until a process decides to not forward it. Basically callbacks like you said.

Though, rather than an "advantage", I think NoFlo's use of callbacks is its disadvantage. Perhaps I'm not understanding what you really mean there.

Response from Herman Van Goolen

Hi Ken,

<I have dropped some biographical information - JPM>

The reason why IBM Research was against the flow-based concepts [I didn't know this!] is probably the same as why IT-science in general does not embrace it: it is too close to the natural human real world. In other words, it is too complex to be able to manage it in a scientific way. And I must admit that even after the reading of the detailed descriptions in Paul’s book it is almost impossible to describe the basic ideas of FBP. (The description mainly consists of a large number of technical elements needed to explain the implementation.)

On the other hand, it is probably due to the same fact that the basic FBP approach is so close to the natural human world, that it attracted me and many others. But almost everybody seems to be struggling with several of those technical elements. And a lot of those smart enthusiastic programmers try to present better solutions. But over again these solutions only apply to certain environments, are incomplete,… And the Google forum is loaded with new ideas, deviations,…

And I assume that those endless discussions and proposals will go on and on. As [neither] ‘classical’ FBP nor NoFlo has an essential architecture, –i.e. both are the result of a combination of techniques they personally do prefer,– there will never be a consensus. Unless a ‘big brother’ will stand up with a (technical) flow-based approach that fits his needs. We’ll see which way the wind blows.

I’ve seen very few architectural descriptions. E.g. what is a port? a connection?... I only encountered descriptions of the technical implementation of ‘classical’ FBP.

As such it is evident that this results in diverging implementations. E.g. should the buffering of IPs (if required) occur as part of the connection, or within the port?

The discussion about the execution sequence of components is related to an essential, but dark element of FBP: what is “flow-based”? Or more precisely: what is “flow-driven”? What is the exact relationship between getting or losing control of an IP (receive and send in classical FBP) and getting or losing execution control? This is the very beginning of combining data control with execution control. What is the solution to a combination of the execution controlling flowchart and the dataflow chart? 25+ years ago I proposed to introduce a (programmer controllable) mechanism that makes it possible to couple both, as this is an essential element to upgrade flow-based to flow-driven. This has been swept under the carpet. Now partial solutions are being introduced. As an outsider, (still close to the era of the slide-rule?) I’m not able to discuss those architectural problems in a JS style as used by the forum.

What is needed is a description of the essential ideas of  a “flow‑driven” approach. However, I’ve never seen any attempt in that direction. I got the impression that only (smart) programmers are joining the Google forum, whereas the contribution of architecture-minded persons is needed.

Herman Van Goolen - additional comments:

Paul,

You wrote “I don't know anything about NoFlo implementation - I don't even know if it has a driver!” I got the impression that NoFlo is using a hidden driver: the CallBack listener. A clever solution in itself, in line with what I learned from an experienced advisor 40+ years ago: don’t reinvent, just steal!

“Flow-driven” refers to the flow of data and its (implicit) execution driving power. As such it implies 2 activities: (1) data flow control and (2) execution control. As far as I can remember DFDM [the 2nd IBM mainframe implementation] was using its own storage allocation mechanism and an (execution controlling) driver. I assume that in the current implementations you are using (‘stealing’) the allocation mechanism of OO. ‘Stealing’ the mechanism around the CallBack of JS is not that different; it is a simple solution to a complex task. But it implies that they have to accept all the restrictions that are resulting from its implementation, e.g. the way the queuing mechanism works, the fact that you are not able to deal with other parameters,… That’s why I am assuming that there may be specific structures of components where the CallBack mechanism may cause problems. But maybe experienced NoFlo users and specialists (as the NoFlo developers themselves) can circumvent most of these problems. But undoubtedly this may remain a very weak point of the NoFlo implementation.

Assuming that the CallBack function is set up to deal with the sequence of interrupts as required in the intended environment, it is obvious that this ‘driver’ will do its job as expected. However, when other kinds of priority are involved, it may/will fail, or other (rather risky?) mechanisms have to be invented. Anyhow, I believe that this may increase the complexity of the complex flow-based approach.

Comments from Paul Morrison:

Fascinating note, Herman!

<some paras about posting this correspondence to the Group skipped>

... I had never heard the comment you made that starts, "The reason why IBM Research..."  The story I was told was that IBM Corporate wasn't comfortable with basic "research" being done from countries other than the US, and especially not from IBM Internal Systems... even though I was told we had claimed R & D tax breaks.  And there was no mechanism within IBM for money made from such a product to come back to IBM Canada.  When we did get some acceptance (the last FBP implementation) we were told we had to interface with every hare-brained scheme being pushed by IBM Corporate, whether it made sense or not!

Also the specific point they brought up was nonsense, as one of the main selling points of OO was precisely that it allowed the program to mirror the real world (turned out not be true, but still...).

So, yes, you're absolutely right - but I am not the right person to pull all the different threads together - I just know how to build systems! :-)

Comments from Ken Kan:

Hi Herman,

I really appreciate Paul for having CC'ed me on this thread. Your message absolutely made my day.

Paul is right. The fact that the forum is lacking non-programmers is precisely the reason why you should be part of the conversation.

<some biographical information dropped>

Perhaps because of my background of "just hacking things around", I'd always thought that there wasn't really any structured way to development. Just make it work! Nothing else matters, I thought. That is, until I stumbled upon FBP. It is precisely your expertise on architecture that the community needs. I certainly learned a whole lot about proper architecture since learning of FBP and doing some research on my own.
In fact, I started the [thread called "An Existential Model for FBP"] regarding a model of FBP because of the exact same confusion that you mentioned: "it is almost impossible to describe the basic ideas of FBP." To this day, I still can't confidently explain FBP in terms other than mechanics like ports and IPs.

Architectural thinking, a flow-driven approach, and your thoughts in general are precisely what we need.

So please, please, please contribute on the forum. I'm looking forward to learning more from you!

(To start things off, what exactly do you see with a flow-driven approach?)

Response from Herman Van Goolen

I’ve been using “flow-driven” instead of “flow-based” only to highlight the fact that it is not correct to assume that the (normal) flow of data between components is sufficient to fully control the execution sequence of the components (or subsequent parts of these components). Sometimes additional control information has to be provided, as e.g. by sending an empty IP. And the internal structure of the component has to be such that execution can be halted and resumed in precise harmony with data passing. The sequential structure of (classical) FBP (often denigratingly referred to as Von Neumann style programming, whereas we should admire him for that style suitable for that pioneering era), as well as the event-driven structure of the NoFlo components require the programmer to think about the fact that it is “flow-driven”.

I don’t think it would be meaningful to bring this up to the Google forum, as it is not in favor of FBP to introduce a deviating term. And… before starting such a discussion there is a need for a definition/description of the term “flow”.

Response from Ken Kan:

Herman,

I am really intrigued by this. How would you define "flow"? I've been looking for a definition myself for some time now.

I'm specifically intrigued by your statement, "... require the programmer to think about the fact that it is 'flow-driven'." There is enough room in that statement for incorrect assumptions that I'll refrain from interpreting it. What do you mean by that?

This topic is now open from comments!  Any architects out there?  Engineers, logicians, musicians, artists, drummers...? JPM

[Francisco León]

When an architecture is designed focusing on the Event Driven approach, is very probable that you would obtain a system with a high resource contention in order of controlling the order of execution of processes. Flow Based by message passing gives you some flexibility to the spontaneously arrive of data from different processes and handles concurrency better. With some additional parameters you could implement some kind of message sorting at Component Ports queues for provide a manageable order of execution even on non deterministic conditions with some time interval frame window attribute in messages. The idea of "Tea-Time" present in Actor programming from SmallTalk is an example of how to synchronize the execution of concurrent components.

[Ged Byrne]

Hi Paul,

Interesting that this should coincide with my other post.

As I argue there, the flow in FBP is something in the real world reflected in the software. It is by capturing of flow, something human's intuitively understand, that FBP allows non-programmers to collaborate in the design process. This is how the use case described in the books opening paragraphs is realised.

Van has understood exactly what I have been trying to explain. It is about the sequencing of events that form the interaction between human and machine. In the UI this must be a tight feed back loop. The machine's response must immediately follow the human's action.

See also Norman's cycle of interaction:
http://www.w3.org/People/Bos/DesignGuide/simplicity.html

Get the sequence wrong and the user's flow is broken.

For FBP to support this it must allow for different approaches to the sequencing - what I have been calling scheduling.

See how overloaded the term flow is? The flow of IPs through the graph. The user's flow. The flow of events through time. Yet there is an underlying force that is common to all of these. Something that is intuitively understood yet so difficult to explain.

Regards,


Ged

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Tarvydas]

This topic is now open from comments!  Any architects out there?  Engineers, logicians, musicians, artists, drummers...? JPM

I studied Physics (and got a 3-year degree), then I studied EE (and got a 4-year degree).  I’ve run software consultancies for 30+ years.  I want to understand what my employees are doing, and I want my customers to understand what I am doing, hence, I think like an architect - show me.

I got into EE because of guitar-playing - poor student, couldn’t afford to buy amps and pedals, so I learned how to design and build them myself.  Then, when I’d built every kind of pedal possible, I got bored.  I read the 1st anniversary issue of Byte Magazine and decided to build my own CPU (Z80-based) based on the centrefold(!) of that magazine.

I learned that I could design hardware and make it reliable.  Yet, when I built software, it was unreliable.

I’ve spent the last 30+ years trying to figure out what the difference is between reliable hardware design and unreliable software design.

FBP, and my variant bmFBP, is the answer, I feel.

Asynchrony, and the flow of information between asynchronous components is the basis.

One must design components that *cannot* rely on synchrony, i.e. make components asynchronous from the outset.  That’s the paradigm, that’s the brain-shift.

All the stuff about processes, stacks, bounded and unbounded queues, ports, fan-out, fan-in, etc, etc, are just details on how a particular implementation (“language”) deals with asynchrony.

A stream of IP’s is the same as a bunch of events.  A stream is just a continuous flow of data.  Events are just a stream with big gaps of nothingness (time) in-between the data units. 

One kind of FBP (a “language” variant) deals with continuous streams better - the “syntax” is sugared to hide the non-essential bits.  Another kind of FBP deals with events better, with a syntax better-suited to irregular arrival of bits of data.

It is very simple, yet elusive.  Make *everything* asynchronous.  One has to think about software differently when everything is asynchronous…  One has to think in the manner that hardware designers (e.g. TTL) need to think.  This leads to Engineering and reliable designs.

pt

[Paul Morrison]

I have said this before many times, but here it is again, based on my attempts to produce reusable components that handle streams of data (I had no success until stumbling across FBP!):

The logic within long(ish)-running processes tends to involve loops within loops, with decision points scattered around inside them.  The (von Neumann, or some one else?) storage model we have come to know and love uses an array of pigeonholes, with non-destructive readout.  This means that you don't get any notification that a) you have reached into one of them and retrieved garbage, or b) you have pushed a pigeon in on top of one that's already in your pigeonhole.  This means that things tend to appear and disappear rather unpredictably. And it gets worse as the logic becomes more complex.

Think of the difference between

    A <- B

    B <- C

and

    B <- C
    A <- B

   

Very different meanings!  You can't afford to drop a deck of cards!  :-)

Or consider the following very standard filter logic which deletes selected records from a file (IN): 

         read into a from IN
         do while read has not reached end of file
              if c is true
                    write from a to OUT
              endif
              read into a from IN
         enddo

As I say in my book,  "What action is applied to those records which do not satisfy our criterion ["c"]? Well, they disappear rather mysteriously due to the fact that they are not written to OUT before being destroyed by the next “read”. Most programmers reading this probably won’t see anything strange in this code, but, if you think about it, doesn’t it seem rather odd that it should be possible to drop important things like records from an output file by means of what is really a quirk of timing?"

Changing the records to tangible "things" made all the difference!

I think this leads naturally to Paul T's point - if everything is asynchronous, and the designer does not know when exactly things are going to happen, she is not tempted so much to rely on sequence to control event logic...  This actually makes a lot of programmers very nervous!

[Ged Byrne]

Hi Paul,

I believe that achieving asynchronous processes is only half the story.

For me the key benefit of being asynchronous within the machine is that it then allows us to be synchronous with the real world.

What really matters is the sequence of events that occur in the real world. I purchase an insurance policy, I make monthly payments, I make claims every so often. The sequences can last for years, even decades.

FBP gives a simple approach where these long running processes can be clearly reflected in the implementation.

For these very long running processes a slightly different approach is needed to interleaved the real world and machine events. The principles remain the same.

We model he journey that an IP takes from the beginning to the end. From the first application form to the time the policy expires is described by a single thread of execution through a graph of possible routes.

For me asynchronous with the machine is just a step on the way to synchronised with the real world.

This is all primarily informed by the work of Michael Jackson.

http://mcs.open.ac.uk/mj665/

Regards,


Ged

[Tom Young]

One way I look at FBP is as an approach to avoid the additional effort and  added complexity (and obfuscation of the underlying system and its design)  of converting a network of processes as defined in a dataflow diagram(DFD) into the synchronous model represented by a structure chart;  as described at length in http://www.yourdon.com/strucanalysis/wiki/index.php/Chapter_22
.

What is needed, of course, is a framework(the glue) to connect the DFD processes into a coherent program.     Such a framework is not too difficult to implement, but the resulting system will operate somewhat differently, in some respects(mainly timing) than a system built from a synchronous model.

 
If these differences are important, the issues should be resolved in the requirements and embedded within the original design; rather than adopting the synchronous approach.

The benefits of FBP are even more significant when it becomes necessary to modify the original design, and also  when the framework and other components can be reused.

It is possible to define the particulars (queuing, priorities, etc.)  of the 'glue' framework, I suppose, but much more useful when the problem domain has been defined and a DFD created.

  ---twy

"...the shell syntax required to initiate a coprocess and connect its input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

[Paul Morrison]

Hi Ged,

I think you have verbalized something very important - this makes a lot of sense!

FBP is just so natural to me that I sometimes can't see the forest for the trees!

Best regards,

Paul

On Mon, Oct 5, 2015 at 4:09 AM, Ged Byrne <ged....@gmail.com> wrote:

[Ged Byrne]

Hi Tom,

Have you read DeMarco's 1979 classic "Structured Design and System Specification"?

The full text is available here:
http://extras.springer.com/2002/978-3-642-59413-7/4/rom/pdf/DeMarco_hist.pdf

It is a great book, so many insights. I particularly like this one:

"Whatever analysis is, it certainly it certainly is not very similar to the work of designing, debugging and programming computer systems...

The implementation disciplines are straightforward, friendly, definite and satisfying. Analysis is none of these things...

Analysis is frustrating, full of complex interpersonal relationships, indefinite and difficult. In a word, it is fascinating. Once you are hooked the simple pleasures of system building are never again enough to satisfy you. "

It seems to me in recent decades that the challenges of genuine analysis have been rejected for the easy pleasure of coding.

The way this has been achieved has been by making the code very complex, abstract and obscure.

This reminds me of my all time favourite quote from Tony Hoare's Emporer's Old Clothes:

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature."

The software industry is eating the world, and it is doing so with systems so complex that the deficiencies are hidden beneath countless layers of indirection.

With FBP Paul has demonstrated some of the simple laws which underlie the complex phenomena of software's nature.

No wonder the software industry keeps rejecting it.

Regards,


Ged

[Tom Young]

Don't think I have read DeMarco's book.  The quote about analysis is interesting. 

It occurs to me that a program, such as Firefox, with a huge set of plug-ins, cannot be fully tested as all the plug-ins cannot fit into one system.   If the plug-in system could be distributed (which would be relatively easy if FBP had been used), then comprehensive testing would be possible.

---twy 

"...the shell syntax required to initiate a coprocess and connect its input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

[Paul Morrison]

On Mon, Oct 5, 2015 at 4:09 AM, Ged Byrne <ged....@gmail.com> wrote:

We model he journey that an IP takes from the beginning to the end. From the first application form to the time the policy expires is described by a single thread of execution through a graph of possible routes.

For me asynchronous with the machine is just a step on the way to synchronised with the real world.

Pragmatically we model this logic as a flow through a chain of processes.  Now just as processes can be stretched in time, the chain can be "stretched" by cleaving the chain into separate jobs or apps, and putting storage between the processes as required.  Storage can easily be added to the app sequence by putting writer processes on the output ends of each app, and reader processes on the input ends.

This is one of the fantastic things about FBP:   in  old-style programming, you have to make the decision about the job boundaries very early on, and then they are very hard to change, while in FBP, you can design a if there are no job boundaries, and then add them in later.  This is discussed in Chap. XV of the 2nd edition, which starts with a description of building applications, which sounds exactly like what Ged has described (with references tot he Jackson inversion).

Thanks for the good feedback, Ged!

[Paul Morrison]

Ged, thanks for this post - these are great comments!  And, Tom, nice to get your comments - you've been quiet for a while!  For some reason your post didn't make it to my inbox!

When we started our FBP Meetup in Toronto, I naively thought we'd be discussing real-life applications built using JavaFBP or C#FBP, reusable components, performance, etc.  Instead all of the attendees are busily building their own FBP-like implementations, and it's true we do get into some interesting philosophical discussions about basic concepts, but I do get that, for most of them, building software is more fun than just learning how to build applications - whereas for me it's the other way around!  Hence a bit of a disconnect!  Apologies to any of the members who may be reading this...!

Maybe it's like the difference between scientists developing new materials, and engineers building bridges...  de gustibus non est disputandum!

Regards,

Paul

[Ged Byrne]

Hi Tom,

I mention it because it is the book that introduced me to Data Flow Diagrams and Structured Analysis.

I think a data flow approach to plugins would be easier to test because the boundaries are clearer.

In OO the plugin is often 'inserted' into the object model, through inheritance or dependency injection. The plugin becomes part of the overall system.

With components connected via channels it is much clearer where the core system ends and the plug in begins.

Regards,


Ged

[Tom Young]

I agree even though OO was originally thought of as using message passing to communicate between objects.    IMO FBP is better defined than OOP.

Programs like Firefox may be essentially untestable even with FBP where multiple plugins update and depend on global state variables.

Cheers,

         ---twy

Tom Young
47 MITCHELL ST.
STAMFORD, CT  06902

When bad men combine, the good must associate; ...
  -Edmund Burke 'Thoughts on the cause of the present discontents' , 1770