do node-red and noflo have different "programming models"?

[Darren Cruse]

I see this is a general FBP group but me I've mainly used node-red and am just starting with noflo/flowhub and I was curious:

at first glance comparing node-red and noflo graphs the node-red ones seem to have fewer lines/edges in general

I just googled to confirm and the node-red docs say "A node can have at most one input port and many output ports".

(and from what I've seen playing with node-red they more often than not have the one output port - so one in and one out most of the time)

but from my very limited playing with noflo/flowhub so far it seems far more common to see multiple input ports and output ports.

I'd come to appreciate a little that node-red's model is very simple it's mainly that nodes/components are kind of just functions that receive a JSON message and transform it in some way to make their output message.

but it's typical in node-red for their gui to have you configure the node with settings that are not visible in the graph at all.

so there's a hard distinction between the node's "settings" and the "message" it processes thru the flow.

is it the difference I'm seeing mainly just that noflo doesn't make such a distinction?  so often these multiple "in ports" include the things node-red makes into it's "node settings" e.g. I see in flowhub it's common to simply type in certain in port values you don't otherwise "wire" in the graph.

So is the big difference that in noflo you *can* wire anything in the graph whereas in node-red you're more limited?

Or are the differences deeper than this? 

(I haven't tried noflo enough yet to be confident wondered if folks here have used them both)



 

[paul tarvydas]

> ... I've mainly used node-red...

I've looked at node-red only briefly. I have a question that you might be able to answer:

What goes into the one (and only) input pin and how does it get processed? (I think I want a detailed example).

thanks
pt

[Darren Cruse]

node-red simply calls what flows in as the "message" (abbreviated "msg"), which is simply a JSON object.  where (as typical in javascript) this is just an untyped (dynamically typed) bag of properties.

so what's in the message is super flexible different types of nodes put out different types of JSON messages just depending on what they do

there are however some conventions e.g. quite often (if a node mainly produces a single result) nodes write to "msg.payload" for their output.

on the incoming side I notice that MQTT messages use "msg.topic" e.g. to distinguish one sensor from another (I'm not a big IoT/MQTT person but I think I noticed other nodes do sometimes look by convention for "topic" e.g. I noticed an email node used msg.topic for the subject of the email).

me I'm more focused on writing REST apis and the way the "http in" node works it simply sticks the standard node.js request and response objects into the msg as msg.req and msg.res.  if you have prior node.js experience this makes things pretty straightforward e.g. to read path params from an incoming request you use msg.req.params as you would in node.js or msg.req.body for the post body etc.

otoh in one case I wanted a sub-graph to be agnostic to the source of such params so I had it looking for simply "msg.params" and used a node higher up to transfer query params from msg.req.query to msg.params i.e. so I could use the sub-graph in the context of invoking it with a browser request in one case but without a browser request in another case.

so super flexible (like most dynamic languages) - for good and for bad I guess - the one nice is they encourage their nodes to have nice docs of what these message properties are and they show them in the node-red editor when you click on the node.  Also they have a "debug" node that makes it easy to log out the messages contents at any point in the graph.

https://www.youtube.com/watch?edufilter=NULL&v=y05vQUIMIxs

[Paul Morrison]

It's ironic!  FBP was developed in a little corner of IBM, and was IMHO very successful:  it was used for most of the batch systems of a major bank, over a period of around 40 years, as well as a number of data centre applications...  and yet it was never adopted by the larger IBM community.   We weren't an official lab, we weren't in the US, and also FBP was neither a language nor an operating system, so it sort of fell through the cracks!

A few years ago, IBM came out with Node-RED, which is only a (partial) simulation of FBP, and of course it's the greatest thing since sliced bread!  There are now lots of similar products out there, so Joe Witt has suggested we call them "FBP-like" or "FBP-inspired"  - they are not FBP, which is a different (paradoxically, older) paradigm!   Because so many of them are using the term "FBP", I have taken to calling FBP "classical FBP" - Joe feels this is not necessary, but if there is anything I can do to reduce confusion...!  

From the discussion above, apparently Node-RED nodes can only have one input stream, which means that it cannot do things like FBP Collate, or ConcatStreams, which are basic for doing real data processing with FBP...  But... its practitioners don't have to give up their procedural thinking, which they have all been taught is the only way to do data processing... even though it's much more difficult, and more error-prone, and pushes up the costs of all major data processing applications!  But hey, true programmers are supposed to suffer a bit!

When I discovered/invented FBP, I was not as brain-washed by procedural languages as people more recently introduced to computing - because my first experiences with application development did not in fact use computers!   Weird, right?!   But in fact there was a technology that was used around the world for data processing, and is now largely forgotten.   I have tried to capture this idea in a tutorial on FBP that I am working on with some colleagues - https://github.com/jpaulm/fbp-tutorial-filter-file .  In the background piece, https://github.com/jpaulm/fbp-tutorial-filter-file/blob/master/Background.md , I talk about "Unit Record":  in the first two thirds of the 20th century (Wikipedia), data processing was largely done using something called "Unit Record" - https://en.wikipedia.org/wiki/Unit_record_equipment - chunks of data (cards) being transported (hand-carried by humans) between parametrizable components (machines using plug-boards) - all totally asynchronous. Worked pretty well, actually. I once debugged an accounting machine plug-board over the phone, from home, soaking wet (in the early '60s)!   Do all of this within a computer (or multiple computers), and you essentially have FBP.

"FBP-like" or "FBP-inspired" systems are all attempts to simulate FBP using procedural approaches - FBP involves a paradigm shift, but FBP-inspired doesn't.  Maybe that's the basis for their appeal!   Yes, you've all been taught that asynchronous is difficult, but data flow matches the way the real world works, so surely it makes sense to take advantage of the skills humans have built up over many thousands of years!

On last comment: many of the FBP-like systems that I've seen are what might be called "toy" systems - we were running millions of transactions through our system over night - so it got a pretty good workout!  And it was being maintained by people who, in many cases, weren't even born when it first went live!  The first few implementations of FBP all used "green" threads", but the later "red thread" implementations that I have built do not require you to change this mindset significantly, whereas FBP-like systems on the other hand do involve a different mindset, and I remain convinced they are significantly more complex and less maintainable.  A couple of years ago, I spent 3 days trying to get an app working using one of them (I won't say which one!) - and literally couldn't...  I then tried using a recent (classical) FBP implementation, and it took an hour!

Just had to vent!  Sorry!

Paul M.

[paul tarvydas]

Darren, thanks for the explanation.

When you "send" (deliver) a message to a node-red component, does the caller need to wait for the called component to finish its processing and for the called component to execute a RETURN?

pt

[Darren Cruse]

Hi Paul and Paul!! :)

Seems your replies are semi-related so maybe replying to both:

@Paul Tarvydas: no a node-red component doesn't wait there's not a "caller/callee" situation.  The "programming model" (which I thought all "data flow" systems share is this true Mr Morrison? :) - is that of messages flowing from one node to another like a "pipeline" of processing.    

But it is true that within the boxes/nodes/processes (whatever their name) traditional function calling may be happening in the course of processing the message received to come up with the output to send along the line to the output arrow(s).

So there's kind of two "levels" going on: there's the traditional function calling call tree traditional coding stuff happening "inside the boxes", but at the level of looking at the graphical "flow" drawing it's (at least conceptually) about "concurrency" i.e. each box should be thought of as if they were independently/simultaneously running little machines sending the messages over the lines/arrows one to another.

To Mr. Morrison's point (and credit!!) as someone getting more into this stuff I really like the separation of those two levels in my experience there's always been a lot of talk about "reusability" and "components" and so forth but I'm appreciating more how the graphical flow view really can make this real.  i.e. I appreciate more the separation between the traditional coding down in the boxes/nodes/processes versus those truly being "black boxes" for the person working at (or just looking at) the graphical view.  And that this is really powerful and better than "all code" approaches (and I haven't thought this way until very recently so I'm laughing at myself right now!!).  i.e. a better way of "abstracting" techie details since the graphical view is something people like my boss can relate to (esp. when you use a hierarchy of flows using boxes that expand to more details "sub-flows").

@Mr. Paul Morrison re node-red and Collate/ConcatStreams I'm not 100% sure if there's details I'm missing (seems like each of these tools are using slightly different names for similar things right) but I *think* node-red does support these things fine i.e. though it's limited to one "in port" you can absolutely attach multiple lines/arrows flowing to that port and there are nodes for merging such flows (with likewise nodes for the reverse i.e. splitting a single flow to have messages processed in parallel and then later merged back together).  Not trying to represent node-red I'm still learning hopefully you've had a chance to try it yourself...  maybe this article is a help (e.g. patterns "5"/"6"/"7"):
https://medium.com/node-red/node-red-design-patterns-893331422f42

To your broader comments about the mind-set shift of FBP and the differences between the classical-FBP versus FBP-inspired systems I admit I'm still getting my head around such differences (e.g. the title of my OP).  I've been been looking at these FBP systems partly thru the lens of having looked at and played with other systems focused on concurrency (which I agree are definitely a mind-shift) e.g. I've been thinking of these graphical flow diagrams as most similar to things like Actors (Erlang/Elixir, Akka) or CSP (go i.e. goroutines and channels, clojure's core.async, js-csp).  But depicted graphically obviously.

Maybe I'll appreciate more after I do your tutorial (still on my todo list) - but I guess I was fuzzy on some of your points e.g. all of these systems are to some degree a "simulation" of the message flows to the extent they use different implementation strategies under the covers isn't that true?  so threads in java, "lightweight processes" in erlang, "cooperative multitasking" in node.js...

In my OP when I asked about the "programming model" I was thinking/wondering - independent of those implementation details - how the way to think about the graphical flow does/doesn't differ.

I know one point you've made is whether a message is truly a single thing (like a real world object would be) as opposed to like a pointer passed potentially to two nodes/processes that now (magically in the real world) see two "copies" of what was really one message.

I'm still getting my head around other differences.  I can see the concurrency model (e.g. true threads in java versus "cooperative" in node.js) has ramifications on being able to handle like high volumes leveraging multiple cores (streams of audio or what not), but those use cases aren't big ones for me personally I'm more dealing with lower volume things (REST apis i.e. i/o bound not cpu bound).  I'm not sure if that aspect is part of what you're referring to in "classical-FBP" versus "FBP-inspire" or not (to my thinking that's more "implementation" and not part of the "model" but maybe I'm wrong?)

Anyway interesting stuff apologies if I wrote too much. 

[Darren Cruse]

@Paul Tarvydas apologies just saw some of your other posts here I'd mis-read your question about node-red nodes "call"/"return" as you new to FBP stuff but really you were just asking about node-red sorry.

Though (possibly?) you're not real familiar with node.js and how it works?

Certainly the pipeline of nodes are not just a call chain (if that was what you were asking).

They are closer to node.js/the browsers notion of event handlers.

So the receipt of a message is like an event that is processed by the node to generate a new message that ends up being sent as a subsequent event that can trigger the "handling" of the downstream nodes.

Of course the event loop of node.js/the browser is acting as the "scheduler" letting the different nodes run independently though not *truly* simultaneously.

(the "cooperative multitasking" I referred to above) 

[John Cowan]

On Wed, Jul 3, 2019 at 10:42 AM Darren Cruse <darren...@gmail.com> wrote:

To your broader comments about the mind-set shift of FBP and the differences between the classical-FBP versus FBP-inspired systems I admit I'm still getting my head around such differences (e.g. the title of my OP). 

I think the most significant difference is this:  in classical FBP, a component *decides* when it will read a packet from its input port(s), just as a file-based or database-based program decides when to read the next thing from those sources.  Most FBP-inspired systems don't: the code is entered when a packet is available and is expected to complete processing quickly so that the system can move on.  It's usually tricky to determine which kind of system an implementation is from the thick clouds of hype that generally surround it.

(Some early OSes actually signaled a process when a block of data was available from a file, and it was the application's responsibility to get the data and do something with it before the kernel buffer was overwritten by the next block coming from disk.  This made for contorted programs indeed.)

Having more than one input port makes patterns like "read two packets from port Data, one from port Op, apply the operation named in the Op packet to the Data packets, and send the result to port Out" possible.  You can engineer around such things with a read function that can read the next packet of a given type from the input, but it's messy, and most input ports don't have that much buffering.

I know one point you've made is whether a message is truly a single thing (like a real world object would be) as opposed to like a pointer passed potentially to two nodes/processes that now (magically in the real world) see two "copies" of what was really one message.

My personal view on that is that you should avoid mutating packets.  Drop them and create new ones instead (in GC languages, dropping just decrements the count of packets owned by this component, which should be zero at the end of a run).  Then it doesn't matter how many copies may or may not exist.

John Cowan          http://vrici.lojban.org/~cowan        co...@ccil.org
My confusion is rapidly waxing
For XML Schema's too taxing:
I'd use DTDs / If they had local trees --
I think I best switch to RELAX NG.

[Paul Morrison]

Thanks, John, you've answered my point about Collate and ConcatStreams.  Collate combines 2 or more streams based on "control field" values, so theoretically you could build up a backlog of millions of data chunks at one port before the control field values on the data at another port allow them to be processed...  John probably explained it better than I have!

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flow-based-programming/CAD2gp_Svs%2Ba%3Dk82JO%2BCnS9x2n5q7kxCbsY%3DzMy5O_6z%2Bdvyu4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[ern0]

> I think the most significant difference is this: in classical FBP, a component *decides*
> when it will read a packet from its input port(s), just as a file-based or database-based
> program decides when to read the next thing from those sources. Most FBP-inspired
> systems don't: the code is entered when a packet is available and is expected to
> complete processing quickly so that the system can move on. It's usually tricky to
> determine which kind of system an implementation is from the thick clouds of hype
> that generally surround it.

There are many characteristics which seems to minor questions, but
makes one system very different from another. This issue, we're
talking about, is one of these issues. The interesting part is, that
it's pretty "hidden", probably you don't know the correct answer even
if you have used the system, and probably you even shouldn't take care
of it.

Have you ever noticed that Unix piping is using buffers between
components? Does it matter? Do you really mind that components run
parallel? In MS-DOS, piped components run one after another, passing
data in tmp files, but we are only noticing that it's slower, and we
can't process endless streams (and of course, there are not as many
components).

There are some other important properties of
component-based-data-passing systems:
- Sync or async?
- When editing the net, can user write a component in some script language?
- Can multiple producers attach to a single consumer?
- Connections (messages, packets, IP-s etc.) have only one type, or
there are different types?
- Components are running in separate threads/processes, or not, or mixed?

These are so important design decisions, that if one system differs
from another only in one of them, then they are complete different
architectures.

What does it mean?

1. I don't know, whether we should give 2^n name for FBP-like systems,
depending on important characteristics parameters (where n is the
number of system architecture properties). If we don't do this, it's
not better, we should list almost n parameters to describe the type of
the system.

2. FBP/dataflow/etc. is sooo large area, and we are still at the
beginning of the road.
--
ern0
dataflow evangelist

[John Cowan]

On Thu, Jul 4, 2019 at 3:23 AM ern0 <er...@linkbroker.hu> wrote:

There are many characteristics which seems to minor questions, but
makes one system very different from another. This issue, we're
talking about, is one of these issues. The interesting part is, that
it's pretty "hidden", probably you don't know the correct answer even
if you have used the system, and probably you even shouldn't take care
of it.

You definitely do know if you have to call a read-packet function, or if

your own code has to be structured as a function that takes a packet

as an argument.   "The distinction is unsubtle."  (Miles Vorkosigan)

Have you ever noticed that Unix piping is using buffers between
components? Does it matter?

I'm constantly aware of it.  I think there are three classes of buffering:

bounded by 0 packets, bounded by n > 0 packets, and unbounded.

The last is known to cause problems when there are loops in the net

and maybe at other times too.  I'm not sure if there's a non-performance

distincton between buffering and non-bufferining; probably JPM does.

In any case, though, the receiver never has to wait for a pipe buffer to

fill; a read will return data if any is available.  This is not true of

buffering internal to components such as stdio.h functions provide,

and this can cause almost instant deadlock if you want to send

packets to a component and get other packets back.  If the

component does output buffering, there is almost instant deadlock.

 

Do you really mind that components run
parallel? In MS-DOS, piped components run one after another, passing
data in tmp files, but we are only noticing that it's slower, and we
can't process endless streams (and of course, there are not as many
components).

The performance difference is very notable with large files, which of course

DOS did not have either.  Also, juggling wihout concurrency would be

very boring: throw a ball in the air, catch it, throw another ball, etc. 

- Sync or async?

I don't really know what that means.  Cooperative threading aka green threads

aka coroutines are, I think, the minimum necessary. It has been said that the

deepest divide in programming languages is between those that have only

classical control and those that have coroutines.

Classical FBP needs async I/O if any components actually do I/O, so that a

file-reading component doesn't block a file-writing component or vice versa.

In process-based systems, the asynchrony is in the kernel, of course.

 

- When editing the net, can user write a component in some script language?

The answer to that is unequivocally yes.  There are many such languages available

for the JVM and the CLR, the C++ implementation uses Lua, and of course

in a process-based system you can use what languages you like.

But if you mean literally "while editing", then it depends on whether your GUI

(which hopefully generates a textual representation of some sort) also has

an escape to a text editor that allows you to write components, whether in

Lua or Fortran.

- Can multiple producers attach to a single consumer?

Again, I know of no systems that prevent this.  Fan-out is more controversial,

because you don't know if it means "send to all", "send to one at random",

"send to one in round-robin", etc.  Better to have individual components

that do these things, though unlike most components they may need

internal multiprocessing to avoid blocking all their downstream components

when only one can't take the data yet.

 

- Connections (messages, packets, IP-s etc.) have only one type, or
there are different types?

I'm not sure if you are talking about the static type of a packet or about

"types" in some other sense, like internal buffers vs. TCP vs. carrier pigeons.

 

1. I don't know, whether we should give 2^n name for FBP-like systems,

I think it's a matter of JPM wanting to protect the term he invented from

misuse by people who are only implementing part of what it names.

Ted Codd lost that battle for "relational database", with consequences

that reverberate throughout the industry to this day.

John Cowan          http://vrici.lojban.org/~cowan        co...@ccil.org

If [Tim Berners-Lee] has seen farther than others,
        it is because he is standing on a stack of dwarves.  --Mike Champion

[Tom Young]

Hi Ern, 

On Thu, Jul 4, 2019 at 3:23 AM ern0 <er...@linkbroker.hu> wrote:

Have you ever noticed that Unix piping is using buffers between
components? Does it matter? Do you really mind that components run
parallel?

Yes; Yes; Yes:

Yes, but Unix runs in 'immediate mode' when connected to a terminal or is coerced into thinking it is so connected. 

It matters, because 'Nx  buffering can lead to deadlock in cyclic networks.  Go uses ring buffers which are less likely to deadlock, but defaults to no buffers, perhaps saving some time.

Yes, I want to use all my CPUs,  finish, and go play tennis without filling up storage with intermediate files.

 

 

 

There are some other important properties of
component-based-data-passing systems:

Such as:  subnets(subgraphs, if you will), definition language, 

configurability, naming conventions, extensibilty, and scalability. 

 

Cheers, 

twy

 

   

 Tom Young

47 MITCHELL ST.
STAMFORD, CT  06902

When bad men combine, the good must associate; ...
  -Edmund Burke 'Thoughts on the cause of the present discontents' , 1770

[Darren Cruse]

thanks for all the interesting comments I just re-read the whole thread (except for mine I wrote too much I TLDR myself :).

this distinction between systems that assume the nodes/processes automatically run when IP messages are available versus those where the nodes actively read from the ports had been lost on me thanks for that (node-red is of the first type which maybe makes sense since it was designed for IoT e.g. MQTT messages generated from smart home devices, while otoh I'd looked briefly at the code for a custom node/process in NoFlo and noticed it was like trying to read from an in port and then checking if it got anything and returning early if it didn't - not understanding this distinction I was confused why that was there).

the enumeration of the different design decisions different FBP/FBP-inspired systems have taken was helpful to me as well (if only as a reminder of what I can expect looking at different systems).

I am familiar with the concurrency in unix streams and how the processes are running in parallel while the text lines are flowing thru the pipeline as opposed to each process running one after another (I confess I went for years not knowing that and when I first learned of it my thought was "wow!! genius!!"

So if I follow part of the classical-FBP versus FBP-inspired distinction is this focus on streaming and (maybe broadly?) partly a question of use case? so high-performance-streaming-use-all-possible-cores-to-get-highest-performance-possible (e.g. audio/image/video processing etc), versus less-cpu-intensive-more-about-io-bound-distributed-programming less about cpu stuff more about networked stuff?

I say this partly thinking back to my particular use case which is low traffic restful apis that respond to http requests by hitting other restful apis that return JSON and then do the equivalent of a "join' on the data returned to merge and transform the multiple JSONs into it's response.

So the frequency of http requests is low (I don't think of them as "streaming"), and otoh though the JSON being processed is definitely arrays of JSON "records" I don't think they can be processed as a stream because the "join"-ing is relating *arbitrary* records from the other apis,

i.e. having one record with a certain id and finding the corresponding record from the other api implies I'm either getting one record at a time from the other api (terrible performance) or that I've pre-loaded the entire response from the other api (which of course rules out a streaming solution).

 

[Paul Morrison]

I totally agree with all your points, John!  Including the "protection" point:  Joe Witt has suggested that we just call (classical) FBP "FBP", and the other ones that have some of FBP's aspects but not all should be called "FBP-like" or "FBP-inspired" - while I do like the latter, I can't see this terminology sticking!  For now, I am using "classical FBP" for want of a better term!   The "relational" example is salutary - I confess I didn't realize there were other variants out there.  When Ted Codd offered me a job to work on "relational" data bases, I thought I understood them!

Last minor point: I never could get my head around 0-capacity connections - is this what you meant by "buffering bounded by 0 packets"?  It seems to imply a very tight coupling...

Regards,

Paul

[John Cowan]

On Fri, Jul 5, 2019 at 1:13 PM Paul Morrison <paul.m...@rogers.com> wrote:

 

For now, I am using "classical FBP" for want of a better term!

Fair enough.

 

The "relational" example is salutary - I confess I didn't realize there were other variants out there.  When Ted Codd offered me a job to work on "relational" data bases, I thought I understood them!

The usual SQL tables are not relations in Codd's sense at all, both because the columns are inherently ordered (by SELECT * and in other ways) and because the rows can be duplicated and have inherent order.  Codd's relations identify attributes (columns) by name and tuples (rows) by value.  Except for a small body of people who want to change things, these non-relations have completely taken over the term "relational".

Last minor point: I never could get my head around 0-capacity connections - is this what you meant by "buffering bounded by 0 packets"?  It seems to imply a very tight coupling...

I understand them as an abstraction of Ada's rendezvous operation.  If a process wants to send or receive and its partner is not already waiting, it waits until both are waiting.  Then the packet is transferred and both processes carry on.  There is a ridiculously simple program that deadlocks if there is 0 capacity between its two components but not if the capacity is 1 or more.  Process A writes a value to the shared connection and terminates (but with 0 capacity it has to block).  Process B reads with timeout from the connection, passing either the value read or a timed-out indication to its downstream.  If the timeout hits, then A will block forever, since B will never receive anything.

(In the original implementation of JavaFBP, attempting to write to a closed connection would immediately terminate the writing thread, a feature copied from Unix pipes.  The rationale is that if the downstream process has closed you, there is no point in your doing any further work, since nobody cares.  I don't know if that feature is still present.)

This example comes from a paper at <https://songlh.github.io/paper/go-study.pdf>, which is a study of concurrency bugs in Golang, which implements the process-and-connection model of classical FBP.  I haven't read it all yet.

John Cowan          http://vrici.lojban.org/~cowan        co...@ccil.org

Assent may be registered by a signature, a handshake, or a click of a computer
mouse transmitted across the invisible ether of the Internet. Formality
is not a requisite; any sign, symbol or action, or even willful inaction,
as long as it is unequivocally referable to the promise, may create a contract.
       --Specht v. Netscape

John Cowan          http://vrici.lojban.org/~cowan        co...@ccil.org

Business before pleasure, if not too bloomering long before.
        --Nicholas van Rijn

[Tom Young]

Last minor point: I never could get my head around 0-capacity connections - is this what you meant by "buffering bounded by 0 packets"?  It seems to imply a very tight coupling...

In Go, zero buffered channels will cause the sender to wait until the the channel is read by another goroutine.  Then the IP can be copied from the sender's storage into the receivers's storage and both can continue.   (The Go developers strongly recommend copying by value, not reference, in most cases.  'Do not communicate by sharing memory; instead, share memory by communicating.'  is their motto.  Surprisingly, this is often more efficient, in Go anyway.)

The routines(above) will not block, if and only if, they employ the Go 'select channel'  possibly with a 'default' clause.    'Select channel '  can be used for multiple channels, possibly both sending and receiving.  Go will process the first available channel.    The optional default clause allows for other operations to proceed when all channels are blocked.

Buffer values greater than zero may reduce blocking (and possibly prevent deadlocks) but require additional memory and cpu cycles. 

Go not only provides for automatic deadlock detection and reporting; and also for manually(--race option) requested limited race detection.  The limitation is mainly that an actual race must occur for Go to detect it.   Non-reentrant Go language library code(mostly OS related I think) is identified and confined as 'unsafe'.   

   

Empty channels are sometimes closed to signal significant events (like termination) with no data ever sent or intended to be sent. 

Hope this helps. 

Tom Young
47 MITCHELL ST.
STAMFORD, CT  06902

When bad men combine, the good must associate; ...
  -Edmund Burke 'Thoughts on the cause of the present discontents' , 1770

On Fri, Jul 5, 2019 at 1:13 PM Paul Morrison <paul.m...@rogers.com> wrote:

--

You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/flow-based-programming/5d4e690b-a44a-4f1e-916c-f8a9fc8e0490%40googlegroups.com.

[Paul Morrison]

Hi John, I hadn't realized that about "relational"!  Interesting!

In both JavaFBP and C#FBP, "send" returns "false" if the downstream port is closed.  In this case the component has to dispose of the packet some other way (discard, attach, etc.).   I followed this for the other implementations as well.

Cheers,

Paul

[ern0]

> In Go, zero buffered channels will cause the sender to wait until the the channel is read
> by another goroutine. Then the IP can be copied from the sender's storage into the
> receivers's storage and both can continue.

If the processing is synchronous (the initiator clock message must be
processed within a tick, e.g. audio processing) or realtime (the
initiator message must be processed as soon as possible, and as quick
as possible, e.g. home automation), it _requires_ non-buffering
message handling.

"Classical FBP" is not a bad name, I like it. If you want a new one,
you should take one (max. two) of the properties of the architecture
characteristics, or, better the purpose/domain.
--
ern0
dataflow evangelist

[Vladimir Sibirov]

Darren,

Speaking of Node-RED, NoFlo, and RESTful services, you might be interested in https://github.com/noflo/noflo-assembly which implements a similar "msg" approach in NoFlo. It still employs multiple inports for purposes such as configuration and joining of streams.

Dear all,

As for classification of NoFlo, with the redesign of its Process API (https://noflojs.org/documentation/components/#the-process-function) it ended up looking more like in classical FBP: a process function is responsible for checking if the data is available and reading it from input buffers. But technically speaking, it is still FBP-inspired, because underneath that API lies good old JavaScript's event loop, so the activation of the processes is still decided by the scheduler rather than by the processes themselves. Unless you use some third party implementation of threads or MsgFlo to put the nodes in different threads, apps or machines. That also means, there is no back pressure. And packets can be dropped implicitly.

I think, taking the name back to its origin and using FBP instead of "classical FBP" and calling other systems FBP-inspired is a good idea and would be fair to its creator. Probably it's time to update https://github.com/flowbased/flowbased.org/wiki accordingly, emphasizing essential characteristics of FBP systems and how FBP-inspired usually divert from them. Some common sense is needed though, because anathematizing anyone who violates a single rule of FBP would end up in this already quite fragmented community falling apart completely. While it takes lots of time and practice to realize that most of those principles matter altogether.

Best regards,

Vladimir

сб, 6 июл. 2019 г. в 16:36, ern0 <er...@linkbroker.hu>:

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/flow-based-programming/CALhephT%3DCbm16ZGeOxRJ6_OONLj8Cf0jB_yrK4G2j01g808%2BOw%40mail.gmail.com.

[Darren Cruse]

@Vladimir thanks just by coincidence I'd stumbled on noflo-assembly earlier today before seeing your message.  I did see the similarity to node-red and liked the simplification I've found noflo harder to get my head around (not sure why though they could definitely use more/better tutorials).

(my ideal would be a merger of node-red and noflo :) I like node-red's simpler model - even the way the nodes are configured - and I like the look of their editor - but noflo's support for running in the browser is awesome and stuff like msgflo looks gr8 too)

[Paul Morrison]

@Vladimir, would I be out of line in thinking that you have some plan to tweak the wiki as you describe?    No pressure... but anything you could do to clarify the different lines of thinking would be fantastic!

Regards,

Paul

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-programming+unsub...@googlegroups.com.

[Darren Cruse]

This is a bit old now but thought I'd apologize to this thread as I've recently learned that what I wrote below was more what I assumed was true about node-red than what was actually/precisely true.

Turns out they have a new release which only now is making the message passing between nodes truly asynchronous in the way I'd assumed had been the case all along.

https://nodered.org/blog/2019/08/16/going-async

Most interesting to me is that (I *think*) they're doing this change because they're trying to make out-of-the-box node-red better able to support distributing a single graph across multiple devices for easy IoT/cloud type of scenarios.  So similar to what this project did (see slides 11/12/13) but I believe they had forked the node-red code base to do it:
https://www.slideshare.net/MichaelBlackstock/wo-t-2014-blackstock-2

[Matthew Lai]

Sync or async you still need a device-to-device message transport mechanism to relay the message from one computer to another computer. For my tcl fbp implementation I use TCP/ IP to handle node A (machine 1) to node b (machine 2) connection and communication. Not sure if node red already supports TCP/IP.

[Paul Morrison]

Hi Matt,

I just came across https://github.com/samuell/awesome-fbp by Samuel Lampa - I am wondering if you have (or could write) a description of your TCL implementation and add it to his web page - you can do it via a Pull Request...

Thanks, and best regards,

Paul

--

You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flow-based-programming/5dfce635-ace0-4d5f-a675-283af47b6d5c%40googlegroups.com.

[Paul Morrison]

Hi Darren, just reread this post, and wanted to make two points about "patterns": 

a)  I don't like their pattern "6" : generally there is no guarantee that both of the paths will have the same number of information packets (IPs) - one side might have millions, while the otehr side has none... for a given input IP.  In FBP we generally recommend that people not split streams if they think they might want to recombine them later.  This is in fact one of  the patterns that our users are taught to recognize as deadlock-prone - "divergent/convergent"

b) FBP does yield many characteristic patterns.  I wrote a paper on this a few years ago - it can be found at https://jpaulm.github.io/fbp/morrison_2005.htm - apologies for the rather primitive formatting! :-)

[John Cowan]

On Wednesday, July 3, 2019 at 10:42:05 AM UTC-4, Darren Cruse wrote:

 

To your broader comments about the mind-set shift of FBP and the differences between the classical-FBP versus FBP-inspired systems I admit I'm still getting my head around such differences (e.g. the title of my OP).  I've been been looking at these FBP systems partly thru the lens of having looked at and played with other systems focused on concurrency (which I agree are definitely a mind-shift) e.g. I've been thinking of these graphical flow diagrams as most similar to things like Actors (Erlang/Elixir, Akka) or CSP (go i.e. goroutines and channels, clojure's core.async, js-csp).  But depicted graphically obviously.

The easiest way to understand the difference is to take the point of view of the component programmer.

In classical FBP, all components are autonomous.  They can read from any of their input ports whenever they want to, and can write to any of their output ports whenever they want to.  If the input port is empty or the output port full, the component waits transparently until things change.  This is a model familiar to all programmers, because it is exactly how files are processed.  A program does not sit in an event loop waiting for the OS to push the next block of a file at it, or send an event that says "Write your next block to the output file now".

In FBP-inspired systems, typically any component can write to a port whenever it wants, but components only have a single input port and sit waiting for a packet to be pushed to them.  This behavior allows a single-threaded implementation of the whole system, but each component is controlled by its upstream partner(s) rather than being autonomous.  This pattern has become very familiar to GUI programmers, whose programs typically are event loops because they need to be responsive to unpredictable user actions, but the program has to be turned inside out (an "inversion of control") which makes certain natural techniques like recursion difficult or impossible.

John Cowan          http://vrici.lojban.org/~cowan        co...@ccil.org

Is not a patron, my Lord [Chesterfield], one who looks with unconcern
on a man struggling for life in the water, and when he has reached ground
encumbers him with help?        --Samuel Johnson

Maybe I'll appreciate more after I do your tutorial (still on my todo list) - but I guess I was fuzzy on some of your points e.g. all of these systems are to some degree a "simulation" of the message flows to the extent they use different implementation strategies under the covers isn't that true?  so threads in java, "lightweight processes" in erlang, "cooperative multitasking" in node.js...

In my OP when I asked about the "programming model" I was thinking/wondering - independent of those implementation details - how the way to think about the graphical flow does/doesn't differ.

I know one point you've made is whether a message is truly a single thing (like a real world object would be) as opposed to like a pointer passed potentially to two nodes/processes that now (magically in the real world) see two "copies" of what was really one message.

I'm still getting my head around other differences.  I can see the concurrency model (e.g. true threads in java versus "cooperative" in node.js) has ramifications on being able to handle like high volumes leveraging multiple cores (streams of audio or what not), but those use cases aren't big ones for me personally I'm more dealing with lower volume things (REST apis i.e. i/o bound not cpu bound).  I'm not sure if that aspect is part of what you're referring to in "classical-FBP" versus "FBP-inspire" or not (to my thinking that's more "implementation" and not part of the "model" but maybe I'm wrong?)

Anyway interesting stuff apologies if I wrote too much. 

On Tuesday, July 2, 2019 at 2:09:53 PM UTC-5, Paul Tarvydas wrote:

Darren, thanks for the explanation.

When you "send" (deliver) a message to a node-red component, does the caller need to wait for the called component to finish its processing and for the called component to execute a RETURN?

pt

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/flow-based-programming/68181c19-90c9-449b-9a94-83059c53fed1%40googlegroups.com.