splitting output to many inputs

[Paul Tarvydas]

This is a backup demo (no sound) of something I presented at ECLM’09.

At the 9-second mark, one can see how I like to think about UI design using FBP-like thinking.

Note that the “clear” line visits every button on the diagrams, the point being that when one button is pressed, all other buttons must “unpress” themselves visually.

This is an example of where I think that output to many inputs is necessary. Adding “clone” parts would mess up the diagram, IMO.

Is there a different way to approach this problem?

https://www.youtube.com/watch?v=8vZ8Pi32oMo

pt

[Paul Morrison]

Hi Paul T.,

Strangely enough, similar examples keep cropping up when people discuss FBP, where people try to mirror something in the real world with an FBP network.  In most cases, my experience is that some kind of table is usually a better way of handling this.

Suppose we just had a global-ish table, describing the buttons: status, action, etc., then it is it trivial to switch the others off when one is switched on.  It also becomes easier to add more buttons, names, colours, more actions, etc.  One process could construct the table from, say, a relational database, then put a reference to it in a named global field.  Other processes can then access it in read-only mode - just make sure that they don't access it until the creating process has finished!

You'll also find your network becoming vastly simpler.  :-)

Regards,

Paul M.

pt

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Humberto Madeira]

Hi Paul,

On Wednesday, 18 May 2016 12:17:33 UTC-4, Paul Tarvydas wrote:

This is a backup demo (no sound) of something I presented at ECLM’09. 

This is an example of where I think that output to many inputs is necessary. Adding “clone” parts would mess up the diagram, IMO.

This is in accordance with my own approach.  I have fan-out capability on any and all outputs.

Is there a different way to approach this problem? 

To make it work reliably, the IP`s should be immutable (again, as per my own approach)

Of course, not all programming languages support immutability, but most of the major ones have support for it or will soon enough

(except for C/C++ since they don`t prevent casting out const)

With regular FBP, I suppose you could try one of the following

a) cloning of some sort (as you mentioned), or

b) a notification in some sort of specialized FireEvent component 

that would fire out an (immutable) event to a bunch of event listeners without going through FBP, or

c) you could also just set up the receiving components to retransmit after they receive but before they perform their task, or

d) just fire out the same IP (or clones of it) to multiple targets via an array outport

You might not be as averse to some of these alternatives as I would be.

Regards,

---Bert

[Paul Morrison]

I really don't understand why people are putting so much effort into providing this capability!  In 40+ years of FBP, I don't think I ever cloned an IP, let alone a whole IP tree!  The point being that trying to reassemble the copies is absolutely a recipe for a deadlock.  And a lot of writer components have optional output ports, so you can often string processes together in a "string of pearls" pattern.  If you just want to log IPs going across a connection, just insert the logger in between the sender and receiver processes.

Lots of possible solutions, and no need for an expensive (and potentially dangerous)  piece of infrastructure...  :-)

If I were of a suspicious frame of mind, I would think many of you are still thinking procedurally, not data flow!  :-)

Cheers

[Humberto Madeira]

Hi Paul,

On Wednesday, 18 May 2016 19:14:58 UTC-4, Paul Morrison wrote:

I really don't understand why people are putting so much effort into providing this capability!

Because broacasting an output is something one naturally expects in an electrical diagram (very useful for control signalling).

And electrical diagrams are the most likely kind of diagram to be associated with flow (especially for programmers with hardware backgrounds)

 

In 40+ years of FBP, I don't think I ever cloned an IP, let alone a whole IP tree! 

Interesting - I was under the impression FBP already had such a component.  My mistake.

 

The point being that trying to reassemble the copies is absolutely a recipe for a deadlock. 

Please explain how that relates. What does cloning or broadcasting an IP have to do with reassembling the copies?

 

And a lot of writer components have optional output ports, so you can often string processes together in a "string of pearls" pattern. 

I believe that was my option c)

 

If you just want to log IPs going across a connection, just insert the logger in between the sender and receiver processes.

I`m not sure how this relates either.  Could you please clarify?

Lots of possible solutions, and no need for an expensive (and potentially dangerous)  piece of infrastructure...  :-)

True enough, when you can go ahead and just modify the code in any component, 

such as in the string of peals example, to add in the retransmit feature, if it wasn`t there before...

But if a user has go into the component code to tweak it just so it fits the structural needs of the dataflow, 

then IMHO we don`t truly have black box reusability...

As for my own situation, since all my transmissible data items are already strictly immutable, the cloning issue is moot. 

I just broadcast the data items directly if I need to broadcast at all.  

No expensive or potentially dangerous cloning necessary :-)

 

If I were of a suspicious frame of mind, I would think many of you are still thinking procedurally, not data flow!  :-)

Hmm, I am completely unaware of anything I have said or done that could give that impression.

May I be enlightened as to what observation led you to such a belief?

Thanks and Regards,

--Bert

 

[John Cowan]

Paul Morrison scripsit:

> Lots of possible solutions, and no need for an expensive (and
> potentially dangerous) piece of infrastructure... :-)

The real problem with splitting isn't with matching up again, it's
with the splitting process itself. A classical Fanout component can't
reliably read one port and write to many, whether it copies IPs or not,
because if it gets stuck writing to one of the output components because
its input buffer is full, the whole Fanout component grinds to a halt.
This is a general problem with any classical component with multiple
output ports, but it's especially bad in this case.

--
weirdo: When is R7RS coming out?
Riastradh: As soon as the top is a beautiful golden brown and if you
stick a toothpick in it, the toothpick comes out dry.

[Paul Morrison]

Of course!  That makes a lot of sense!  Also, in JavaFBP, IP trees are not sufficiently self-describing that infrastructure can automagically clone a whole tree!  That could be fixed with a new set of chaining API calls, but... why bother?!

Bert, I would never suspect you of thinking procedurally!  However, I can't seem to grasp the attraction of immutable IPs!  Suppose that one IP is travelling down two paths in parallel (which you allow) - then I get that that IP is never actually modified, but if one path does a copy/modify new copy/delete old copy procedure, pragmatically that means that one path sees the old data about a customer, say, and the other path sees the new values (in two separate IPs).  I still think it's dangerous!  Maybe we will have to agree to disagree!

And Bert, I have never written a cloner for anything more complex than an IP containing one string - and that was just to do some performance tests...

Regards,

Paul

[paul tarvydas]

Correct.

Bounded buffers add too much complication.

pt

[Humberto Madeira]

Hi Paul,

On Thursday, 19 May 2016 21:29:05 UTC-4, Paul Morrison wrote:

Of course!  That makes a lot of sense!  Also, in JavaFBP, IP trees are not sufficiently self-describing that infrastructure can automagically clone a whole tree!  That could be fixed with a new set of chaining API calls, but... why bother?!

Bert, I would never suspect you of thinking procedurally! 

:-) 

 

However, I can't seem to grasp the attraction of immutable IPs!  Suppose that one IP is travelling down two paths in parallel (which you allow) - then I get that that IP is never actually modified, but if one path does a copy/modify new copy/delete old copy procedure, pragmatically that means that one path sees the old data about a customer, say, and the other path sees the new values (in two separate IPs).  I still think it's dangerous! 

OK.  I think I see here the reason for the confusion.  I believe it comes down to a difference in basic assumptions.

It all boils down to what the definition of an object is, and to what you think you are passing in the data flow.

The way I see it, what most end users would think of as an object are what Richard Hickey would refer to as an Entity.

Something that has both identity and state, and can be modified.

If you are interested in Richard Hickey`s comments about entities (and other things) he has a video and there is also a transcrpt available if you prefer to read it.

With Richard Hickey`s approach, the data you pass between calls are not entities, but entity snapshots instead.

Entity snapshots represent an entity at a specific point in time.  As such, snapshots are immutable.

So when you "modify" a snapshot, you create a new snapshot - not of the entity at a point in time, but of the entity at a possible new point in time.

To actually change the entity, you need to somehow update the entity with the state from a snapshot.

So my "IPs" aren`t entities - they are just snapshots, value aggregates or simple values.

They are all just (immutable) values.  That is why I don`t call them IPs - they do not fit the definition.

If you duplicate a snapsnot - and "modify" one of the duplicates (actually replace it with a different snapshot)

Then they represent two different potential future (what if) versions of the entity.

It is up to you to decide which one becomes real, or possibly both could be real, just at different points in time.

There is never any possibility of conflict between different snapshots, even if they are of the same entity.

In the JavaFBP approach.  The object being passed (the IP) can be an entity (although I would suspect that generally it is not). 

The rule that applies is that the IP is supposed to be owned only by the process of the component that it is being processed in,

and such ownership transfers as it flows between components.

I thought originally that this rule was intended to preserve the visibility of the IP to changes 

(so that it couldn`t be changed by two components at the same time) 

But now I believe it was intended to preserve the integrity of the current state of the entity being passed 

(in other words - to prevent multiple versions).

I`m not sure you remember, but in one of the FBP meetups, I asked if the IP being passed was attached to something outside the flow.

And you answered with the above rule.

For me.  An entity exists in a particular location.  I has an identifier, and lives in a container of some sort (a database, a memory cache, the system itself)

You can`t take the entity out of its container without changing its discoverability, and as far as the rest of the system is concerned, 

that effectively destroys the entity (its identity) and replaces it with another..

So for me. If an object is owned purely by the flow process - then it can`t possibly be a usable entity - because it is not discoverable via its identifier and container.

And if that is the case, whether the object is immutable or not, then the object must be just a snapshot.

In which case, you don`t need to worry about entity current-state integrity - because it is not an entity - just a snapshot - it is OK to have multiple versions and copies.

I guess that I`ve internalized this approach so strongly that I think I have finally completed my transition out of object-think.

It`s a victory of sorts... but it makes communication difficult with (the vast majority of) other programmers who haven`t yet made the transition.

I suppose an entity could be transmitted through a flow and changed within the flow 

as long as you first removed it from the original container and placed it back at the end.

The downside is that the entity would temporarily be unavailable for any other purpose while it was in transition,

but I suppose it could be acceptable if no activity was expected against it.

With the kind of applications I tend to work with (mostly distributed and web-based) 

it is unacceptable to make an entity unavailable for any period of time, so I tend to push that view of things.

In fact, given my focus on distributed programming, I have a hard time even understanding where it could be considered acceptable...

I tend to think of it as an error - or at least something that is potentiallty dangerous.

 

Maybe we will have to agree to disagree!

 

Possibly so.   I already suspected my approach was too divergent to be compatible with classic FBP.

But I would hope there is still some commonality of approach.

I would appreciate hearing your thoughts.

On Thu, May 19, 2016 at 8:20 PM, John Cowan <co...@mercury.ccil.org> wrote:

Paul Morrison scripsit:

The real problem with splitting isn't with matching up again, it's
with the splitting process itself.  A classical Fanout component can't
reliably read one port and write to many, whether it copies IPs or not,
because if it gets stuck writing to one of the output components because
its input buffer is full, the whole Fanout component grinds to a halt.
This is a general problem with any classical component with multiple
output ports, but it's especially bad in this case.

I don`t see where the splitting problem is any different than the general problem with multiple output ports.

If a component is meant to be a black box then it doesn`t matter what caused the downstream blockage.

You still have to handle the situation.

Regards,

--Bert.

 

[Paul Morrison]

Don't understand!  IMO bounded buffers are key to classical FBP!

pm

[Paul Tarvydas]

I can see why you don’t understand :-).

I mostly misunderstood John’s comment.

pt

[Armando Drummond]

May I came in?
I would like to introduce to you the "sks", a socket server, that reads standard input flow and broadcast it to all that are connected to server tcp raw port. It also sends to standard output all flows received from client connections. As our 17/07/2014 post "A Linux framework to FBP"  explains, the flow must be a sequence of lines (each line is a message). The reading of standard input is line buffered (fgets) and the broadcasting to remote connections is unblocking in order to avoid the halt problem mentioned. The incoming flow from connected clients is buffered in order to send to standard output a not mixed line buffered flow. Connections can be done to make traces and logs and, mainly, to allow many applications process the same flow. 

Armando Drummond

www.labcomsistemas.com.br

tel 19-3579-9331 (Fixo)

cel 19-99648-8938 (TIM)

[Humberto Madeira]

Hi Armando,

Welcome to the discussion.

I am not sure I am understanding you correctly.  

Please bear with me as I ask some clarifying questions.

1) You say that the input feed is line buffered. How is the buffer cleared?

a) line-by-line as each line is marked consumed by all feeds, or

b) once all the feeds are done (in other words - the entire input stream is cached)

2) how is the broadcasting done?

a) an actual broadcast to all receivers, or 

b) individual feeds

3) what happens when only one of the upstream receivers needs to stop for a while?

4) how do you handle an infinite input stream?  Or, at least, one which is bigger than all of memory + storage?

Thanks and Regards,

--Bert

[Armando Drummond]

Hi Humberto,

Thank you for your questions.

1)Line buffered is one of buffering options in Linux. Try: "man setvbuf" in Linux.

Sorry, I was wrong, the standard input feed is not line-buffered. When there is something to be read in standard input, it is read and write, the same content, to all connections. This writing is unblocking, via fcntl O_NONBLOCK flag.

2)As explained above, the answer is a.

3)I don't know but the others keep going.

4)In our applications all streams are infinite, let me give you an example:

Suppose you have a serial port ttyS0, to broadcast it do:

$ sks -p 3000 <ttyS0 >ttyS0 & #tcp port 3000

To log it do:
$ cat </dev/tcp/localhost/3000 >log &

To process it do:
$ procserial </dev/tcp/localhost/3000 >&0 & #this work to send the program output do tcp connection.

All this goes forever (you have to do something about the size of log file).

Up to now, line buffering was not important, but suppose you want to mix flows (the opposite of broadcasting flows). Let's say:
$ p1 >_p3 & #process 1
$ p2 > _p3 & #process 2
$ p3 < _p3 & #read named pipes _p3

As messages are lines, it need to be assured that lines are not mixed. For this, the output in p1 and p2 must be line buffered and also the input in p3( using fgets).
And finally, the "sks" have a broadcast side (from stdin to clients) and a mixing side (from clients to stdout) and this mixing side must be line-buffered.

Do I make myself clear?

Best regards,
Armando

[Paul Morrison]

Thanks, Bert, things are becoming clearer!  I should take a look at Richard's writing, but I have a pretty good idea of what I mean by "IP" - we used to call it "entity" too, but realized this was causing confusion.  Think of it as a bottle in a bottling plant -  in the plant it travels through various stations, being filled, capped, labelled, etc.

Your concept is very similar to what one of my mentors, Nate Edwards, used to call the "accountant's view of data", which he first told me about in the early '70s, I believe.  This holds that a database is a snapshot of the data at some point in time - like "as of" on a balance sheet.   The underlying reality is the transaction log, which cannot be changed - it can only be added to!  If you want to delete a record, you add a transaction stating that it has been deleted; if you want to reverse a transaction, you add a transaction reversing the old one.   And of course every transaction has a date and time stamp.  So theoretically, given enough capacity and computing power, you would be able to wind a database backwards or forwards in time.  However Nate did not see this concept as applying within a program, but at the database level... just as a bottling plant does not have future bottles or past bottles - it only has bottles.  If a bottling plant had a machine that could make a copy of a bottle (unlikely), it would be a different bottle!

So I think your concepts are orthogonal to FBP, applying at a different level and applying rather to states of a database, not to processing within a program.  It seems to me that there is a role for both sets of concepts in future computing - it will take someone brighter than me to weld them together seamlessly!  :-)

Thanks for the good explanation,

Paul M.

--

[Humberto Madeira]

Hi Paul,

On Monday, 23 May 2016 13:17:18 UTC-4, Paul Morrison wrote:

Thanks, Bert, things are becoming clearer! 

I am glad.  I was losing hope about the clarity of my communication.

I should take a look at Richard's writing, but I have a pretty good idea of what I mean by "IP" - we used to call it "entity" too, but realized this was causing confusion.  Think of it as a bottle in a bottling plant -  in the plant it travels through various stations, being filled, capped, labelled, etc.

Yes, I do see the bottle - but I vew it as multiple states of the bottle.  Each state a new snapshot.

 

Your concept is very similar to what one of my mentors, Nate Edwards, used to call the "accountant's view of data", which he first told me about in the early '70s, I believe.  This holds that a database is a snapshot of the data at some point in time - like "as of" on a balance sheet.   The underlying reality is the transaction log, which cannot be changed - it can only be added to!  If you want to delete a record, you add a transaction stating that it has been deleted; if you want to reverse a transaction, you add a transaction reversing the old one.   And of course every transaction has a date and time stamp.  So theoretically, given enough capacity and computing power, you would be able to wind a database backwards or forwards in time. 

Yes - I started from there - or something very likely derived from there. My additional realization is that 

1) not all the intermediate states need to be recorded in the entity store

2) aggregation of changes can be done as a succession of transforms and snapshots.

I also concur that if you store all the transitions (the important ones) you could replay it or even go back in time - it`s the notion of a versioned database or a VOB (versioned object base)

I picked up these concepts in the mid-90`s.  Various vendors were pushing that sort of thing back then.

 

However Nate did not see this concept as applying within a program, but at the database level...

To some extent, my framework has elements of a database engine (or at least, conceptually so)

One of my early studies involved reading the book on the O2 database engine (the only database engine at the time for which I could get documentation for the design of the internals)

But the whole database vs application separation was due to the original two-layer client-server mindset.

Things have evolved since back then.  Multi-layer and distributed architectures are much more common now.

 

just as a bottling plant does not have future bottles or past bottles - it only has bottles.  If a bottling plant had a machine that could make a copy of a bottle (unlikely), it would be a different bottle!

So I had some exposure to processing plants, and in those environments, we always needed to describe or label what was passing between stations.

As features got added - the type of thing being passed can also change.

In OO terms, the rough log object becomes a wood chip object collection then a pulp object collection then eventually a roll of paper object.

And according to OO, each time the class of the object changes, you actually have a different object.

This is true even with bottles.  Empty bottles and bottles of pop are not seen as the same type of thing.

If the processing plant stops because of a power outage.  What you are interested in is the bottles that are finished and what kinds they are.

Also the raw materials. The WIP (work in process) will mostly just have to be chucked.  Or partially recycled.

So even with OO, the thing being passed isn`t necessarily the same object all the way through, even if it is actually the same thing in real life.

Unfortunately, OO is not necessarily a good model to describe the sorts of transformations that happen in a factory.

As per Alfred Korzybski - the map is not the territory.

To get a better model of a transformative process, I believe it is important to distinguish between entities and snapshots.  

The advantage of snapshots is that it is a way to decouple the effects of time.

 

So I think your concepts are orthogonal to FBP, applying at a different level and applying rather to states of a database, not to processing within a program. 

Actually it _is_ the processing of the program - just modeled as a series of transformations.  

It is very similar to FBP.  Also similar to an FSM (finite state machine)  Essentially, a combination of the two.

You can view a process as a series of transformations, or as a series of states.  It`s really the same thing.

So yes, I would concur that what I think of as the FSM part is somewhat orthogonal to the FBP part, but not really separate.

The effect is approximately equivalent to that of a fine-grained workflow engine.

It`s actually a very non-OO approach, so there might be some cognitive dissonance for pure OO practitioners..

 

It seems to me that there is a role for both sets of concepts in future computing

I certainly think so. :-)

I think OO (in the sense of class-oriented programming) has had a good run, but it needs to go away now in the interests of future progress.

I believe these concepts will help form at least part of the way forward.

 

it will take someone brighter than me to weld them together seamlessly!  :-)

I am trying to do that very thing (and also blending in one other additional major concept not mentioned here) 

Sadly, I am not always as bright as necessary, so it has taken a long time so far. :-(

And sometimes I get stuck along the way until something else comes along to unstick me. (I have been fortunate)

I do know for sure I wouldn`t have gotten as far as I have if I had tried to design it all at once.

As for seamless - let`s just say that it`s a work in progress.  I do have good hopes for the final result.

Regards,

--Bert

[Humberto Madeira]

Hi Armando,

On Monday, 23 May 2016 13:13:16 UTC-4, Armando Drummond wrote:

Hi Humberto,

Thank you for your questions.

1)Line buffered is one of buffering options in Linux. Try: "man setvbuf" in Linux.

Sorry, I was wrong, the standard input feed is not line-buffered. When there is something to be read in standard input, it is read and write, the same content, to all connections. This writing is unblocking, via fcntl O_NONBLOCK flag.

2)As explained above, the answer is a.

OK.  So this appears to be similar to what I do. I pump output directly to broadcasting and each receiver can pull its own data independently.

3)I don't know but the others keep going.

I have individual buffering for each downstream component feed (sort of).  If one stops, the others keep going.

If I restart that one.  It depends how much was buffered whether it can recover.

 

4)In our applications all streams are infinite, let me give you an example:

Suppose you have a serial port ttyS0, to broadcast it do:

$ sks -p 3000 <ttyS0 >ttyS0 & #tcp port 3000

To log it do:
$ cat </dev/tcp/localhost/3000 >log &

To process it do:
$ procserial </dev/tcp/localhost/3000 >&0 & #this work to send the program output do tcp connection.

All this goes forever (you have to do something about the size of log file).

Up to now, line buffering was not important, but suppose you want to mix flows (the opposite of broadcasting flows). Let's say:
$ p1 >_p3 & #process 1
$ p2 > _p3 & #process 2
$ p3 < _p3 & #read named pipes _p3

OK.  I think you mean to merge the flows.  

 

As messages are lines, it need to be assured that lines are not mixed.

Do you mean that you are streaming bytes, and you need to keep the lines distinct from each other?

 

For this, the output in p1 and p2 must be line buffered and also the input in p3( using fgets).
And finally, the "sks" have a broadcast side (from stdin to clients) and a mixing side (from clients to stdout) and this mixing side must be line-buffered.

Do I make myself clear?

I think the only remaining issue Ihave is withe the term line-buffered.

In FBP terminology, I think you mean to say that your IPs (Information Packets) are lines.

If so it would imply that your inputs are buffered to read in line by line (line processing) and your ouputs are buffered for line by line streaming.

Based on your comments abouthe broadcasting, I would then assume the lines are being output to a single stream which then broadcasts (I assume broadcast UDP/IP) to a bunch of listeners.

If all that is so, then I believe I understand.

So my expectation would be that I just fire line-delimited output via stdout at sks from one or more apps and the lines get piped via broadcast UDP/IP to the stdin of one or more apps via procserial agent listening on the broadcast port.

Would that be correct?

Regards,

--Bert

[Armando Drummond]

Hi Bert,

I'm not sure I understood your last sentence, so let me try another way.

Yes my IPs are text lines. Any IP is a sequence of bytes that can be encoded in a hexadecimal format and terminated by the newline character. 

From the Linux Programmers Manual: "The  three  types of buffering available are unbuffered, block buffered, and line buffered.  When an output stream is unbuffered, information appears on the destination file or terminal as  soon  as  written;  when  it  is  block buffered  many  characters  are  saved up and written as a block; when it is line buffered characters are saved up until a newline is output".

Another important characteristic of this solution is  atomicity of pipe I/O (The GNU C library). 

With these 3 things together (messages are lines, process read/write line buffered and atomicity of pipes) we can build a FBP framework (where flow are sequences of messages) in Linux.

Next came the problem of splitting output to many inputs and we build "sks", a TCP/IP raw socket server that reads stdin  and writes (flag O_NONBLOCK)  the read content to each and all clients connected. That is the principal idea.

Now that I'm trying to explain, I see how difficult is this task. Anyway we have many systems in production in a efficient and reliable behavior.

Best regards,

Armando

 

Armando Drummond

www.labcomsistemas.com.br

tel 19-3579-9331 (Fixo)

cel 19-99648-8938 (TIM)

[Humberto Madeira]

Hi Armando,

On Tuesday, 24 May 2016 15:08:47 UTC-4, Armando Drummond wrote:

Hi Bert,

I'm not sure I understood your last sentence, so let me try another way.

The last sentence was me trying to understand the mechanism you were using to actually broadcast.

If it is networking-based, there are two approaches that I am aware of.

1) UDP which can network broadcast to a range of IP addresses, but which is not reliable and has no back pressure.

(noting that it is possible to build a reliable framework on top of an unreliable one)

2) TCP which is point to point and reliable, and has back pressure.

But because TCP can`t broadcast (it is point to point) the actual broadcast distribution is in-memory somewhere.

Typically the second approach is implemented via sort of publisher-subscriber style messaging system,

possibly also including store-and-forward capability to improve reliability.

Typical examples include TIBCO, ActiveMQ, JMS etc.etc.

These systems usually store and forward using the file system, so they generally don`t have back-pressure support except by returning an error message (the sender must send again)

Anyways, I am just trying  to understand how your broadcasting part compares to these approaches.

(where it fits in) 

Yes my IPs are text lines. Any IP is a sequence of bytes that can be encoded in a hexadecimal format and terminated by the newline character. 

From the Linux Programmers Manual:

Ah. That might explain some of the confusion :-) 

 

"The  three  types of buffering available are unbuffered, block buffered, and line buffered.  When an output stream is unbuffered, information appears on the destination file or terminal as  soon  as  written;

Actually they mean: as soon as written to the output stream, noting that the output stream is just a buffer that in turn writes bytes to a socket on a separate thread (or possibly a separate process - given that this is Linux).

 

 when  it  is  block buffered  many  characters  are  saved up and written as a block; when it is line buffered characters are saved up until a newline is output".

Noting that there is no particular reason (IMHO) to do this saving if the output is a continuous stream (and not separate requests):

as long as the delimiters can be processed in order, the delimeter processing can be done equally well on either side of the socket

And you are going to have to stream the data though the socket anyway (assuming TCP), so I don`t see why you need to hold up the data in the sender.

The only need for output stream buffering is to allow the thread loading the output stream to release its data into the stream and proceed without blocking 

(the socket`s stream has its own separate thread) but any type of buffering will do.

BTW, in HTTP 2.0 there is now a push API that can send individual messages over a connection without having to reconnect.

(actually, even HTTP 1.1 can do this to some extent)  Essentially it is message-delimited (AFAIK)

 

Another important characteristic of this solution is  atomicity of pipe I/O (The GNU C library). 

Not sue what you mean by that.

 

With these 3 things together (messages are lines, process read/write line buffered and atomicity of pipes) we can build a FBP framework (where flow are sequences of messages) in Linux.

OK, with the caveat that you are referring to generic FBP and not classic FBP 

(these are the terms we agreed to use on this forum to distinguish between Paul`s original FBP and FBP in a more generic sense)

 

Next came the problem of splitting output to many inputs and we build "sks", a TCP/IP raw socket server that reads stdin  and writes (flag O_NONBLOCK)  the read content to each and all clients connected. That is the principal idea.

Now that I'm trying to explain, I see how difficult is this task.

Yes, I find that I often have the same problem.  Terminology is always suspect, specially between people of varying backgrounds.

And it is hard to describe things precisely.  I usually require several attempts at restatement.

Anyway we have many systems in production in a efficient and reliable behavior.

IMHO that`s the important part. :-)

Best Regards,

--Bert