Is FBP alive?

[Ralf Westphal]

I´m interested in knowing who of the members of the group is using FBP in day-to-day business?

Who is actually implementing production software using FBP?

Or who is actually using it at least in larger examples?

Or who is using it in teaching others how to approach software development in a different manner?

Or who has written about FBP within the past 6 months more than a single page/blog post?

Or who has presented FBP to an audience of more than 3-5 people (i.e. his/her own team)?

Or to be more blunt: Who has made any money by using/advocating FBP in the past 6 months?

I´m asking this because the more I listen to the conversations going on here I get the impression FBP is, well, dead. Dead in practical terms, meaning nobody is actually using it in a way he´s depending on. FBP seems to be a toy, a nice concept to discuss here - but without any impact on day to day business.

But of course I´d love to be proven wrong by many answers to my questions telling of how FBP makes to them a difference every single day.

Looking forward to them...

[ern0]

> Who is actually implementing production software using FBP?

O/ (it's a head and one hand raised)

> Or who has presented FBP to an audience of more than 3-5 people (i.e.
> his/her own team)?

I've done it a year ago at Budapest New Technology Meetup.

A friend of mine and me has developed a prototype of a home automation
system based on FBP. It's completelly written in C++ and it runs on
small ARM Linux devices like GuruPlug or Raspberry Pi. We're now kinda
suspended the project, two reasons:
- lack of $
- the system is a prototype, it needs complete rewrite to be able
control houses or anything safe way.

Anyway, my friend's house runs on the prototype system, he's working
on Wago/EnOcean driver (component), it has a f*d protocol. These
components will be the last ones for our prototype system.

The prototype system has no real dispatcher, we've no gfx editor, but
we have about 50 components, incl. UDP send/recv, MIDI(!), sound
output, web widget (the component's state appears on a web interface,
and also events in the browser causes message fire on the server
side), etc.

Currently we're working on the production system specification. Due to
lack of financial background, we're doing this in our spare time.
--
ern0
Haben Sie Fragen?

[Paul Morrison]

Great question, Ralf! I'm really interested in the answers - naturally!

I think there are also at least some people who are not members of the
group - Mike Beckerle (quoted several times in my book) is one who
springs to mind...

Regards,

Paul

--
http://jpaulmorrison.blogspot.ca/

[Tom Young]

Hi Ralf,

I did use FBP in business many years ago, before retirement.   That company no longer uses
FBP and seems to be foundering ( perhaps for unrelated reasons, but who knows).

I am forming my own business and fully intend to leverage FBP in the process.   It is difficult
to say exactly how without giving away the store.

One reason for seeming lack of FBP adoption is the requirement for some kind of infrastructure
to connect components into working programs.    That infrastructure needs to work on all modern platforms in order to be widely adopted.

The current Java efforts here are one solution to this problem.    To gain wide adoption, FBP needs to catch the next big wave, so to speak.  Java may just be the last big wave.   Maybe it's Google Chromium, Dart, node.js,  or something else.   Wish I knew.

          Cheers,

                      -Tom Y.

--
"...the shell syntax required to initiate a coprocess and connect its input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

[Paul Morrison]

Along the same lines, would you include products that are based on
similar concepts - or are you restricting your question to people using
the term "FBP"? If the former, what about products mentioned in my
book, such as Proto Software, Inforsense, Knime, Pentaho's Kettle,
expecco, etc....?

Regards,

Paul

[Tom Young]

Hi Ralf,

Forgot to mention that I presented my working FBP system to a group of ten or more several years ago.   The consensus seemed to be this was really great, but that a graphical editor was needed that could generate diagrams from the network definitions.   I now have such an editor, but am not entirely happy with the resulting diagrams.

Since then others have insisted that an FBP system must run on native M/S Windows(not Cygwin) to be acceptable.    

There are  many FBP implementations in use that may not be represented in this forum, so please don't take our answers as necessarily representing the entire industry.

While I was being paid to do software development, there was surprisingly considerable resistance to any form of re-usable modularity or componentry.   Every program was a monolith (outside of I/O libraries, etc.) and that was what developers and management were used to dealing with.   I assume that some remnant of those attitudes remain -- perhaps as 'Not invented here.' syndrome.

             -Tom Y.

--
Thomas W. Young, Founding Member
Stamford Data, LLC
47 Mitchell Street, Stamford,  CT  06902
Phone: (203)539-1278
Email: TomY...@stamforddata.com

[Brad Cox]

Buried but thriving. In products like those mentioned, plus Unix pipes, Yahoo pipes, RapidMiner, LabView.

--
Cell: 703-594-1883
Blog: http://bradjcox.blogspot.com
Web: http://virtualschool.edu
Manassas VA 20111

[Sam Watkins]

For me, FBP is very much alive, but maybe not quite born yet!

I think I'll try to start using FBP in my day job again.

I'm valuable enough to the company, they won't give me the boot
for using some less common programming techniques. (I hope!)

Really one of my life's ambitions is to create a ~FBP graphical
programming system. Using FBP and working on that project as part
of my day job would make it much more of a 'dream job' for me.

Thanks Ralf, for giving me a push in the right direction...!

I think the most practical approach (at work, for now) will be
to use UNIX processes as components, and pipes/sockets/files
for the channels. Most packets will probably be lines of text,
or chunks of key: value pairs delimited by a blank line.

There might be a certain performance penalty, but on the other hand
UNIX is very stable, has memory protection, multiprocessing,
language independence, and a wide range of existing tools;
Plain-text formats and protocols work well for most applications.

UNIX does provide most if not all of the stuff needed to program FBP.
I'll have to figure out how to handle signals, errors and process groups
properly, for a program composed of many sub-processes.

... I hope tomorrow I can report back, that I did earn a day's wages
with the help of FBP, and my existing arsenal of shell tools!

The biggest challenge for me will be to code it so that other people can
understand and maintain it. That's where a graphical editor would be
very good; or I could resort to commenting the code; or use lots of
sub-components with meaningful names (structured programming).
That might work best for now. If I get some sort of good style working
with it, I'll post some examples here.


Sam

[Paul Morrison]

This feedback from Mike Beckerle:

... the Apache Pig language is FBP on top of Hadoop.

I'm told "streaming hadoop" is happening. I will find out if it is real
over the next 6 months.

But, per his comments about active advocacy, I am not making a living
building FBP-based systems, nor in writing/teaching about it.

I do have a student (from former work at IBM) who is still trying to get
a paper published about a system named Ohua which is FBP +
fault-tolerance/guaranteed-delivery.

[Bob]

On Tuesday, July 3, 2012 7:26:50 PM UTC+1, Paul Morrison wrote:

> But, per his comments about active advocacy, I am not making a living building FBP-based systems, nor in writing/teaching about it.

I appreciate your continuing support for DrawFBP and engagement with this group.

To address Ralf's questions with one answer:

* I am trying FBP at work (flat file processing for error detection and for learning FBP).

* I'm saving some small examples and I provided a few text-oriented JavaFBP components. I aim to post the samples somewhere when I have a handful.

* I post to this group and provide a little support DrawFBP by ad hoc testing
* So far I do not have the implementations to promote this at work

[Sam Watkins]

Bob wrote:
> * I'm saving some small examples and I provided a few text-oriented JavaFBP
> components. I aim to post the samples somewhere when I have a handful.

I would be very grateful even for a very small 'hello world'
sort of runnable DrawFBP example, with the Java code needed... please!!

I guess I could figure it out myself but really it's a huge obstacle to
adoption, that there are/were no examples shipped with the system.
At least, I didn't see any.

Sam

[Paul Morrison]

Hi Sam, I put quite a longish answer up to your last (very similar)
query on
http://groups.google.com/group/flow-based-programming/browse_thread/thread/b7f9849dc6fdf8e5
- it starts "Sam, I do appreciate your frustration"... Did you not see
this, or is it not what you were looking for?

The diagram mentioned here -
http://www.jpaulmorrison.com/graphicsstuff/MergeSortDisplay.drw - is a
runnable diagram, containing only stock JavaFBP components. Perhaps if
you could read that post again, and then be more specific about what
seems to be missing...?

Thanks,

Paul M.

--
http://jpaulmorrison.blogspot.ca/

[Bob]

Sam - very brief starter for ten:

CONTEXT

Windows: Start > Run > cmd or (Windows-Key + R and then enter cmd)
C:\Users\Bob>cd \FlowBasedProgramming

C:\FlowBasedProgramming>prepare <-- this is a command file that I set up so that the Path & CLASSPATH start as below

C:\FlowBasedProgramming>set Path=.;C:\Program Files\Java\jre6\bin;C:\Program Files\Java\jdk1.6.0_05\bin;C:\Program Files\Java\jre6;C:\Program Files\Java\jre6\bin;...

C:\FlowBasedProgramming>set CLASSPATH=.;C:\FlowBasedProgamming\JavaFBP-2.6.jar;.;C:\Program Files\Java\jre6\lib\ext\QTJava.zip;C:\FlowBasedProgramming\JavaFBP-2.5.jar...

TEST by using java -version and javac to make sure you have 1.6 and you can see the compiler.

CREATE / GENERATE Java network

Use Draw FBP to draw a network or to open one of Paul's diagrams attached elsewhere in this group (AppKata thread or this thread perhaps).
File > Generate > Java and save the compiled code, say C:\FlowBasedProgramming\sam\whatever.java

COMPILE AND RUN

C:\FlowBasedProgramming>javac sam/whatever.java

C:\FlowBasedProgramming>java sam/whatever

CHECK OUTPUT

If you send to WriteConsole expect the output in your cmd session.

Good Luck/Good Night
Bob

[Sam Watkins]

Paul Morrison wrote:
> Hi Sam, I put quite a longish answer up to your last (very similar)
> query on http://groups.google.com/group/flow-based-programming/browse_thread/thread/b7f9849dc6fdf8e5
> - it starts "Sam, I do appreciate your frustration"... Did you not
> see this, or is it not what you were looking for?

Thanks Paul. Sorry I did not recieve that at the time, or I might have
accidentally deleted the thread before reading it. Thanks very much!

I will be having a look at your example, and Bob's.
I'll try to make some components and programs too!

[Ged Byrne]

Hi Ralf,

I don't use any of the FBP implementations like JavaFBP.  They're just no suitable for what I do.

However, I use the principles of FBP all the time to earn my living.  Reading Pauls' book allowed me to consolidate a whole lot of hard earned, poorly understood principles that I had been following for years.  I had a whole load of superstitions and folklore that I considered while designing.  FBP placed firmly into my grasp the things that I had long been groping for.

I design networks, primarily routes within Apache Camel.

Camel is a wonderful product, but very complicated.  It's hard to create good, clean routes following the Enterprise Integration Patterns.  EIP gives you some names to help you describe the complex mess you've created.  FBP gives you some guiding principles that helps you keep it clean and simple.

Is an explicit implementation of FBP paying my bills?  No.

Is a nice concept helping me earn a reputation for good design?  Yes.

Do you believe that nice concepts have no value?

Regards, 

Ged

[Sam Watkins]

> ... I hope tomorrow I can report back, that I did earn a day's wages
> with the help of FBP, and my existing arsenal of shell tools!

g'day again,

I haven't done much that's directly useful for my job yet, but:

I wrote 'net2sh' a small self-hosting FBP shell script 'compiler',

net2sh graph: http://sam.nipl.net/code/net2sh/g/net2sh.dot.png
net2sh source: http://sam.nipl.net/code/net2sh/net2sh.net
net2sh shell: http://sam.nipl.net/code/net2sh/net2sh

and a tool in it, 'net2dot' makes graphviz dot files for plotting.

net2dot graph: http://sam.nipl.net/code/net2sh/g/net2dot.dot.png
net2dot source: http://sam.nipl.net/code/net2sh/net2dot.net
net2dot shell: http://sam.nipl.net/code/net2sh/net2dot

There's one other simple test program, to count files in a directory.

test1 graph: http://sam.nipl.net/code/net2sh/g/eg/test1.dot.png
test1 source: http://sam.nipl.net/code/net2sh/eg/test1.net
test1 shell: http://sam.nipl.net/code/net2sh/eg/test1

It uses standard tools, some reusable tools, and some custom parts.
One of the net2sh components is another little FBP program (a sub-net).
Components are normal shell tools, and can be written in any language.

The idea is to run a bunch of programs (shell commands) in parallel,
connected by a bunch of named pipes, and wait for them to finish.
None of those namby-pamby coroutines this time around!!

To avoid deadlock, I wrote 'buf', a file-backed buffer component.
There is a simple tokenizer (and token re-joiner) in there, etc.

I haven't tried running anything very large in it.


Sam

[Chris Brody]

Looks nice. Can you add some more documentation, especially what are
the requirements of tools that have to be installed? Also do you have
a license in mind for this or do you just put this out as public
domain?

[Paul Morrison]

On 05/07/2012 5:22 PM, Xenogeek wrote:
>
> I would deconstruct this into four nodes: A schedule_node, a
> holiday_node, a database_extract_node, and an email_sender_node.
>
> Schedule -> Holiday -> Extract -> Sender
>
> The point here is most would put the holiday functionality in with the
> schedule_node. When in reality it makes better sense to separate them
> as the required logic is cleaner and simpler if the schedule did not
> have to worry about holidays and the holidays did not have to worry
> about schedules. The holiday_node is coded to check if 'now/today' is
> a holiday and if so then throw away the message else pass it on
> down-stream. This is 'single responsibility' at the right granularity.
>
Your point about holidays is well-taken. I worked as a contractor on a
(non-FBP) job scheduling system, and holidays are a real rat's nest!
Unlike your schedule node, the holidays node has to be aware of
locality. Here in Canada we have federal, provincial, and municipal
holidays, and two official languages, so it makes sense to encapsulate
this logic into a self-contained, table-driven component (or subnet).
Things get even more complicated if your application has to worry about
holidays in two locations, e.g. brokerage apps.

I'd also like to highlight the 3rd paragraph quoted above - it takes
experience to know/sense/intuit that this piece of the logic should be
separated out. It's better to be too granular initially, rather than
not granular enough - it's very hard to unmix a martini! Perhaps this
is why I'm so happy to hear that there are people out there (even if
only a few) who are using FBP (and not necessarily even FBP software, as
Ged has pointed out) to build real-life, industrial-strength
applications. As I've said many times before, you cannot give someone a
pile of girders and a book of instructions, and tell them to go build a
bridge!

[Paul Morrison]

On 05/07/2012 5:22 PM, Xenogeek wrote:

> a flexible and dynamic platform is required that can distribute the
> "black-boxes" that make up a workflow across processes, servers, and
> clouds. Not only will these black-boxes need to be deployed and
> managed, but re-arranged and scaled-up and scaled-down to most
> efficiently use resources.

One other aspect of this that hasn't appeared on this thread is the
ability to multiplex processes to take advantage of multiple cores. What
was for me a fascinatingexample is the database scanner that I wrote for
a bank, where the run time was reduced by 85% by simply adding one
(general purpose) component, and reconfiguring the network. Note that
this couldn't have been done if the logic hadn't been granularized
appropriately into the logic nodes that Xenogeek talks about. Even more
importantly, this "granularization" (ugh!) was not done with eventual
multiplexing in mind! It was just a sensible way to decompose the
problem. This process (from initial design to final running network) is
described in the chapter called "Performance Considerations" (Chap. 21
in the 2nd edition - fig. 21.1; Chap. 22 in the 1st edition - fig. 22.1).

There has been quite a bit of discussion on this group about wanting to
be able to modify FBP networks dynamically. It would certainly be nice
to be able to vary the multiplexing factor in such applications at run
time, but I want to stress that in JavaFBP and C#FBP the multiplexing
factor can be modified *prior* to running the network, just be assigning
a value to a variable - and you don't have to worry about stranded
information packets IPs)! This capability is well worth experimenting
with even if you don't have all the bells and whistles you might hope
for! http://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good
(Voltaire, apparently).

[Doug]

Paul,

Our platform is written in Ruby and there  are 38,000 GEMS that can be woven into 'nodes' (our name for the black-boxes) one of which is a holiday GEM. It has both Canadian and US holidays. I am in Canada as well and yes the holidays need the locality.

Incidentally, we ended up using workflows internally to build the system as it gave us the most flexible mechanism for extending the Platform itself. We were trying to stay true to the FBP principles while still using the best approach to get the job done. Time and again the best way ended up being the workflow approach.

It is a neat thing when you can build a product with itself.

One of the biggest advantages of a FBP approach is the code you don't have to write. As the cost of software is directly related to the interdependencies of the logic, there is a big advantage in the ability to code in 'isolation'. In that the separation of responsibilities reduces the problem scope to a very small use-case. Development and maintenance time drops accordingly and since points of failure are typically at a single node it is easy to trace a problem even if it was generated up-stream by a different node.

  

Cheers,

Doug

[Paul Morrison]

On 06/07/2012 2:32 PM, Doug wrote:
> Our platform is written in Ruby and there are 38,000 GEMS that can be
> woven into 'nodes'

How do they keep track of 38,000 gems? Just curious!

[Sam Watkins]

Chris Brody wrote:
> Looks nice. Can you add some more documentation, especially what are
> the requirements of tools that have to be installed? Also do you have
> a license in mind for this or do you just put this out as public
> domain?

hi Chris, thanks for your interest.

I'll write some more sample programs and doc, and post back.
There is a list of tools that are needed in the README.txt,
but I might have missed a few. It's not very polished yet,
doesn't handle error conditions very well. All my projects
are public domain. I'll add new features as I need them.

The first app I'm going to attempt to write with it will create
Excel spreadsheets from CSV data, using a template in XSLX format.
It's something we need to do at my work, to make customers happy.


Sam

[Paul Morrison]

Hi Sam,

Could you change the subject back to "Re: Is FBP alive?" in your posts
to this topic - or start another topic, specifically about net2sh?
Google keeps changing the subject of this discussion to "Re: net2sh",
and it makes it look as though there are 22 posts, all about net2sh!

TIA,

Paul

--
http://jpaulmorrison.blogspot.ca/

[Sam Watkins]

Paul Morrison wrote:
> Google keeps changing the subject of this discussion

That's very silly of it!
Ah well, I'll keep that in mind and use new threads.

It's a pity, it can be useful to change the subject in mail, makes it
much easier to see what each mail is about. I thought people are just
being lazy when they leave the subject the same for each post! We have
threads for grouping, it's not based on subject.

My evil plan is progressing, to introduce shell-based FBP at work!

I might switch to something else like JavaFBP or my C coroutines
at some point, the machine is creaking at the seams with all the
processes I'm launching!

Well, I can live with a bit of delay starting up.
It should work fairly well for large data sets once it gets running.
Now I'm off to convert CSV files into something more sensible.


Sam

[Paul Morrison]

Of course, it did it again! Maybe this will be the last time...?

Can/will you start a new topic - I am sure you will have more to say
about net2sh! I was able to start a new topic successfully once - I
could try if you want....

--
http://jpaulmorrison.blogspot.ca/

[Paul Morrison]

Excellent, Sam! "Is FBP alive?" is still alive, and net2sh has its
own topic! Looking forward to seeing how it develops!

Paul

[Doug]

http://rubygems.org/

It looks like its actually 41,320...

[Vladimir Sibirov]

Hi Doug,

That's impressive!

Could you give an example how Ruby gems can be used as nodes of a cloud application? Could you also tell us how you got to the idea of using FBP to build the system and what is different in your current workflow approach from FBP?

2012/7/10 Doug <do...@nodally.com>

[Paul Morrison]

The question remains: how do they keep track of 41,320 gems?!  I.e. repository, standard documentation, search engines, etc....?  You can't know all the names by heart, surely... :-)

[Sam Watkins]

Doug wrote:
> Our platform is written in Ruby and there are 38,000 GEMS that can be
> woven into 'nodes' (our name for the black-boxes) one of which is a
> holiday GEM.

I would guess that ruby's gems are mostly object-oriented, not designed
for FBP. Are there any ruby gems that are designed to work like FBP
or data-flow / CSP components?

Or, does ruby make it easy to use OO / procudural code directly in a FBP
system? Do you use ruby's coroutines to achieve independent components?

Any sort of reusable module is good, for sure. Are they well organized,
so we can find quality modules easily for a certain task?

Lots of questions from a ruby n00b!! thanks for sharing.

Sam

[Paul Morrison]

Relative to the question of how do you keep track of 38,000 (or
41,320) gems, my ex-colleague in Europe has this to say:

I believe that the fact that OO has a mechanism to deal with this is
one of the strongest elements in the success of OO: classification. (I
regard the isolation of storage management at the top level as the
most important part of OO.)

"FBP seems to have a rather good storage management approach of the
IPs, but it misses a look-up mechanism.
"As I highlighted in my pdf:
Application development = documentation processing

"One of the main ideas of documentation is a systematical method of
recording, to ensure an easy retrieval. Classification is very
successful approach, as it is in line with one of the ways our brain
does it. That is one of the reasons of the success of OO. But it is
not such that if somebody uses classification would mean that his/her
approach is object-oriented.

"Back in 1990 I was using the term subject-oriented, because in
business applications classification has almost nothing to do with
objects, but with the subjects of the applications. A subject can be a
real world object, but also an abstract object, a technical software
object, a person, etc. What is stored in the databases are often
conglomerations of subjects, because a record may contain information
that is only added because of its relationship with one or more
specific applications. That may be one of the main problems with the
use of OO at the highest levels of business applications. (E.g. in
order to add a record to the data base, an application must have some
knowledge of fields that have to be added to support other
applications, causing complex interdependencies of object that should
be independent.)

"In FBP a lot of the components deal with rather abstract technical
objects. E.g. Sort is totally independent of the logical contents of
the IP. In OOP the same happens to some degree, but it is more or less
hidden in its object-oriented classification system. E.g. a lot of
"is-a" relationships are only an expression of the way a set of
objects is technically kept together. All collection classes have been
introduced to be able to classify technical processes.
Because FBP does not support a mechanism to express the relationship
between the activity of a component with a technical type of IP (like
methods assigned to an object), classification becomes more difficult.
However, I think that it would be possible to set up some technical
layering.

"Just like OO uses the "is-a" relationship to express a specific type
of technical implementation, A (logically defined) component can be
implemented as an "is-implemented-as" component as a network of a
single component. This is part of what I mentioned in that first pdf
with "multiple views", where the non-technical people will follow the
flow of the logical elements, whereas the programmer will mainly deal
with the technical flow, but such that the relationship between both
views will remain part of the development process (documentation).

"Such a mechanism should support a dual classification scheme: subject
and activity. In that way I believe it could even improve on the OO
classification scheme."

He also feels that his concept of "application development = document
processing" is extremely important, but may be more meaningful to
business types than to academics. As he says, application development
should be efficient from a business point of view.

Any comments on this, Brad, based on your experience with Objective C?

He and I also both agree that powerful search engines are important in
locating useful components, but he says, "In order to build a good set
of reusable components, there must be logically structured set-up of
components. I am not aware of any better approach than the
(hierarchical) classification. Of course there should be enough
flexibility in it, like the use of synonyms or even less strict
equivalence, where the search mechanism may be a great help."

[Brad Cox]

Any comments on this, Brad, based on your experience with Objective C?

This is mainly post-ObjectiveC (Java) experience...

The Ruby Gems library is miniscule compared to the Java library, which is so big it can't even be counted. Nobody "keeps track" of anything that big. We just aim search engines at it and take what comes up.

The Google repository isn't at all object oriented. Just a string search over vast piles of named blobs. OO is entirely inside the blobs google serves up. It does help organize fine-grained modules but not much beyond that.  

[Doug]

Hi Sam,

One key concept of the platform is that it uses FBP concepts at an infrastructure level. So the nodes are just programs run as processes. They could be created in .Net, Java, or even Errlang. The platform just needs a mechanism for running those type of programs. Nodes are completely independent they have no knowledge of each other, they can be distributed anywhere.

Here is an example (in Ruby) of an echo node that writes messages to it's children (down stream nodes):

class EchoNode < Xenograte::NodeBase

  

  attr_reader :node_id, :node_config

  

  def initialize(options)

    super

  end

  

  def run

    super

  end

  

  def startup

    # this is where you can run code on startup of the node

  end

  

  def process_message(message)

    write_to_children(message)

  end

  

  def process

    # this gets called every 'n' milliseconds

  end

  

end

# this gets called from the infrastructure

EchoNode.new(ARGV)

The '#' indicates a comment.

Gems are typically managed in an application with Bundler http://gembundler.com/

We would use Gems to provide the functionality we want our nodes to have to do their work. Like JARs in Java, or apt-get in *nix.

So the infrastructure works a lot like *nix command line pipes '|' in that output from one program is piped into another. The magic of the platform is that it will do that across processes, servers, and clouds. Without having to manage end-points.

When I first looked at FBP then tenets and principles fit perfectly with what we were trying to do. But the 'P' in FBP doesn't really apply to an infrastructure approach, as we are orchestrating nodes (applets?) into workflows that become the application. And since you can aggregate running workflows into other workflows the word application might not quite fit either.

As for CSP/Data-flow, if you are referring to Communicating Sequential Processes then sort of.

The Ports, IP, and Components in FBP directly translate to our infrastructure where Components are nodes and IPs are messages and Ports, well you don't have to worry about those. A single node can receive messages from and send messages to any number of other nodes, so while the message processing by a single node is sequential, the system takes full advantage of multiple cores and parallel processing as nodes are evented with non-blocking I/O to handle concurrency.

We are thinking about applications in a different light. For us it is about data not being owned by the application, and applications being able to share running processes (nodes) to do work, and data flowing to where ever it is needed.

Sorry for the long reply, I hope it doesn't cause any tl;dr

Cheers,

Doug

[Doug]

Hi Paul,

How we will organize nodes is through a node library. Each account has their own library that they can use to organise nodes in a hierarchical manner. They can write nodes and upload them to the library as they wish. We will be looking at a Node Store (akin to Apple's App store) where open source and commercial nodes will be offered, and people can add them to their library through the store. Anyone could write nodes. Each of these (Store and library) would have search capabilities and the ability to present documentation on nodes so users can decide which nodes to add their library.

Users can also add documentation on the workflows, and their will be a mechanism to copy workflows and upload them to the store if you have a generic workflow you think people would need.

Ruby gems are just code libraries. So when I said wrap a gem into a node I was referring to using the gem's code to facilitate the work the node should perform. The same way you would use a code library in any language.

I can understand that if FBP is implemented at the language level then then OOP techniques would not fit. We are implementing FBP concepts at the infrastructure level, so OOP or any other programming technique is relevant in creating the code that will run as a node. The granularity is different. Objects and components can be used to code a node. Nodes are orchestrated to create workflows (we call them solutions) and workflows can be orchestrated to create business systems. Orchestration is telling the platform where to direct a node's output. 

 Cheers,

Doug

[Doug]

Hi Vladimir,

Ruby gems are just code libraries. Since nodes are just Ruby programs they can be coded like any other ruby program.

We have nodes that provide web server functionality, web client functionality, watch file systems, parse attachements from emails, apply XSLT to XML, update/insert data to a database, extract data from a database, and parse EDI documents into XML. An many other nodes that do many different things. 

It is very easy to use functionality from a gem, typically your just use a require statement in your Node such as: 

require 'rest-client'

Which would give you web client functionality where you could make calls like:

RestClient.get 'http://example.com/resource

As for what is different in our current workflow approach from FBP, lets look at the definition of each:

Wikipedia's definition for FBP:

[flow-based programming (FBP) is a programming paradigm that defines applications as networks of "black box" processes, which exchange data across predefined connections by message passing, where the connections are specified externally to the processes.]

My definition of Xenograte:

[Xenograte is a Flow-based Platform as a Service that defines applications as a network (solutions) of "black-box" processes (nodes) which exchange data across processes, servers, networks and clouds by message passing where the connections are automatic, transparent and external to the processes.]

The biggest difference is that Xenograte is a platform and FBP is a programming paradigm (according to wikipedia). I'd argue that Xenograte is also a programming paradigm but not in the sense of coding nodes but in the sense of orchestrating those nodes as running processes. Because in essence when you create an orchestration of nodes you are programming the solution (application), its just that the granularity is different.

To me a node is an application, albeit a very small one with a single responsibility, and perhaps not even very useful by itself. But when you can connect a bunch of nodes together you get the aggregate functionality which can solve business problems which I call a "solution" as application doesn't seem to fit as I am weaving several tiny applications into a bigger one. 

I actually did not know about FBP when I first had the idea. I simply needed a better way of reusing higher level functionality in an agile and extensible manner to solve business problems. It turned out that the tenets and principles I was using were the same as FBP. 

I think FBP will be prominent in the future of business systems. FBP principles will let us create ecosystems of solutions where applications do not own the data, logic in a running application will be useable in another application and data flows anywhere it is needed.

Cheers,

Doug

On Tuesday, July 10, 2012 12:15:42 PM UTC-7, Vladimir Sibirov wrote:

Hi Doug,

That's impressive!

Could you give an example how Ruby gems can be used as nodes of a cloud application? Could you also tell us how you got to the idea of using FBP to build the system and what is different in your current workflow approach from FBP?

[Paul Morrison]

On 11/07/2012 9:35 PM, Doug wrote:
> I'd argue that Xenograte is also a programming paradigm but not in the
> sense of coding nodes but in the sense of orchestrating those nodes as
> running processes.

Hi Doug,

Yes, that seems to define my idea of FBP pretty well - maybe even better
than my old formulation :-) For me also, FBP is much more about reusing
black-box components than about coding new ones. Hence the reason for my
questions about a repository, which people who are oriented around
writing new stuff tend not to worry about much, in my view. :-) In my
book I followed Gelernter in describing Linda as a "coordination
language", rather than as a "programming language". Since this
coordination language is pretty much just a list of connections, you
probably don't even need the word "language"!

Regards,

Paul

[Paul Morrison]

On 11/07/2012 7:01 PM, Doug wrote:
> and Ports, well you don't have to worry about those

Just curious: why not? To me, ports, whether named or numbered, are one
of the key building blocks of FBP. However, I am aware there are other
approaches to this function...

[Ralf Westphal]

For the concept of FBP I´d say it does not matter what kind of black boxes are connected using data flows, as long as it´s, well, black boxes and data flows :-)

So if some implementation uses Java classes as black boxes, that´s fine.

If another implementation uses Ruby gems, that´s fine, too.

On a piece of paper a certain flow graph would not look different for both implementations.

I´m currently playing around with IronWorker and IronMQ (by www.iron.io). It´s a very lightweight cloud platform for processes plus a queue infrastructure in the cloud.

And I´d say, if I upload 3 workers in 3 different languages (Ruby, Python, C#) and let them communicate via IronMQ (and Json data), then that´s FBP, too. All those workers can run in parallel or asynchronously.

Communication might be comparatively slow via a cloud queue - but if the solution does not call for much performance but more for scalability... why not?

[Doug]

Hi Paul,

You are correct ports (connections) are fundamental. We just hide the responsibility of port definitions from the user so they only need to draw lines between nodes to facilitate communications no matter where those nodes are deployed. Orchestration and deployment are two different actions and do not constrain each other. This way any node can talk to any other node no matter where it is deployed. This is especially true in a virtualized environment where we handle server provisioning and node deployment and communications seamlessly. For an in-house server behind a corporate firewall a channel (VPN etc...) needs to be established wich is not an automatic process for security reasons. Once a connection is established and security information is exchanges then the system will be able to deploy nodes to that system as well.

[Graham Chiu]

On Fri, Jul 13, 2012 at 5:55 AM, Ralf Westphal <ral...@googlemail.com> wrote:

For the concept of FBP I´d say it does not matter what kind of black boxes are connected using data flows, as long as it´s, well, black boxes and data flows :-)

So if some implementation uses Java classes as black boxes, that´s fine.

If another implementation uses Ruby gems, that´s fine, too.

On a piece of paper a certain flow graph would not look different for both implementations.

I´m currently playing around with IronWorker and IronMQ (by www.iron.io). It´s a very lightweight cloud platform for processes plus a queue infrastructure in the cloud.

NASA is using Amazon's SWF for a similar purpose.  See http://aws.amazon.com/swf/testimonials/swfnasa/ for a case study


--
Graham Chiu

[Paul Morrison]

On 12/07/2012 2:00 PM, Doug wrote:

so they only need to draw lines between nodes to facilitate communications

Still having a bit of trouble with this... sorry :-)  Ports are (also) the way that the inside of a component communicates with the outside, so, suppose you have a component that tests incoming IPs (messages), and sends them either to ACC or to REJ,   I can certainly see displaying the component so that the ACC and REJ (and IN) ports are "sticky".  But what happens if you haven't written the component yet?  I guess you could have something like a component signature which can be used to tell what ports to display, without requiring running code to be available, but then I would guess you would have to have a standard signature format, independent of the language the component is (going to be) written in.   In DrawFBP, we took the opposite tack: you hook up the components, and then fill in the port names - which of course have to match the port names used by the components.  In JavaFBP and C#FBP, annotations (attributes) were added later, allowing some checking to be done, once the components have been (at least partially) written.  What am I missing?!

Regards,

[Henri Bergius]

Hi,

On Tue, Jul 3, 2012 at 6:46 AM, Ralf Westphal <ral...@googlemail.com> wrote:
> Who is actually implementing production software using FBP?

We're using NoFlo in production at Nemein. Here are some example cases:

* Driving information from a back-end system to a large number of
information displays in a conference center
* Business workflows of an unnamed start-up done by routing message
queue packets with NoFlo

...I also use it for various smaller Extract-Transform-Load tasks, but
these are often one-shot implementations that I wouldn't really call
production. For example when I converted my blog to Jekyll:
https://gist.github.com/3105838

> Or who has presented FBP to an audience of more than 3-5 people (i.e.
> his/her own team)?

I had a talk in JS.Everywhere last autumn:
http://www.youtube.com/watch?v=pgP4v9b5DvE

In addition to the conference audience a few hundred people have
watched the video.

/Henri

--
Henri Bergius
Motorcycle Adventures and Free Software
http://bergie.iki.fi/

Jabber: henri....@gmail.com
Microblogs: @bergie

[Henri Bergius]

On Fri, Jul 13, 2012 at 8:10 PM, Andrew Klofas <akl...@gmail.com> wrote:
> There's a pretty big community around FBP for robotics. It's a bit more
> synchronized than what the book goes over and a few other tweaks (IP can be
> silently dropped, etc).

This is actually how it is in NoFlo as well, any component is free to
drop IPs if they so feel like. It would be difficult to prevent this
in a dynamic language like JavaScript, though probably the network
could at least keep track of how many packets went in, and how many
went out from a process.

> Andrew

[Paul Morrison]

Yes, I was struck by the uses of FBP I ran into for robotics - I mention several in the 2nd edition of my book, e.g. Urbiscript, Robotflow.

And thanks for the kind words, Andrew, Henri and others!  I dunno, is almost 75 old?!  I seem to go back quite a long way in the computer business, but I still feel pretty young :-)

On Friday, July 13, 2012 2:10:36 PM UTC-4, Andrew Klofas wrote:

There's a pretty big community around FBP for robotics. It's a bit more synchronized than what the book goes over and a few other tweaks (IP can be silently dropped, etc). Willow Garage built ROS (http://www.willowgarage.com/pages/software/ros-platform) and there's also openrdk (http://openrdk.sourceforge.net/) and others. I myself have been playing around with FBP for robotics (https://github.com/maxkernel/kernel)

Andrew

On Tuesday, July 3, 2012 6:22:01 AM UTC-7, Paul Morrison wrote:

Great question, Ralf!  I'm really interested in the answers - naturally!

I think there are also at least some people who are not members of the
group - Mike Beckerle (quoted several times in my book) is one who
springs to mind...

Regards,

Paul

On 03/07/2012 12:46 AM, Ralf Westphal wrote:
> I´m interested in knowing who of the members of the group is using FBP
> in day-to-day business?

>
> Who is actually implementing production software using FBP?

> Or who is actually using it at least in larger examples?
> Or who is using it in teaching others how to approach software
> development in a different manner?
> Or who has written about FBP within the past 6 months more than a
> single page/blog post?

> Or who has presented FBP to an audience of more than 3-5 people (i.e.
> his/her own team)?
>

> Or to be more blunt: Who has made any money by using/advocating FBP in
> the past 6 months?
>
> I´m asking this because the more I listen to the conversations going
> on here I get the impression FBP is, well, dead. Dead in practical
> terms, meaning nobody is actually using it in a way he´s depending on.
> FBP seems to be a toy, a nice concept to discuss here - but without
> any impact on day to day business.
>
> But of course I´d love to be proven wrong by many answers to my
> questions telling of how FBP makes to them a difference every single day.
> Looking forward to them...


--
http://jpaulmorrison.blogspot.ca/

[Paul Morrison]

Hi Doug,  Not sure if you saw my question...  When you talk about just drawing lines between nodes, how do you handle the case where a node has more than one input (or output) port?  In JavaFBP and C#FBP, we have ports are named, but they can be either array ports or non-array ports - Concat might be an example of the former...

[ifknot]

Ralf,

Who is actually implementing production software using FBP? No research 

Or who is actually using it at least in larger examples? Yes

Or who is using it in teaching others how to approach software development in a different manner? Yes

Or who has written about FBP within the past 6 months more than a single page/blog post? Yes

Or who has presented FBP to an audience of more than 3-5 people (i.e. his/her own team)? Yes 

Or to be more blunt: Who has made any money by using/advocating FBP in the past 6 months? Research grant 

ifknot

[Samuel Lampa]

This is awesome! 

I got struck by the idea as well the other day, to do a shell based FBP system, but now that there is stuff available already, I might not need to reinvent the wheel.

It would be very nice to use github as a platform though, as pull requests and forking is so much easier (not to mention code visibility). Any plans for such a move? (or a mirrored contrib repo on github at least? :) ). If not, how do I check out this code myself?

I also had some ideas for how to create multi-node jobs by combining named pipes with netcat or ssh (but the latter would be slow I guess), but maybe you have done work on that already?

Anyway, interesting stuff, and would be very happy to help out with hacking on an awesome shell based FBP framework! 

(makes a lot of sense in a HPC cluster environment like where I work)

Cheers

// Samuel

Den fredagen den 6:e juli 2012 kl. 10:44:08 UTC+2 skrev Sam Watkins:

> ... I hope tomorrow I can report back, that I did earn a day's wages
> with the help of FBP, and my existing arsenal of shell tools!

g'day again,

I haven't done much that's directly useful for my job yet, but:

I wrote 'net2sh' a small self-hosting FBP shell script 'compiler',

    net2sh graph:   http://sam.nipl.net/code/net2sh/g/net2sh.dot.png
    net2sh source:  http://sam.nipl.net/code/net2sh/net2sh.net
    net2sh shell:   http://sam.nipl.net/code/net2sh/net2sh

and a tool in it, 'net2dot' makes graphviz dot files for plotting.

    net2dot graph:   http://sam.nipl.net/code/net2sh/g/net2dot.dot.png
    net2dot source:  http://sam.nipl.net/code/net2sh/net2dot.net
    net2dot shell:   http://sam.nipl.net/code/net2sh/net2dot

There's one other simple test program, to count files in a directory.

    test1 graph:   http://sam.nipl.net/code/net2sh/g/eg/test1.dot.png
    test1 source:  http://sam.nipl.net/code/net2sh/eg/test1.net
    test1 shell:   http://sam.nipl.net/code/net2sh/eg/test1

It uses standard tools, some reusable tools, and some custom parts.
One of the net2sh components is another little FBP program (a sub-net).
Components are normal shell tools, and can be written in any language.

The idea is to run a bunch of programs (shell commands) in parallel,
connected by a bunch of named pipes, and wait for them to finish.
None of those namby-pamby coroutines this time around!!

To avoid deadlock, I wrote 'buf', a file-backed buffer component.
There is a simple tokenizer (and token re-joiner) in there, etc.

I haven't tried running anything very large in it.


Sam

[Matthew Lai]

Or even plain old socket programming will do.

I have my d-fbp components distributed onto a pc (Windows 7) and 2 Raspberry Pi boards (Linux).
All the outport to inport connections are just TCP/IP sockets; they work well either within a computer or
across different computers with different OSs (assuming the OS supports socket programming of course).

Matt

[Samuel Lampa]

Interesting! What is d-fbp? Is there any info about it on the net?
(sorry, haven't been following this group to regulary, only just
recently turned on email subscription)

Best Regards
// Samuel

[Sam Watkins]

On Fri, Oct 04, 2013 at 06:24:21AM -0700, Samuel Lampa wrote:
> Anyway, interesting stuff, and would be very happy to help out with hacking
> on an awesome shell based FBP framework!
> (makes a lot of sense in a HPC cluster environment like where I work)

thanks Samuel.

cool, I'm interested to continue it too. I got a bit stalled with making a
fancy demo program, thanks for encouraging me to continue with it. And I'd
appreciate help too. Crafting nice reusable components is good fun too.

[Matthew Lai]

Distributed FBP (d-fbp) is my Tcl implementation of FBP. The idea is to execute the
FBP graph in multiple computers. I keep the implementation simple: each component
maps to a stand-alone Window/*nix process, and each component/outport -> component/inport
connection is just a socket connection, so it works whether the two components are
on the same computer or on different computers. The bounded buffer criteria (finite number
of IP allowed on each connection) and the blocking send behaviour are supported, again
either within the same computer or between computers. I do deviate from the true (whatever
true means...) FBP
- no use of IIP
- use max number of IP per inport rather than max number of IP per connection
- no support for selective inport read by the component (yeah, the GHOST of collate!!!).
  Not an issue for my financial data crawler work, so I don't need to fix it anytime soon :)

To run the FBP graph on multiple computers, a daemon process, fbp_agent, exists in
each computer to interpret the FBP graph and spawn the necessary processes.
There is a corresponding fbp_mgr that works with all these fbp_agents for socket
connections setup. See below.

A simple walkthrough of running FBP graph in distributive mode:
- Admin user starts the fbp_agent daemon on each computer. In my case that would be the
  Toshiba ultrabook (Windows or Linux), Raspberry Pi board 1 and board 2 (both Linux).
- User constructs the FBP graph.
- User performs a network discovery i.e. each fbp_agent responds with its IP address
- User assigns each component to an IP address.
- User "runs" the FBP graph.
  fbp_mgr sends the FBP graph to each fbp_agent on each computer.
  fbp_agent spawns the process for those components that have a matching IP address.
  fbp_mgr communicates with each fbp_agent to run the socket connection setup
  - enable the sockets on the inports (in Tcl jargon these are called the server sockets) of each
    component, then
  - connect the component outport sockets to other component inport sockets according to the graph layout
  - Once the connection setup is finished the FBP graph can start running.
  - Runtime data like per process cpu utilization on each computer, and the number of IPs waiting in
    each inport, are collected by the fbp_agent, sent back to fbp_mgr, and displayed in the GUI.

Notice there is no message broker here: all connections are point to point socket
communication. Also no message queue (eg. zeromq) is used here because
(a) I have my own load balancer component and
(b) I want to achieve as much pipelining as possible i.e. component has an output IP, do not buffer, sends
it to the outport for the next component to process.

If you are interested in knowing more about Tcl (event processing loop, async IO, socket API...etc)
I can provide some reading material and sample code to try.

Yours,

Matt

[John Cowan]

Matthew Lai scripsit:

> - no use of IIP

IIPs are trivial; command-line parameters serve the same purpose, and
fit better into the OS-process way of doing things. So I wouldn't worry
about that.

> - use max number of IP per inport rather than max number of IP per
> connection

Again, a trivial distinction unless the number of connections to an
inport varies dynamically.

> - no support for selective inport read by the component (yeah, the
> GHOST of collate!!!).

That's problematic for a general-purpose framework, but if you don't
need it, you don't.

--
I now introduce Professor Smullyan, John Cowan
who will prove to you that either co...@ccil.org
he doesn't exist or you don't exist, http://www.ccil.org/~cowan
but you won't know which. --Melvin Fitting

[Sam Watkins]

I did some more work on my wacky 'net2sh' for FBP in the shell, but hit
a snag with the named pipes.

I found that if one upstream process opens several named pipes to write,
and a downstream process opens those pipes to read, then they must be
opened in the same order! otherwise both processes block, waiting for
the other. Same problem can happen fan-out / fan-in scenario.

This is not good for a general FBP system, we need to be able to open
ports in any order. I tried 'non-blocking open', it doesn't help.

I hacked up something to avoid this deadlock for my test program: it
uses 'cat' to feed data in from the pipe, but that is ugly.

So yeah... I'm not sure how to overcome this without changing the
approach to avoid named pipes. And I don't know how to do FBP in the
shell without named pipes.

An 'FBP shell' in C could use regular anonymous pipes, and be more
flexible. But I liked that this uses the regular shell.

[Matthew Lai]

Sam,

Interesting observation.

I have not run into this problem with my sockets implementation. May be it is
the startup sequence I use in my FBP network:

(a) All inports on all processes (nodes in the FBP graph), on all computers,
      are "enabled" first. In Tcl jargon this "enabled" translates to setting up the
      server sockets to listening mode. The server sockets are set up in no particular
      order.

      This phase is coordinated by the interaction between a fbp manager (resides
      within my FBP gui process) and fbp agents (one per computer).

(b) All outports on all processes, on all computers, are "enabled". In Tcl jargon this
      "enabled" translates to connecting to the server sockets enabled in step (a).
      No order is specified for this connection setup.

      Again this phase is coordinated by the interaction between a fbp manager (resides
      within my FBP gui process) and fbp agents (one per computer).

This scheme works fine in my 3 machines test setup (ultrabook + 2 RasPi boards).

Matt

[Tom Young]

If a process is opening multiple input ports (as I guess your "downstream" process is, then

deadlocks are likely, even if you are able to get everything "open"ed.   Even an "'FBP shell' in C"

can have this problem unless things are synchronized properly.

'cat' may seem ugly, but if it does the job ....

Send code if you like.

          Cheers,

                     -Tom

      

 

"...the shell syntax required to initiate a coprocess and connect its input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Sam Watkins]

On Tue, Oct 22, 2013 at 01:36:53PM -0400, Tom Young wrote:
> If a process is opening multiple input ports (as I guess your
> "downstream" process is, then deadlocks are likely, even if you are
> able to get everything "open"ed. Even an "'FBP shell' in C" can have
> this problem unless things are synchronized properly.

I can see a couple reasons for that:

- buffers not big enough
- buffers hold onto data too long
- bug ... and it can be tricky to debug FBP programs at first!

I might avoid those problems using a good buffer component, which can
expand as needed, make data available to read immediately, and
optionally log to a file. At the moment, named pipes seem to work well
most of the time, and I can plug in a "super buffer" as needed.

Is there some other problem? Using multiple inputs seems absolutely
necessary to me, e.g. unix tools such as join(1) and comm(1) do that.

> 'cat' may seem ugly, but if it does the job ....

It does do the job, but this shell model is already heavy.
I'm not keen to have an extra cat for each stream, getting too ugly.

Still, for me it's more about learning how to make FBP programs with
help from the unix toolkit (and my own tools). It's doesn't have to be
very efficient at this point.

[Sam Watkins]

On Tue, Oct 22, 2013 at 05:49:52AM -0700, Matthew Lai wrote:
> Sam,
>
> Interesting observation.
>
> I have not run into this problem with my sockets implementation

As I understand, my 'open deadlock' problem only happens with named
pipes, not regular pipes or sockets. :(

You're using TCP sockets not unix domain sockets, right?

[Matthew Lai]

Correct. I use the socket api inside Tcl, which is really a TCP socket.
The socket connection works fine on both Windows/Linux OS, whether
it is single computer or multiple computers (Windows <==> Windows,
Linux <==> Linux, Windows <==> Linux).

Matt

[Tom Young]

Deadlock occurs when process A is blocked waiting on input x, while process B is blocked

waiting to write output y (before writing output x).   (This can happen with pipes, or sockets, or almost any I/O.)

The solution is for A to block waiting on input x or input y, which may not be possible with a shell script.   It is
easily done in C using  the 'select' function.   

Buffering can exacerbate the problem by delaying the data availability until the buffer is full.  There are *nix

facilities to prevent buffering by forcing interactive (immediate) mode. 

Attached is a graph of a working network, which includes a number of bash script processes.  Processes with

multiple inputs are highlighted in orange.  They are all written in C and use the select function (except  the Concat component which does not need to use select).   None of the script processes have multiple inputs, but most have

multiple outputs.   The Join and Concat components can handle any number of input ports.  

The Concat function could have been included in a script, and that might run slightly faster, but doing so would

hide functionality from the network graph and require more development, since the C language  'Concat' component was already available.  

 

I agree that multiple inputs are often necessary.   Elsewhere I have argued that we need to support acyclic networks

and understand and deal appropriately with the potential deadlock problems that can result.

I think you are right to avoid emphasis on efficiency.

---

twy

[Tom Young]

Meant to say "... support cyclic networks..."

Thomas W. Young, Founding Member
Stamford Data, LLC
47 Mitchell Street, Stamford,  CT  06902
Phone: (203)539-1278
Email: TomY...@stamforddata.com

[Sam Watkins]

On Wed, Oct 23, 2013 at 05:19:51AM -0700, Matthew Lai wrote:
> Correct. I use the socket api inside Tcl, which is really a TCP socket.
> The socket connection works fine on both Windows/Linux OS, whether
> it is single computer or multiple computers (Windows <==> Windows,
> Linux <==> Linux, Windows <==> Linux).

That is appealing, to be able to run over several computers
transparently. The only thing that concerns me with TCP sockets is lack
of access control. Although my named pipes don't have much in the way
of access control either.

Can you explain more how you use the TCP sockets, is there one listener
which is shared for different components, or one listener per component,
or one listener per input port...? I'd be interested to see how it
works. Do you have a write up and/or demo on the web somewhere?

Sam

[Sam Watkins]

On Wed, Oct 23, 2013 at 09:36:53AM -0400, Tom Young wrote:
> Deadlock occurs when process A is blocked waiting on input x, while process
> B is blocked
> waiting to write output y (before writing output x).

If you can inserting large buffers as needed, a process can't block
indefinitely waiting to write.

> The solution is for A to block waiting on input x or input y, which may not
> be possible with a shell script.

The components in my dodgy shell script FBP are typically not written
in shell script. The network runs as a shell script.

Components can be written in C, perl, python, Java or whatever. So I
can use select, threads, or other mechanisms for non-blocking IO.
e.g. my "unlimited buffer" component although it has only one input and
one output does use select.

> Buffering can exacerbate the problem by delaying the data availability
> until the buffer is full.

I don't like that kind of buffering. Maybe I was using the wrong word.
A unix pipe for example can hold up to 4KB of data, but as soon as any
data has been written to it, it is available for the other end to read.

e.g. while true; do echo foo; sleep 1; done |
while true; do read B; echo $B; done

My large buffer component works in the same way. It can store as much
data as needed, but makes that data available to read immediately.

An alternative, to reduce thrashing, is for the buffer component to
introduce a very short delay, so there is a chance for data to
accumulate before notifying the downstream. I also wrote one like that,
but I'm not sure where it is at the moment.

> Attached is a graph of a working network, which includes a number of
> bash script processes.

That program graph looks interesting, thanks for sharing it.
Do you have any full runnable demos of your FBP system that I could try?


Sam

[Matthew Lai]

Sam,

- One listener per input port (or server socket in Tcl jargon).
- Each node is a stand alone Windows/*nix process, listening to the input port(s).
- Each node runs in an async IO and reactive event processing mode i.e. IP arrives
  in a server socket, which triggers the Tcl event processing engine to run, which in
  turn invokes the listener code.
- Bounded FIFO buffer i.e. the maximum number of IP allowed in a given connection,
  is supported by a buffer node. I believe I described it as the
  kludgy process in some other discussion threads.
- Selective input port read (eg. with 3 input ports configured, application code wants
   to listen to input port 1 first) is not supported. A potential solution was described
   in the Toronto meetup yesterday.

No write up yet. No website yet. Too lazy doing documentation, I know...

Only real life face to face demo so far. If you folks decide to visit Toronto just give
me a yell and I can do a FBP show & tell in some coffee shop :)

Matt