Is anyone using FBP for production work? Plea for feedback!

[Paul Morrison]

Every morning a bunch of posts come into my inbox in connection with the FBP Group, but I notice that the majority of them seem to be basically from people developing graphical or data streaming tools, rather than struggling to use existing tools.  NoFlo/flowhub is borderline, as a lot of people seem to be working at extending it/them.

I had expected that more of these posts would be from people trying to use FBP for real-world problems, as one would use Java or COBOL...  Now I know a lot of you are musicians and artists and you probably are using various FBP-like dialects to write music or do art, but where are the payroll and inventory developers?!  Especially as, when I did a survey a couple of years ago, 3/4 of the respondents self-identified as business, rather than academic.   I have appended the pie-chart from this study.

Now, I spent 35 years in business, and I am sure very few business are going to bet the store on some untried technology, so I have to asume that the business types among you who are developing neat software are in fact in some R & D department hidden away in a lab somewhere...  Personally speaking, in my case FBP went further when we were refining it in the context of real-world problems - and I had more fun!  The only products I worked on in bona fide labs (before FBP) never actually went anywhere - and a very mature FBP product that was transferred to a lab got strangled in red tape, literally!

Given that there are 512 members currently listed on the FBP Google group, of which we typically hear from probably less than a dozen, I would love for some of the others to let us know:

- if they are using some dialect of FBP for production work - if so, which dialect

- if they tried it and couldn't use it, or it broke and they gave up - again, which dialect, and what happened

- if they find the concepts interesting, but can't see the relevance to what they are doing

- they would like to use it, but management has decreed that they have to use SAP, say

Please note, we know that some of you are sworn to secrecy, so we don't want you to reveal any trade secrets, or give away intellectual property...

But just a tiny hint...?  Is that too much to ask?!

TIA

Paul M. (aka Morpheus - sorry, in joke)

[Paolo Di Tommaso]

Hello, 

I take the opportunity of this post to introduce myself and the FBP related project on which I'm working. 

I'm a software engineer working in genomic research institute in Spain and I'm interested in computational pipelines for bioinformatics data-analysis. 

Actually I landed in this forum few days ago, after reading the paper "Advances in dataflow programming languages" and in particular the part regarding your work on FBP. 

For the computing requirements of my lab in the last year I developed a pipelines framework and a language based on some dataflow principles that is called Nextflow.  You can find the project home page here: 

  http://www.nextflow.io

In very few words Nextflow allows one to write a computational pipelines by putting together many different tasks (called processes) that can be defined using simple BASH scripts or piece of software that can be executed in a Linux/POSIX platform. Processes are (obviously) executed in a parallel manner, do not share memory and they communicate through asynchronous dataflow queues (called channels). 

The framework itself is built on top of the Groovy runtime and the Gpars library that provides an excellent dataflow implementation for the Java VM. To this I've added a DSL extension to handle the process definition as language primitive, plus some other syntax sugars to manage with ease some common bioinformatics data formats and recurrent patterns.

We have used it extensively in our lab in the last year to run pipelines that launch hundreds of thousand of jobs and run for days. 

You can have a look to some real world Nextflow scripts at the following links: 

  https://github.com/cbcrg/piper-nf/blob/master/piper.nf  

  https://github.com/cbcrg/grape-nf/blob/master/grape.nf   

  

We are pretty happy with it. One of the main benefit of Nextflow is that it allows a bioinformatician to prototype a data analysis pipeline quickly and in a very easy manner. Simply by putting together a bunch of existing Perl/Python/Bash scripts or tools. 

This a very important property because bioinformaticians tend/need to reuse as much as possible their existing piece of software. 

This is also the reason why I found extremely interesting your work on Large-grain dataflow and the concept of coordination language in the FBP model. After reading it I'm very  keen to think the Nextflow DSL as a kind of coordination language, though I'm not sure it can be defined a "pure" FBP processing model, .. I still need to read your book :)

Hope this helps and I'm really happy to share my work in this forum. 

Best,

Paolo 

[Paul Morrison]

Thanks, Paolo, that looks fascinating!  It's great that you have been using this technology productively for at least a year - it sounds like.  You also make a good point about people wanting to reuse software as much as possible.  And it looks like we should learn more about Groovy and Gpars... and NextFlow  :-)

Let's hope your example will prompt other people to share their experiences with us!

Regards,

Paul M.

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Morrison]

This is disappointing!  4 days later, and exactly one person of the 514 (as of today) has posted a response to my plea!  I have also heard informally from one other person, and I know for sure that some of you are working on FBP-like implementations for various devices and systems. 

So can I assume that all of the rest of you are sworn to secrecy?!

John, do you think that a lot of people have signed up, but then switched off their subscribe functions?  So maybe responses will trickle in as they login to the Group...?  Let's hope!  :-)

I do hope that some of you are finding FBP useful, and best regards to all of you!

Paul M.

[Alfredo Sistema]

I'm working on an implementation of FBP and helping with the documentation efforts. My github handle is alfa256.

[Matthew Lai]

Mr.Morrison,

I've been using my custom tcl based FBP implementation to run financial data crawling 

applications in the last two years. The crawler is executed every trading data after the

market is closed to retrieve data for most of the TMX symbols: company data, stock data,

options and warrants data. The financial data is used to drive my personal investment

decisions on (a) fixed yield investments and (b) stock investments focused on TMX.

Matt

[John Nilsson]

I work as an architect in a team who builds typical LoB applications for various clients in mostly small to medium sized projects, nothing fancy or particularly demanding of the infrastructure.

While I find the concept intriguing as a means to clarify designs I'm currently just monitoring this list to learn the concepts and asses the usefulness and applicability of any related technology for my projects.

So far, it seems, mostly people build their own implementations or are working on products not yet ready to bet my clients budgets on. So I'm content with just letting the discussions teach me a thing or two about how things fit together theoretically.

BR,

John

—
Sent from Mailbox

--

[Oleksandr Lobunets]

Hi Paul,

I'm working on yet another implementation of the FBP runtime focusing in the first turn the Internet of Things domain but not limited only to it (after all it's a matter of components we use). The first implementation was rather a general data flow system and now while taking part in the FBP discussions we're refactoring more to conform to FBP baseline. We are planning to open the sources soon.

Kind regards,

Alex

[Paul Morrison]

Thanks, Alex, John, Matt and Alfredo - that's good feedback!  John, I agree with you about not betting the budget on untried software, but JavaFBP has been in use for a few years, and I think it would be safe to give it a try on a non-mission-critical app...  For instance, I used an early implementation of FBP (non-Java) for doing regression testing on transaction suites.  The configurability came in very handy!   Just a suggestion!

Best regards,

Paul M.

[Ged Byrne]

Hi Paul,

Please keep in mind that it takes a long time to get something into production.

I'm not using a FBP product specifically, but it does strongly inform the way in which I am implementing Enterprise Integration Patterns.

Even this has problems.  For example, some people seem fixated on the use of an ESB.

Regards, 

Ged

[Sam Watkins]

I really want to use FBP in production, and was trying to do so,
but my shell-based FBP system has some flaws which will require
a big effort to fix. I should make the effort, so I can write proper
flow-based programs for my work.

Sam

[Paul Morrison]

Thanks for the feedback! 

Sam, you'd be a good person to ask: why are people so afraid of using Java or C#?!  This seems to be common motif in the FBP world!  And yet Java is still the most popular programming language... although I grant you JavaScript runs a close second!

Sam

[Matthew Lai]

Mr. Morrison,

I doubt "afraid" is the right word here :) A search in the monsters or any other job boards reveals java developers

are still in high demand these days. May be folks just "prefer" to use duck typing language like javascript? I like

Tcl better ha ha (yes it is duck typed too...).

I understand that it takes time for FBP to evolve, to get to a production quality.

But there is another path as suggested by Paul Graham quite a while back:

"Here it is: I like to find (a) simple solutions (b) to overlooked problems (c) that actually need to be solved, and (d) deliver them as informally as possible, (e) starting with a very crude version 1, then (f) iterating rapidly."

http://www.paulgraham.com/newthings.html

That's how I "evolved" my tcl fbp implementation. Right from day 1 it worked in the sense

that it retrieved the financial information I need to drive my investment decisions. And that

day 1 was more than 3 years ago. The very first version was bare-bone: just direct socket

connections among Linux processes within 1 computer, with spice-like data file for setting

up the node connections. Over the years I put in features like bounded FIFO support, GUI

support to stitch up nodes, distributed processing to map nodes to different computers,

GUI disconnect/reconnect, all the while running the financial crawler application every trading

day. I shall not claim my FBP is in production mode, but as long as it is evolving, and growing,

I'm a happy camper :)

Matt

[Paul Morrison]

Hi Ged,

On Tuesday, July 1, 2014 8:56:08 AM UTC-4, Ged Byrne wrote:

Hi Paul,

Please keep in mind that it takes a long time to get something into production.

Yes, but 40 years is ridiculous!  :-)

I'm not using a FBP product specifically, but it does strongly inform the way in which I am implementing Enterprise Integration Patterns.

Actually that can be a very good strategy..

Even this has problems.  For example, some people seem fixated on the use of an ESB.

Has anyone compared ESBs vs FBP?  The Wikipedia article says it does "message and event queuing and sequencing"...

Regards,

Paul

[Paul Morrison]

Hi Matt, good feedback!  I know we have kicked this issue around a number of times over the past few years, but there is always something new to be said!

On Tuesday, July 1, 2014 10:22:00 AM UTC-4, Matthew Lai wrote:

I doubt "afraid" is the right word here :) A search in the monsters or any other job boards reveals java developers

are still in high demand these days. May be folks just "prefer" to use duck typing language like javascript? I like

Tcl better ha ha (yes it is duck typed too...).

Re duck typing, I can't see it!  When I asked Henri what was the appeal, I think he said that it reduced the number of keystrokes when he is coding in stations between trains!  There has got to be more to it than that!

As I asked the other day, if I am processing people, companies and cities, what's wrong with calling them that?!  Maybe it's just a different domain...?  I know strict typing helped a lot when I was coding DrawFBP!
 

I understand that it takes time for FBP to evolve, to get to a production quality.

But there is another path as suggested by Paul Graham quite a while back:

"Here it is: I like to find (a) simple solutions (b) to overlooked problems (c) that actually need to be solved, and (d) deliver them as informally as possible, (e) starting with a very crude version 1, then (f) iterating rapidly."

http://www.paulgraham.com/newthings.html

That's how I "evolved" my tcl fbp implementation. Right from day 1 it worked in the sense

that it retrieved the financial information I need to drive my investment decisions. And that

day 1 was more than 3 years ago. The very first version was bare-bone: just direct socket

connections among Linux processes within 1 computer, with spice-like data file for setting

up the node connections. Over the years I put in features like bounded FIFO support, GUI

support to stitch up nodes, distributed processing to map nodes to different computers,

GUI disconnect/reconnect, all the while running the financial crawler application every trading

day. I shall not claim my FBP is in production mode, but as long as it is evolving, and growing,

I'm a happy camper :)

I would say FBP followed a similar trajectory.  We used prgogressively more and more powerful versions within the Montreal Data centre - which actually was a great environment for this kind of thing as we owned the code!  So when I took the first implementation (AMPS)  to the bank, by then I had a high level of confidence in it, and was able to sell management on using it.  And one of the production programs was definitely still running at the beginning of this year (after almost 40 years in production!).   JavaFBP itself has been undergoing fairly constant improvement for at least 6 years, so maybe it's time someone took a chance on it - for a non-mission-critical application - at first :-)  And preferably someone who is allowed to talk about it!

Regards, and thanks,

Paul

[Oleksandr Lobunets]

Re duck typing, I can't see it!  When I asked Henri what was the appeal, I think he said that it reduced the number of keystrokes when he is coding in stations between trains!  There has got to be more to it than that!

Actual the CoffeeScript is what saves keystrokes, not JavaScript :-) JS is a joke, a misstate (from language and infrastructure POV) that accidentally became popular  :)

Well, they try to fix a lot in ESMAScript version 6 would probably fix some weirdnesses, but still it looks like a bad karma..

 

[Oleksandr Lobunets]

Hate this auto-correction: Actual->Actually, misstate->mistake. Sorry for that.

[Rob Halff]

I think the main problem for production usage is the way a project is presented.

If as a developer you know your system is capable enough to be used within production,

the way you present and advocate it becomes important.

e.g. when I search for JavaFBP ( https://www.google.nl/search?q=javafbp&oq=javafbp )

I'm not really overwhelmed by information, this has nothing to do however with the production readiness of the implementation.

The main entry to JavaFBP seems to be the book and when I find the source I get redirected to the book and a personal website.

Visibility, support, documentation, examples, services etc. are important.

All that boring stuff developers are not fond of, because well the code works so what's the problem :-)

If there is already an entry-level problem for developers, for managers to adopt it will be even less feasible.

Academics will adopt fbp more easily I guess, because they live within an environment where they are mostly

free to choose or make their own tools, the developer decides.

Stories of long running implementations are great, however hidden away within a newsgroup with mainly only idlers won't cut it.

Explained thoroughly as use cases or (real) testimonials on a website, they would be good selling points.

At the moment I'm writing my own fbp-ish system and try to obscure it as much as possible,

hence I will not yet mention it's name and I know my project is not ready for production at all and maybe it will never be.

Saying this, I think I'm actually explaining the issues I have with my own project here, but it just feels better to project them upon JavaFBP ;-)

Greetings,

Rob Halff

[Matthew Lai]

On Tuesday, 1 July 2014 14:43:01 UTC-4, Rob Halff wrote:

e.g. when I search for JavaFBP ( https://www.google.nl/search?q=javafbp&oq=javafbp )

I'm not really overwhelmed by information, this has nothing to do however with the production readiness of the implementation.

The main entry to JavaFBP seems to be the book and when I find the source I get redirected to the book and a personal website.

 

The InfoSphere Stream has quite some information if one wants to be overwhelmed:

http://www-03.ibm.com/software/products/en/infosphere-streams/

Or just attend the next InfoSphere Stream's meetup (if one is in Toronto) to get even more overwhelmed by info
as well as pizza slices!

Hack there is even a VM one can try with the big blue's stream; setting it up though, is quite
overwhelming, as described by the IBMers there ha ha.

Yours,

Matt

[Ron Lewis]

On Monday, June 30, 2014 2:01:02 PM UTC-4, Paul Morrison wrote:

This is disappointing!  4 days later, and exactly one person of the 514 (as of today) has posted a response to my plea!  I have also heard informally from one other person, and I know for sure that some of you are working on FBP-like implementations for various devices and systems. 

I do hope that some of you are finding FBP useful, and best regards to all of you!

Paul M.

My FBP effort has been to develop a representation in C/C++ (i.e. C++ without the objects). And using that C/C++ representation, develop a source code obfuscator. The obfuscator drains all meaning from the source code yet the source code can still be compiled (or interpreted for interpreted languages). A data flow diagram maps to a C/C++ text representation, and vice versa, the C/C++ code maps to a data flow diagram. So, this representation of FBP translates either way.

I have these programming elements:

• processors, represented by C/C++ functions.
• connectors, represented by variables.
• connector-types, represented by struct definitions.

I call this Low Level Data Flow Programming (LL-DFP), and use the acronym of the acronym, LD, for short. I feel LD is very appropriate for this representation of FBP because LD looks like an abbreviation for "load". And LD as a programming is preoccupied with whether a connector (represented in C/C++ as a variable) is "loaded" or "unloaded". The basic types of C (int, long, double, etc.) are placed in structs. Here is a struct definition for int:

struct ip_int 

{

bool loaded;

int value;

};

 

On entering a processor (represented as a C/C++ function), the input ports are tested for being loaded or not. Until all processor-required inputs are loaded, the processor will not run, but instead will return. Likewise, the processor-required outputs must be unloaded, or the processor will not run. 

I remember presenting an example of my early attempt at FBP, I forgot what I was doing specifically, but Paul Morrison corrected my design so that I would not have a processor's output feeding back into itself. As I tried to work with LD representation, I saw that I have a problem with feeding a processor's output directly back into itself. The problem is, all processors must reset inputs to unloaded, and set whatever is output to loaded. The same IP on a connector cannot be both loaded and unloaded at the same time. For example, a processor sets its feedback output to loaded, then resets its feedback input (same connector) to unloaded. What just happened is setting its feedback output to loaded when it was already loaded (won't execute unless its feedback input is loaded). Then it sets its input feedback (which it just finished using) to unloaded. Now its feedback output is unloaded. Reversing the order of setting loaded and unloaded does not work. When the processor first starts, it checks its input for loaded. OK. But then it cannot execute because its feedback output is also loaded. I did not understand this until I tried to implement it.

I also follow a rule: A variable can only appear 2 or 3 times: (1) its declaration (for example, "int a"), (2) as an output from a processor, (3) and as an input to another processor. #1 is necessary for the connector that the variable represents to exist in the C/C++ program. When a connector is pre-loaded, then you have #1 and #3. You have #1, #2, and #3 when a connector connects two processors. A variable is not allowed to appear 4 or more times because one variable represents only one connector. If you try to connect one output port to 2 input ports in an FBP flow diagram, then it looks like you really have 2 connectors, not one connector. 

I see that there are five special processors (using the int type as an example).:

• load_int() - takes a regular, plain int, and loads an connector with an IP for that int.
• unload_int() - empties the connector
• pass_int() - This handles the case in which you want the IP to feed back into the processor. The processor passes the IP to pass_int() which then passes the IP back to the original processor.
• dup_int() - This takes one input IP. It replicates the IP on two output connectors. It handles the case in which you need one output to go to two processors. 
• merge_int() - The opposite of dup_int(), takes two inputs, and outputs them one at a time on one output port.

All functions are defined with a void return, for example "void f(ip_int& one_in, ip_int& two_in, ip_int& one_out, ip_int& two_out);" The reason is we are not using any kind of substitution paradigm for LD. Instead, we are using a connection paradigm.

I found that an LD processor quickly becomes very complicated as sub-processors are added. So far, this complexity problem is solved by using the data flow diagram concept of leveling, and composing processors of very few sub-processors.

There is a lot more to say and explain. And this is but an overview of what I have been doing. Any corrections? Comments? Questions?

[Paul Morrison]

Good comments, Rob!  This cuts to the heart of how you find stuff, and get visibility, on the web.  I assume you looked up JavaFBP in the Netherlands because I had mentioned it.  I would feel it more likely that people would Google "Flow-Based Programming", "FBP" or "Better way to program applications".  Over here (Canada actually), the first one of these give 32,600 hits, of which the 3rd and 4th are references to NoFlo, which uses the term "Flow-Based Programming", and shares a number of its characteristics, although, as we have discussed on this Group, it is different in some fundamental ways.  

Googling FBP gives 1,620,000 hits, of which the 3rd is a Wikipedia disambiguation page, which shows our FBP as its first entry.  There are 8 others - not surprisingly, as FBP is a 3-letter acronym!  A 4-letter acronym would probably have given fewer alternatives!

I tried to leave a trail of breadcrumbs, so everything is cross-connected.  And in fact, you're right - I do want everything to lead back to the book, because, for so many of the comments in the Group, the only real answer is RTFB!  Sorry, but that's the answer!  FBP is a paradigm change, and IMO you can only learn it by trying it on real apps.

Sorry, I didn't understand this: "The main entry to JavaFBP seems to be the book and when I find the source I get redirected to the book and a personal website."  Are you saying that you were hoping for something that you didn't find?  If so, can you explain what that is?

Re your other point, examples are difficult:  if it's too small, e.g. Hello World, nobody takes it seriously - in fact we don't have a Hello World!  If it's too big, TL;DR.  The point is in fact that, as FBP applications get bigger, they don't get exponentially more complex!  But how do you prove a negative, except by living with an app for a number of years, as I have.

Maintainability is key - and that is what FBP delivers, because it is a different way of looking at building apps - not a different way of programming!

The app that is AFAIK still running after almost 40 years is now being maintained by kids who weren't born when it was written - and I can tell you that some of the people who maintained it in between were a bit weak on the concepts - but they could still maintain it, without destroying it!  

Support is not a problem - anyone who speaks Java could maintain it.  Services - not sure which you mean...?  I think there is quite a lot of documentation - maybe it needs restructuring...?  Any suggestions?

So maybe you could tell me - what would you like to see as an easy-to-read entry into FBP?   If you tell me you are too busy to read the book, then I believe we have a problem!  Do you believe that you can give an engineer a pile of girders, and some documentation, and tell him/her to build a bridge?  FBP IMO is an engineering discipline, and I don't know if there is an easier way in than just to give it a try!

Comments, anyone?

Regards,

Paul M.

--

[John Cowan]

Paul Morrison scripsit:

> So maybe you could tell me - what would you like to see as an easy-to-read
> entry into FBP?

"Read" may be the problem.

Consider making a video. The dynamic nature of FBP would be well-suited
to this, and I observe that more and more only the barest documentation
is in prose, and explanations and such are left to videos. I personally
hate this, as I can read the transcript of a 60-minute video in much
less than 6 minutes, but it seems to be a key to popularity.

--
John Cowan http://www.ccil.org/~cowan co...@ccil.org
At the end of the Metatarsal Age, the dinosaurs abruptly vanished.
The theory that a single catastrophic event may have been responsible
has been strengthened by the recent discovery of a worldwide layer of
whipped cream marking the Creosote-Tutelary boundary. --Science Made Stupid

[Paul Morrison]

That's interesting, John.  I agree with you about the time it takes to watch videos - I always check the run time before even starting them.  But I have noticed that they are getting more and more prevalent.  Also, this is still more technology to learn!  Still, it's an excellent idea!

Thanks and regards,

Paul

[Tom Young]

I have the same problem with most video manuals.   Some are OK.  They are not too bad if broken up into short segments and/or to help explain text.   Main problem I see is the pacing.   A way is needed to speed up or slow down and maintain audio quality.

           Cheers,

                         -twy

Thomas W. Young, Founding Member
Stamford Data, LLC
47 Mitchell Street, Stamford,  CT  06902
Phone: (203)539-1278
Email: TomY...@stamforddata.com

[Rob Halff]

On Wednesday, July 2, 2014 5:18:59 AM UTC+2, Paul Morrison wrote:

Good comments, Rob!  This cuts to the heart of how you find stuff, and get visibility, on the web.  I assume you looked up JavaFBP in the Netherlands because I had mentioned it.

 

I've looked at the source before as a reference, for example to see how packets are implemented.

 

  I would feel it more likely that people would Google "Flow-Based Programming", "FBP" or "Better way to program applications".  Over here (Canada actually), the first one of these give 32,600 hits, of which the 3rd and 4th are references to NoFlo, which uses the term "Flow-Based Programming", and shares a number of its characteristics, although, as we have discussed on this Group, it is different in some fundamental ways.  

I think a post about the noflo kickstarter was the initial information packet which re-triggered my interest.

Googling FBP gives 1,620,000 hits, of which the 3rd is a Wikipedia disambiguation page, which shows our FBP as its first entry.  There are 8 others - not surprisingly, as FBP is a 3-letter acronym!  A 4-letter acronym would probably have given fewer alternatives!

 How about fbpx? :p

 

I tried to leave a trail of breadcrumbs, so everything is cross-connected.  And in fact, you're right - I do want everything to lead back to the book, because, for so many of the comments in the Group, the only real answer is RTFB!  Sorry, but that's the answer!  FBP is a paradigm change, and IMO you can only learn it by trying it on real apps. 

Sorry, I didn't understand this: "The main entry to JavaFBP seems to be the book and when I find the source I get redirected to the book and a personal website."  Are you saying that you were hoping for something that you didn't find?  If so, can you explain what that is?

Nothing wrong with how everything is cross-connected, but the piece I miss in regard to JavaFBP is something like how nextflow, introduced above, presents itself to the user: http://www.nextflow.io/ 

 

Anyway, I'm just mentioning JavaFBP because you ask for someone to take a chance on it, a clear entry into JavaFBP (website) would encourage one to try it out.

Re your other point, examples are difficult:  if it's too small, e.g. Hello World, nobody takes it seriously - in fact we don't have a Hello World!  If it's too big, TL;DR.  The point is in fact that, as FBP applications get bigger, they don't get exponentially more complex!  But how do you prove a negative, except by living with an app for a number of years, as I have.

Maybe by providing an example "by using the data flow diagram concept of leveling" as Ron Lewis mentions within his problem domain, (I just take out the fragment and apply it over here). The top level example is simple, the deeper you dig, the more you gain knowledge about how the example actually works, until you dig so deep you arrive at the point you will have to look at the source code of a non-composite component itself. 

 

Maintainability is key - and that is what FBP delivers, because it is a different way of looking at building apps - not a different way of programming!

The app that is AFAIK still running after almost 40 years is now being maintained by kids who weren't born when it was written - and I can tell you that some of the people who maintained it in between were a bit weak on the concepts - but they could still maintain it, without destroying it!  

That's great and would be even greater if software written today could last for just as long.

 

Support is not a problem - anyone who speaks Java could maintain it.  Services - not sure which you mean...?  I think there is quite a lot of documentation - maybe it needs restructuring...?  Any suggestions?

Support on re-usable components / graphs, how to use things, best practices for the implementation, etc. not per se maintenance of the source of a base component.

 

So maybe you could tell me - what would you like to see as an easy-to-read entry into FBP?   If you tell me you are too busy to read the book, then I believe we have a problem! 

 

  Do you believe that you can give an engineer a pile of girders, and some documentation, and tell him/her to build a bridge?  FBP IMO is an engineering discipline, and I don't know if there is an easier way in than just to give it a try!

My point was not really reading into FBP, but promoting FBP implementations to be used within production.

And then of course you'll also want to read how all these things work internally and learn all of it's concepts, which can be found in the book.

Matthew referred to InfoSphere Stream, to me it's not clear whether this means JavaFBP is at the core of it,

if so I'm way over my head here (I already am anyway) :-) 

 

Comments, anyone?

Regards,

Paul M.

On Tue, Jul 1, 2014 at 2:43 PM, Rob Halff <rob....@gmail.com> wrote:

I think the main problem for production usage is the way a project is presented.

If as a developer you know your system is capable enough to be used within production,

the way you present and advocate it becomes important.

e.g. when I search for JavaFBP ( https://www.google.nl/search?q=javafbp&oq=javafbp )

I'm not really overwhelmed by information, this has nothing to do however with the production readiness of the implementation.

The main entry to JavaFBP seems to be the book and when I find the source I get redirected to the book and a personal website.

Visibility, support, documentation, examples, services etc. are important.

All that boring stuff developers are not fond of, because well the code works so what's the problem :-)

If there is already an entry-level problem for developers, for managers to adopt it will be even less feasible.

Academics will adopt fbp more easily I guess, because they live within an environment where they are mostly

free to choose or make their own tools, the developer decides.

Stories of long running implementations are great, however hidden away within a newsgroup with mainly only idlers won't cut it.

Explained thoroughly as use cases or (real) testimonials on a website, they would be good selling points.

At the moment I'm writing my own fbp-ish system and try to obscure it as much as possible,

hence I will not yet mention it's name and I know my project is not ready for production at all and maybe it will never be.

Saying this, I think I'm actually explaining the issues I have with my own project here, but it just feels better to project them upon JavaFBP ;-)

Greetings,

Rob Halff

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-programming+unsub...@googlegroups.com.

[Oleksandr Lobunets]

I agree with Rob here. Nextflow is well presented although it does not fit the needs it provides all affordances to spend some time and learn it. Humans always pay attention to attractive things. Even code-writing cyborgs. I think the JavaFBP community will only increase after moving JavaFBP to github and setting up the proper website, read me, examples and API documentation.

[Paolo Di Tommaso]

It looks like that the more interesting part in the Nextflow project is .. the web site! ;)

Just kidding. Regarding this discussion I agree on that Github could help increasing the visibility of JavaFBP, it defines a code collaboration workflow widely adopted that really helps sharing and collaborate on open source projects. 

Also I think you should provide more intuitive examples. I've tried the "MergeandSort" example when downloading it from sourceforge, but frankly I didn't understand what it should show.

Lastly since JavaFBP flow are supposed to be Java classes (If I understood well) you should provide a build/script for that. Set up a Java build process could be a challenging task even for a Java guy. 

Hope this can help you with JavaFBP. 

A very last thing about Nextflow, I don't want open side discussion about it in this thread but I would be interested to understand the Oleksandr's comment. What do you mean by "it does not fit the needs it provides all affordances to spend some time and learn it" ? 

Best,

Paolo 

 

--
You received this message because you are subscribed to a topic in the Google Groups "Flow Based Programming" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/flow-based-programming/hfYw1gP7FVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to flow-based-progra...@googlegroups.com.

[Oleksandr Lobunets]

On Thursday, July 3, 2014 10:49:59 AM UTC+2, Paolo Di Tommaso wrote:

A very last thing about Nextflow, I don't want open side discussion about it in this thread but I would be interested to understand the Oleksandr's comment. What do you mean by "it does not fit the needs it provides all affordances to spend some time and learn it" ? 

Paolo, I missed the word that probably impacted the understanding. The original intention was to say "it does not fit MY needs". Not a general statement of course! I'm sure the nextflow is solving problems for people if was designed fore.

Regards,

Oleksandr

[wp11033256-ole wp11033256-ole]

Hi Paul and all,

I was able to introduce flows as a light wight pattern for a Java project at work.
The decoupled operations work really nice there. :-)

I have taken a short look at JavaFBP. But it seems to use a different thread for every operation. This overhead limits the minimum size of the operations because it wont make sense to have operations where the overhead is bigger than the real work done. The approach with many thread makes debugging MUCH harder. Special support by the framework is needed because normal stacktraces are useless.

I think data is send between the operations as special Java objects. So everything has to be converted from POJOS to these and back again when interfacing with normal Java libraries.

Altogether JavaFBP feels quite un-Java-like. My own creation ( https://github.com/flowdev/jbase ) is quite hard to adapt for many Java devs already.

Best regards

Ole
--
Trettlachstr. 34
91301 Forchheim
Tel.: 09191/79 40 838
Mobil: 0171/10 7 90 23

[Paul Morrison]

Interesting comments, Ole.  However, personally, I haven't found debugging to be any harder, but I use Eclipse, and there, if a process crashes, I get a nice stack trace for that thread only.  Plus JavaFBP gives me all the advantages of configurability - you can add display processes, create scaffolding, etc.  Plus a library of pre-debugged components!

I didn't quite understand what you meant by special Java objects - granted a JavaFBP Information Packet is an object of a particular class, but its payload is held in a single field called "content", so you just access it by calling the getContent() method.  You can easily reload it by using a Component.create() method call.  So, I don't see the problem.     I may be missing something - but maybe it depends what you are used to!

Regards,

Paul M.

--

[Alfredo Sistema]

I think that an exclusive site for JavaFBP is necessary, and hosting it in github is a good idea. "How To" videos are nice thing to watch when you are on a break or a bit tired, they are also very shareable.

But the biggest problem in my opinion is that we need concrete examples of software made with FBP to fiddle with, well documented and ready to run.

[Rob Halff]

On Thursday, July 3, 2014 10:49:59 AM UTC+2, Paolo Di Tommaso wrote:

It looks like that the more interesting part in the Nextflow project is .. the web site! ;)

Just kidding. Regarding this discussion...

It was indeed regarding this discussion, the project deserves it's own thread I think.

Whatever happened to threaded views anyway? 

This google groups thing should have a promote to new thread button or something (new thread)

Ah well, at least I can choose between Comfortable, Cozy or Compact, that's a start :-)

About those other threads, I wonder if it makes sense to run each subgraph within it's own thread(s).

This way if you really want just one component to run within it's own thread, you can create a new subgraph

with only one component, larger graphs you can also chop up into subgraphs and configure them with their own run configuration.

I'm not really a thread expert though, I only know they exists as a concept. (Javascript noob)

[Lars Kluge]

Hi all, 

I was actually looking libraries / best practices to model workflows on Node.js and found out about this all—so not the worst SEO I guess. 

I'm very much in love with FBP and NoFlo, but the start is very, very painful. I wrote some thoughts down: http://www.inkpad.io/AvGzXFfn2e -- this pad is editable (press i or click the image at the bottom), feel free to contribute or fix my mistakes / misunderstanding. This is what comes in mind first. I'm in this for two days now and didn't get far. Frustrating. 

And yes, I actually have the idea to use NoFlo in production for a new product I'm building. It involves a complex flow and NoFlo would be a perfect fit, but I'm not sure yet if it's more helpful than painful to use. 

The IRC channel was helpful already, but no there is nobody replying anymore. Support is limited and other material too. This def. harms broader adaption. 

However, I hope to solve the current issues very soon—also any support is very much welcome. 

Cheers,

Lars

On Friday, June 27, 2014 6:12:07 AM UTC+7, Paul Morrison wrote:

Every morning a bunch of posts come into my inbox in connection with the FBP Group, but I notice that the majority of them seem to be basically from people developing graphical or data streaming tools, rather than struggling to use existing tools.  NoFlo/flowhub is borderline, as a lot of people seem to be working at extending it/them.

I had expected that more of these posts would be from people trying to use FBP for real-world problems, as one would use Java or COBOL...  Now I know a lot of you are musicians and artists and you probably are using various FBP-like dialects to write music or do art, but where are the payroll and inventory developers?!  Especially as, when I did a survey a couple of years ago, 3/4 of the respondents self-identified as business, rather than academic.   I have appended the pie-chart from this study.

Now, I spent 35 years in business, and I am sure very few business are going to bet the store on some untried technology, so I have to asume that the business types among you who are developing neat software are in fact in some R & D department hidden away in a lab somewhere...  Personally speaking, in my case FBP went further when we were refining it in the context of real-world problems - and I had more fun!  The only products I worked on in bona fide labs (before FBP) never actually went anywhere - and a very mature FBP product that was transferred to a lab got strangled in red tape, literally!

Given that there are 512 members currently listed on the FBP Google group, of which we typically hear from probably less than a dozen, I would love for some of the others to let us know:

- if they are using some dialect of FBP for production work - if so, which dialect

- if they tried it and couldn't use it, or it broke and they gave up - again, which dialect, and what happened

- if they find the concepts interesting, but can't see the relevance to what they are doing

- they would like to use it, but management has decreed that they have to use SAP, say

Please note, we know that some of you are sworn to secrecy, so we don't want you to reveal any trade secrets, or give away intellectual property...

But just a tiny hint...?  Is that too much to ask?!

TIA

Paul M. (aka Morpheus - sorry, in joke)

[Paul Morrison]

Hi Rob,

It sounds like you want to have "islands" of processes where an island has a thread, and the processes within an island use green threads or some other light-weight scheduling mechanism.  I note Java itself abandoned green threads in favour of native threads, but the Squawk project - http://en.wikipedia.org/wiki/Squawk_virtual_machine - might have some interesting ideas that we could use...

Regards,

Paul

--

You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.

[Rob Halff]

Hi Paul,

I don' t really feel I'm knowledgeable enough to give a fit enough reply regarding Java from my Javascript island, however...  :-)

  "But it seems to use a different thread for every operation. This overhead limits the minimum size of the operations because it wont make sense to have   operations where the overhead is bigger than the real work done." 

What I understand from looking at the sourcecode of JavaFBP:

 - A component is a thread and starts itself.

 - A network is also a component and thus a thread.

 - A subnet is a network and etc.

 - A network initializes/starts a component.

instead of always having a component starting it' s own thread, it could be externalized, so the network can decide.

Over simplified, within network.java:

Component c = new Component(); // Runnable

if(componentIsConfiguredToUseThread( c )) {
  Thread myThread = new Thread( c );
  myThread.start();

} else {

  // normal start within current thread.

}

Where Component could be a subnet or normal component.

Maximum flexibility could then be achieved by making these parameters configurable (somewhere).

This way the choice is up to the component developer or even the designer of the network to define when new threads are created.

If then none is configured to use threads at all, effectively the entire network would just run within one main thread.

Greetings,

Rob Halff

[Oleksandr Lobunets]

On Sunday, July 6, 2014 10:57:21 PM UTC+2, Rob Halff wrote:

Maximum flexibility could then be achieved by making these parameters configurable (somewhere).

This way the choice is up to the component developer or even the designer of the network to define when new threads are created.

If then none is configured to use threads at all, effectively the entire network would just run within one main thread.

Hi Rob. 

As a designer I just want all my hardware to be used efficiently, I don't want to be a scheduler and distributed works between CPUs/cores.

As a developer I don't know how many cores will be available on the hardware the component will be used. 

Where's the profit of adding yet another configuration option for at the moment premature optimisation?

Kind regards,

Alex

[Armando Drummond]

Today I came back to this group looking for a solution of, what a call, FPB node broadcast problem, and I found your request. 

We are a company in Brasil (www.labcomsistemas.com.br) that produces telecommunications equipment controlled by FBP software since 2003.

Our flow is telephone signaling, our framework is Linux, our nodes are processes, our communicators are pipes.

Now that I'm willing to report our experience, I would like to have some advice from you.  
Thank you.

[Rob Halff]

As a designer I just want all my hardware to be used efficiently, I don't want to be a scheduler and distributed works between CPUs/cores.

As I understand when you create a thread it does not mean you will be mimicking the scheduler itself or choose the distribution between cores, what you do is describing what can be considered one piece of work.

This article describes it as 'a single sequential flow of control within a program.':

   http://www.oracle.com/technetwork/systems/index-155545.html

To me fbp seems to be very well equipped to decide what this unit should be.

As an equivalent of Java threads I could take web workers for in-browser components/graphs as an example, which introduces the same kind of decision problem for javascript applications.

To make it more convenient for the designer, components could just  flag whether they want to be run within a web worker by default. 

 

As a developer I don't know how many cores will be available on the hardware the component will be used. 

 

But I do think developers can decide whether the components they build are worth putting within a single thread by default. A lowercase component for instance is probably not worth it, as a contrast, an audio converting component is.

This doesn't mean the lowercase component will run within the main thread but it would run within a separate thread of it's subgraph for instance.

Figuring out the number of cpu's is probably something the framework itself should figure out, but at least it will be able to determine the intend of the developer and designer.

Where's the profit of adding yet another configuration option for at the moment premature optimisation?

 

The profit being taking away a barrier some developers have, while not being in the way of users who do not want to be bother with this stuff, something which can be solved by using sensible defaults.

Anyway, regarding my own affords I think 'premature optimisation' would be an understatement :-)

 

Kind regards,

Alex

[Oleksandr Lobunets]

Hello Rob,

I have an impression that FBP tries to solve the issues coming from von Neumann architecture :-) Of course we're still running FBP apps on the von Neumanns machines, I'm not talking about data flow hardware, but the problem you're trying to solve is already solved with green threads: once you map an executing node into a dedicated green thread - it's up to an underlying framework how to multiplex it into system threads/processes. Although I can understand the desire of JavaScript/Node.js developer to do so - node.js is not (yet) a suitable technology for writing multithreaded/concurrent applications (even with recently added pair of crutches like web-workers).

Re designers needs. I'm observing issues even in comprehension of FBP DSL and visual notation) when introducing non-programmers to the tool we're working on. Although the components are very domain-specific they still need to learn the basics of connecting logic, etc. Therefore I'm against any additional parametrisation of how the nodes are executed. Although of course that can be a parameter of the platform itself (hidden from a designer).

Kind regards,

Alex

I still don't understand why you need to configuration to tell the component to be either concurrent or parallel or concurrent/parallel?

[Paul Morrison]

Hi Armando,  good to hear from you!  I think the FBP community would be happy to hear anything you wish to share with them! 

Personally, I would be interested in knowing a little more about the technology you used.  Did you adopt an existing technology, or did you develop your own?   If the latter, did you promote it outside your company?   If so, with what success?   Finally, how well was it accepted within your company?

Thanks in advance,

Paul M.

--

[Rob Halff]

On Friday, July 11, 2014 9:11:40 AM UTC+2, Oleksandr Lobunets wrote:

Hello Rob,

I have an impression that FBP tries to solve the issues coming from von Neumann architecture :-) Of course we're still running FBP apps on the von Neumanns machines, I'm not talking about data flow hardware, but the problem you're trying to solve is already solved with green threads: once you map an executing node into a dedicated green thread - it's up to an underlying framework how to multiplex it into system threads/processes. Although I can understand the desire of JavaScript/Node.js developer to do so - node.js is not (yet) a suitable technology for writing multithreaded/concurrent applications (even with recently added pair of crutches like web-workers).

 

The best I can do at the moment is at least pretend. :-) 

 

Re designers needs. I'm observing issues even in comprehension of FBP DSL and visual notation) when introducing non-programmers to the tool we're working on. Although the components are very domain-specific they still need to learn the basics of connecting logic, etc. Therefore I'm against any additional parametrisation of how the nodes are executed. Although of course that can be a parameter of the platform itself (hidden from a designer).

I guess you are right and if there would be any kind of parameterisation they will have to be much more generic, applying to any platform.

A boolean thread parameter is not one of them.

 

Kind regards,

Alex

 

I still don't understand why you need to configuration to tell the component to be either concurrent or parallel or concurrent/parallel?

 

The idea being that threads could be expensive, flagging the more trivial components could help in reducing the amount threads created.

Anyway, I will humble down on this discussion and limit myself in asking questions instead of raising them (which may or may not be caused by my own lack of knowledge on the subject), besides that I am already way off topic here.  :-)

[Oleksandr Lobunets]

We went way far from the topic :-) This is so typical for theory vs implementation! Actually, would be nice to start collecting implementations (at least their descriptions) somewhere in the FBP wiki... May be here - https://github.com/flowbased/flowbased.org/wiki ?

[Armando Drummond]

Mr. Paul M.

Answering your questions:

1)We develop our technology based in a technology used in the design of the Tropico R switching system. According to that, a system is a group of closed blocks that communicate to each other  uniquely by exchange of messages,possibly in a network. Each block is a finite state machine described in ITU-T SDL, and the messages (as data structures) are documented in a programming language. As a consequence, each block can be implemented and tested individually ( a characteristic  also of FBP). The Tropico project has his own hardware and operating system and we take the challenge of transport this technology to Linux (a open source well succeeded OS).

2) As I said before only now we are aiming  to disclose it.

3)Our company is a small company and was based in a hardware that listens or bypasses the telephone signaling over a E1/T1 channel to/from a serial port, a knowledge of telephone signaling (mainly ISUP) and this technology.

As I said before we want to publish this technology, maybe a paper named "A Linux framework to bidirectional FBP", I stressed bidirectional because our nodes can treat both sides of a stream, I do not know if I am correct in this stress. My I publish this paper in this group? 

[Matthew Lai]

On Monday, 14 July 2014 16:14:39 UTC-4, Armando Drummond wrote:

Mr. Paul M.

Answering your questions:

1)We develop our technology based in a technology used in the design of the Tropico R switching system. According to that, a system is a group of closed blocks that communicate to each other  uniquely by exchange of messages,possibly in a network. Each block is a finite state machine described in ITU-T SDL, 

 

Does SDL support the inport/outport concept? I came across SDL in the 90s but could not remember

if it has formal definition on the ports.

 

As I said before we want to publish this technology, maybe a paper named "A Linux framework to bidirectional FBP", I stressed bidirectional because our nodes can treat both sides of a stream, I do not know if I am correct in this stress. My I publish this paper in this group? 

I believe traditional FBP is unidirectional i.e. IP flows from Node A outport x to Node B inport y but not the other

way round. My own FBP implementation allows data ack/nack flowing back from Nobe B inport y to Node A outport x,

but that is more like an exception to the general FBP implementation. 

A full bidirectional communication is sort of like the communicating FSMs correct?

And how about the bounded buffer between between the Node A outport x and Node B inport y?

Can you configure the maximum number of IPs that can be stored in the buffer? For both directions (since

you mention bidirectional comm)?

Matt

[Alfredo Sistema]

I'm interested in your work, please make a new thread when you have news.

As for bidirectional, that goes against FBP concepts, the usual way of dealing with that is having two unidirectional connections.

--

[Paul Morrison]

I agree with Matt, bidirectional flows seem to run counter to the ("classical") FBP philosophy, unless you are just using the back flow to implement bounded buffers, or provide some kind of back pressure.  The risk in making bidirectional channels the standard is that you could land up with a very synchronous implementation.

I certainly don't see any problem with your publishing your paper in this group, or placing a link on this topic - or a new topic as Alfredo suggested.

By the way, did you look at Erlang at all when designing your system - they seem to be in a very similar application area?

Regards,

Paul

[Henri Bergius]

On Tue, Jul 15, 2014 at 4:13 PM, Paul Morrison <jpau...@gmail.com> wrote:
> I agree with Matt, bidirectional flows seem to run counter to the
> ("classical") FBP philosophy, unless you are just using the back flow to
> implement bounded buffers, or provide some kind of back pressure. The risk
> in making bidirectional channels the standard is that you could land up with
> a very synchronous implementation.

Besides, most cases I've seen of where bidirectional flows might've
been useful can just as well be handled with a connection that loops
back from downstream to upstream. And this gives the additional
ability to plug other components into the flow too.

--
Henri Bergius
Decoupling software, one piece at a time.
http://bergie.iki.fi/
@bergie

[Tom Young]

Show me an undivided physical pipe that can handle simultaneous liquid flow in two directions and I'll consider bi-directtional data flows.

   -twy

"...the shell syntax required to initiate a coprocess and connect its input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

[Armando Drummond]

While I am preparing a new thread:

SDL language is used as a specification and description language not as a programming language.As a specification language you can discuss implementation at the application level and as a description language it works as a documentation of a interface. The program is organized as a sequence of SDL transitions (named state_input) so it is easy to go from the SDL to the program and back.

About bidirectional flow:

Lets implement the protocol stack    A || B || C  (bidirectional pipe  symbol: "||") using unidirectional flows and standard IO

tty-> A, A 1-> tty, A 2 -> B, B 1-> A, B  2-> C, C 1-> B, C 2-> tty

[Thomas Lo]

Paul,

We are a Toronto based software company specialized in institutional financial trading platform. We are currently re-developing our trading platform from ground up again after we have a good successful run with our existing trading platform (3 out of the big five banks+large brokerage house+few mid/small hedge fund prop trading firms as our customers). The reason for the re-development is to accommodate a new business model and at the same time solve some of the fundamental problems we had with traditional OO/multi-threading/RPC approach of developing business software application. We are building a new business software development platform called F1 that is based on data flow programming concept to allow us to develop our trading system which will be declarative, deterministic, and efficient. We found the traditional way of developing business applications has the following issues:

• For business application, we always naturally use the functional orientation to design the system. We want to place an order, cancel an order, get trade confirmation, receive price update, etc. But it seems to be unnatural to use OO to design the system. After all, business people will never tell you what an Order can do. We need software developed in the same way how the business people understand it;

• When a system grows larger and larger, it is more difficult to understand the code without documentation. But in the “real world” no one ever keeps documentation 100% in sync with the actual business application. We need a system that is naturally documented such that business people can visually see how the system works;

• Using traditional thread-based approach to develop concurrent event driven application is simply chaotic. You have to lock here and lock there. Not only is the CPU  being used for waiting, blocking, and context switching most of the time, it is also very deadlock prone since you don't know if the instance method you are calling is trying to lock something else that will create deadlock. We need the system to be running efficiently without any contention and deadlock issue. We need a way to develop concurrent business application easier;

• How many times are we unable to reproduce the same production problem in development hardware just simply because the hardware is different and it changes the threading sequence? We need the system to be absolutely deterministic where we know exactly what the input is and no matter what hardware we run the code on it will produce exactly the same output

For F1, we are injecting some of the advanced concepts of the data flow implementation to make sure it will be suitable for financial trading system development (and any other high demand business application development including gaming) including:

• F1 Data Model Extension (DMX) which allows designer to define domain data declaratively in a real time collaborative web environment. DMX categorizes data into

• Persistent Data - with data store persistence and query capability for eventually consistent data access

• Event Data - immutable input data to trigger Application State Data change in the task of the flow. Event Data naturally forms the API to access the system

• Application State Data - atomic mutable data changed by task of the flow based on incoming Event Data

• DMX is backed by our own elastic binary encoding which can be used as fixed length encoding for off heap data access and variable length encoding for sending to wire. Using elastic binary encoding instead of POJO to eliminate the need to perform serialization & deserialization;

• Fast messaging via memory mapped file (We are using Peter Lawrey's Java Chronicle) with replay capability;

• F1 Flow Model Extension (FMX) allows designer to define business logic declaratively in a real time collaborative web environment. This provides a naturally documented software platform. The visual flow of the logic is exactly the same as the running software. FMX consists of Task and Flow. Each Task has a single Input (we don’t allow multiple inputs. We want to enforce each Task is focus and simple. If a Task requires more than 1 Input then it should be split to multiple Tasks) and multiple Outputs. FMX will generate Java flow implementation and developer will need to develop the actual underlying logic for the task;

• F1 execution will rely on our context based multi-threading approach where Event with the same context is guaranteed to be executed on the same thread (Events with different contexts could be executed on the same thread or different thread). This will eliminate the need to use locking and programming will become much simpler since everything is executed in a single threaded manner. Each execution thread will be affined to the specific core of the machine to avoid context switching;

• F1 has a totally separate unit of I/O performing network input output operation. Input will be based on a single busy spin to select Input Events based on the time of arrival and flow into the corresponding Flow Processing unit based on the context. In this way we can guarantee the fairness of the input sequence;

• F1 will also have an in-memory Persistent Data Cache to provide unified, eventually-consistent Persistent Data read access to the Flow Task via shared memory

We are busy working on the F1 platform and in parallel we are developing bits and pieces of the trading system to make sure F1 is on the right track.

The toughest problem we are facing is how to change a developer's mentality to use the Flow-based development concept instead of traditional OO as developers have been trained for so many years and it will take time to adjust the mentality to use Flow concept. Our objective is to build a world class collaborative business software development platform where business people and technical people can build the high performance business software together. Although we are not in production yet, but will definitely keep you posted on how it will go in production.

[Alfredo Sistema]

Sounds like a great case for FBP, keep us updated. The mentality problem should not be that hard if the language used is the same as before, as long as the graph architect really gets it, the component developers don't have to worry about the external world.

--

[Thomas Lo]

Thanks and we will for sure. We are currently setting up a blog to journal the design and development of F1. And once we feel F1 is mature enough we will open source & open service the entire thing as a give back to the technology community and also get the feedback from the community to make it a better platform. 

In terms of the mentality, it is more changing some of the typical use case from a procedural perspective to event flow perspective. For example, we are currently working on allowing Task to issue Data Query where the resulting Data from the Data Query will come back as a framed sequence of events and another Task will handle it one by one. Which is a bit different then having the same set of code to issue the query, wait for the result set, and loop through the result set. But like you said as long as the designer group gets it, then we can distribute each Task to a lower level of developers to implement and they don't need to worry about the external world. 

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-programming+unsub...@googlegroups.com.

[Paul Morrison]

On Wed, Jul 16, 2014 at 12:48 PM, Thomas Lo <thom...@financialogixgroup.com> wrote:

Thanks and we will for sure. We are currently setting up a blog to journal the design and development of F1. And once we feel F1 is mature enough we will open source & open service the entire thing as a give back to the technology community and also get the feedback from the community to make it a better platform. 

That's really refreshing!  We really need experience with FBP to be pooled, so people can build on what has gone before.

In terms of the mentality, it is more changing some of the typical use case from a procedural perspective to event flow perspective. For example, we are currently working on allowing Task to issue Data Query where the resulting Data from the Data Query will come back as a framed sequence of events and another Task will handle it one by one. Which is a bit different then having the same set of code to issue the query, wait for the result set, and loop through the result set. But like you said as long as the designer group gets it, then we can distribute each Task to a lower level of developers to implement and they don't need to worry about the external world. 

Interestingly, that's very similar to the diagram on the front of the 2nd edition of my book!  And I agree about changing the mindset - we have found the "data factory" image to be helpful - but the old von Neumann mindset is very entrenched!

Looking forward to hearing how it goes - and of course, I think I can safely say that we are all available to help in any way we can!

Best regards,

Paul M.

  

[Paul Morrison]

Totally agree!  And as I think Tom said, it's unusual to see a river with water going both ways!

On Tue, Jul 15, 2014 at 10:19 AM, Henri Bergius <henri....@iki.fi> wrote:

[John Cowan]

Paul Morrison scripsit:

> Totally agree! And as I think Tom said, it's unusual to see a river with
> water going both ways!

Architecturally, FBP flows are one-way. That doesn't mean they can't be
multiplexed over bidirectional TCP connections. (For that matter, in
classical FBP, unidirectional object flows are multiplexed over bidirectional
memory.)

--
John Cowan http://www.ccil.org/~cowan co...@ccil.org
When I'm stuck in something boring where reading would be impossible or
rude, I often set up math problems for myself and solve them as a way
to pass the time. --John Jenkins

[Thomas Lo]

On Wednesday, July 16, 2014 1:12:07 PM UTC-4, Paul Morrison wrote:

On Wed, Jul 16, 2014 at 12:48 PM, Thomas Lo <thom...@financialogixgroup.com> wrote:

Thanks and we will for sure. We are currently setting up a blog to journal the design and development of F1. And once we feel F1 is mature enough we will open source & open service the entire thing as a give back to the technology community and also get the feedback from the community to make it a better platform. 

That's really refreshing!  We really need experience with FBP to be pooled, so people can build on what has gone before.

Totally agree on pooling the FBP experience. We have been planning this since last year and the problem set we are dealing with is probably just tip of the iceberg.

 

In terms of the mentality, it is more changing some of the typical use case from a procedural perspective to event flow perspective. For example, we are currently working on allowing Task to issue Data Query where the resulting Data from the Data Query will come back as a framed sequence of events and another Task will handle it one by one. Which is a bit different then having the same set of code to issue the query, wait for the result set, and loop through the result set. But like you said as long as the designer group gets it, then we can distribute each Task to a lower level of developers to implement and they don't need to worry about the external world. 

Interestingly, that's very similar to the diagram on the front of the 2nd edition of my book!  And I agree about changing the mindset - we have found the "data factory" image to be helpful - but the old von Neumann mindset is very entrenched!

I should probably pick up your book instead of killing our brain cells on certain problem that has been solved before already by the experts :) 

Looking forward to hearing how it goes - and of course, I think I can safely say that we are all available to help in any way we can!

Thanks in advance and will  take all the help as much as possible! And appreciate the generosity. 

[Sam Watkins]

On Tue, Jul 15, 2014 at 01:51:21PM -0700, Armando Drummond wrote:
> About bidirectional flow:

how do you define bidirectional flow?

[Armando Drummond]

Sorry if I am late.

I can give a example of 2 bidirectional flows: TCP/IP and /dev/tty

[Ged Byrne]

Is that the same as the Request-Reply message pattern?

http://www.eaipatterns.com/RequestReply.html

--

You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.

[henrique matias]

Unfortunately i did not manage to read the whole thread, but would like to add some thoughts based on my personal experience.

My first contact with FBP was more/less 6 years ago when i first played with Pure Data, my mind was blown away, for long long time i was only writing code "by typing". Was clear for me that this would change my "programmer mind" and would bring me way more inspiration and problem thinking skills than learning another "written language" or "design pattern". 

End up that i only did FBP on my free time for a few years.. Was the "new fun way" of programming. This really helped shaping my "written code" skills and i could bring a lot of insights to my day-by-day, aka "production" works. 

Node js really helps on this, creating "modules" ( aka one-file-per-function ) is a bliss.

The learning curve was steep, i guess not as steep as someone that never programmed before, but after a few months i could already think how to write any of my "regular code" in a "FBP" way, felt very good and  "interactive" and "refactorable" and "truly OO"  :P

To be honest i could only achieve "production grade" results and "fast learning curve" once i moved into Max/MSP, by this time it was already Max / MSP 5, with a new "Help" / "Documentation integration", and to be honest i think that is where most of the FBP implementations falls apart : Help and Documentation.

I believe "making production work" with a technology is very dependant on "How easy and fast can someone learn on his own" just by googling or opening the "IDE", only a modern and forward thinking company would re-start development of their product using "FBP" i think most of the companies that already have a solution in place would not stop to re-write in an FBP for all non-technical reasons, like "being afraid of changes" or "not having time to learn something new", unless someone strikes in with a solution that would be so neat that even "people without time to learn something new" would "make time to learn this new super cool thing".

From an "end user" i would think:

 - Can i right click and object and get instant Help / Documentation?

 - Can my co-workers learn on their own ? 

 - How many clicks will take them to download the app and get hooked by a running example?

 - Will they get distracted by "too much information" in case they are beginners?

From an FBP lover perspective i think:

 - F&$%! How all those programmers never ever had any contact with such "paradigm" i bet they would be blown away, would be great if everyone would suddenly be into it and help making tools in order to be able to realise our day-by-day work using such "re-mixable" code.

Needless to say i really think NoFlo is awesome, it is open source, it is crowd-funded and runs on the web ( and not only ), i don't think i have to say anything else, if you are here, you know what that means ( :

i hope it makes some, forgive me for my 8:40 AM email.

Have a nice week guys, and hopefully i'll manage to read all the thread soon.

peace

--

❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ 

❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ ❂ ❁ 

[Paul Morrison]

Hi everyone!

For the last few weeks I have again started wondering if anyone out there in the application development universe is using ("classical") FBP for real-life applications, as I am still getting emails from people proposing to implement FBP in yet another programming language which I may or may not have heard of!

So I decided to browse the Google group... and found this rather plaintive post from a bit over 2 years ago, asking exactly that same question!  Since I don't think I can phrase it any better this time around, I thought I would repost it, and see if the situation has changed over the course of the last 2 and a bit years!

Feedback would be much appreciated - positive or negative!

Paul M.

On Thursday, June 26, 2014 at 7:12:07 PM UTC-4, Paul Morrison wrote:

Every morning a bunch of posts come into my inbox in connection with the FBP Group, but I notice that the majority of them seem to be basically from people developing graphical or data streaming tools, rather than struggling to use existing tools.  NoFlo/flowhub is borderline, as a lot of people seem to be working at extending it/them.

I had expected that more of these posts would be from people trying to use FBP for real-world problems, as one would use Java or COBOL...  Now I know a lot of you are musicians and artists and you probably are using various FBP-like dialects to write music or do art, but where are the payroll and inventory developers?!  Especially as, when I did a survey a couple of years ago, 3/4 of the respondents self-identified as business, rather than academic.   I have appended the pie-chart from this study.

Now, I spent 35 years in business, and I am sure very few businesses are going to bet the store on some untried technology, so I have to assume that the business types among you who are developing neat software are in fact in some R & D department hidden away in a lab somewhere...  Personally speaking, in my case FBP went further when we were refining it in the context of real-world problems - and I had more fun!  The only products I worked on in bona fide labs (before FBP) never actually went anywhere - and a very mature FBP product that was transferred to a lab got strangled in red tape, literally!

Given that there are 512 members currently listed on the FBP Google group, of which we typically hear from probably less than a dozen, I would love for some of the others to let us know:

- if they are using some dialect of FBP for production work - if so, which dialect

- if they tried it and couldn't use it, or it broke and they gave up - again, which dialect, and what happened

- if they find the concepts interesting, but can't see the relevance to what they are doing

- they would like to use it, but management has decreed that they have to use SAP, say

Please note, we know that some of you are sworn to secrecy, so we don't want you to reveal any trade secrets, or give away intellectual property...

But just a tiny hint...?  Is that too much to ask?!

TIA

Paul M.

[Matthew Lai]

I'm still running my winfbp (tcl implementation) once a week to extract financial data from public sites like tmx.com, globeinvestor.com, ca.dividendinvestor.com and finance.yahoo.com.

Another real-life application is to extract English to Chinese translations (eg. from Cambridge English to Chinese website) and imbed them into stories from public domain sites (eg. "As a Man Thinketh" from gutenberg.org). The current graphs looks like this

 

and the sample output of the text with imbedded translations look like this

Thought and Character(（人的）品質；性格；（事物的）性質；特性)

The aphorism(格言；警句；箴言), "As a man thinketh in his heart so is he," not only embraces(包括，包含) the whole of a man's being, but is so comprehensive(廣泛的；無所不包的；綜合的) as to reach out to every condition and circumstance(情況，環境；情勢) of his life. A man is literally(實在地，不加誇張地) what he thinks, his character being the complete sum of all his thoughts.

As the plant springs(迅速生長) from, and could not be without, the seed, so every act of a man springs from the hidden seeds of thought, and could not have appeared without them. This applies equally to those acts called "spontaneous(自發的；非出於強制的)" and "unpremeditated(無預謀的；偶然的)" as to those which are deliberately(慎重地；謹慎地) executed(實施，實行；執行；履行).
...

abandoned 被放棄的；被遺棄的 
absolute 純粹的；完全的 絕對的 
abuse 濫用，妄用 
alter 改變 
aphorism 格言；警句；箴言 
application 應用，適用；運用 
armory 軍械庫 
artifice 奸計；詭計 

...

[Paul Morrison]

很有意思 !  Thanks for posting!  Any other neat stuff... anyone?!

Paul M.

[David Ryman]

Morpheus,

I am from the Other slice now, having resided in both biz and acad at some point.

What I think I am doing is looking at FBPhilosophy as the core technology to hang existing technology on.

I am still having problems with the name, Flow Based Programming does not say enough and the letter g does not work on my main keyboard (haha), Flow Based * works though, FBPhilosophy, FBControl, FBNetwork.

I started in the Industry in the mid '70s with punched paper tape and MOP terminals. One of the 'kickers' in my mind is electronic lists, or menus. Everything has menus. Windows is half menus.

I do not know exactly what FBP would do to menus, but if FBP is destined to be the Giant's platform from which the next revolution launches, it has to be everywhere.

More oolin ahead,

David (ANeo)

On Thursday, June 26, 2014 at 6:12:07 PM UTC-5, Paul Morrison wrote:

Every morning a bunch of posts come into my inbox in connection with the FBP Group, but I notice that the majority of them seem to be basically from people developing graphical or data streaming tools, rather than struggling to use existing tools.  NoFlo/flowhub is borderline, as a lot of people seem to be working at extending it/them.

I had expected that more of these posts would be from people trying to use FBP for real-world problems, as one would use Java or COBOL...  Now I know a lot of you are musicians and artists and you probably are using various FBP-like dialects to write music or do art, but where are the payroll and inventory developers?!  Especially as, when I did a survey a couple of years ago, 3/4 of the respondents self-identified as business, rather than academic.   I have appended the pie-chart from this study.

Now, I spent 35 years in business, and I am sure very few business are going to bet the store on some untried technology, so I have to asume that the business types among you who are developing neat software are in fact in some R & D department hidden away in a lab somewhere...  Personally speaking, in my case FBP went further when we were refining it in the context of real-world problems - and I had more fun!  The only products I worked on in bona fide labs (before FBP) never actually went anywhere - and a very mature FBP product that was transferred to a lab got strangled in red tape, literally!

Given that there are 512 members currently listed on the FBP Google group, of which we typically hear from probably less than a dozen, I would love for some of the others to let us know:

- if they are using some dialect of FBP for production work - if so, which dialect

- if they tried it and couldn't use it, or it broke and they gave up - again, which dialect, and what happened

- if they find the concepts interesting, but can't see the relevance to what they are doing

- they would like to use it, but management has decreed that they have to use SAP, say

Please note, we know that some of you are sworn to secrecy, so we don't want you to reveal any trade secrets, or give away intellectual property...

But just a tiny hint...?  Is that too much to ask?!

TIA

[Paul Morrison]

I totally agree!   Menus are ubiquitous in computing, and IMHO are nicely implemented using the substream concept of FBPhilosophy (I like it!).  From this point of view, menus are just a particular form of list, and very often we see screens with a "fixed" section, and one or more lists or menus (usually not more than a couple).

See Chap. 21 in first edition - http://www.jpaulmorrison.com/fbp/scrmgr.shtml , Chap. 20 in 2nd edition.  Do a find on "substream": 2nd occurrence in Chap. 21 of 1st ed., first occurrence in Chap. 20 of 2nd ed.

As Philip Ewing points out in his comments in my book, another advantage of this approach is that it can be used both for interactive and batch applications - in fact, the same components can often be used in both environments. Philip used this approach very effectively on a project he worked on, actually testing online function in batch - see p. 204 of the 2nd ed.

Hope you et the "" fixed soon!

Reards,

Morpheus

--

You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-programming+unsub...@googlegroups.com.

[David Ryman]

Thought I had posted another reply, but it went away somewhere. One  question to you Paul, is there a video lecture series of the book? If not, it may help get folk off and running quicker.

[Paul Morrison]

Hi David,

Yup, it came as a personal message from google@davidryman.com , so I responded there..

Here is the part about videos:

So far, I have:

- https://youtu.be/up2yhNTsaDs - the interview on the FBP web site

- https://player.vimeo.com/video/79329015 - presentation to Toronto Meetup, based on a Powerpoint presentation

- 3 videos demonstrating DrawFBP and how it interacts with Eclipse and JavaFBP (#3 has some stuff about scheduling rules)

    - https://www.youtube.com/watch?v=OrKenPOV4Js

    - https://www.youtube.com/watch?v=9NXYNxDjFWY

    - https://www.youtube.com/watch?v=-AmzfhV2hIU

If you can stand the quality of the last 3, I can put together some more - what would you find useful?  Suggestions welcome!

Regards,

Paul M/Morpheus

[David Ryman]

 I have been mulling over components, and got very bogged down from the start, and it may come down to our visions of FB*.

I have updated my overview to show what I think of as your bits and Flowhub's bits.

Side note question: Sockets would be links from runtime to runtime, e.g. flowing to a JS component from a Java one? or would javaFBP pass the flow directly?

Your components are ​anything that runs in the java runtime.

My components would be anything that runs in any runtime, which poses a different set of issues, probably one of classification.

A Tier system could be adopted. 

• Tier CeS - simple, core. 
• Tier ClS - simple, contextual. 
• Tier SdS - simple, specialized.
• Tier CeC - compound, core.
• Tier ClC - compound, contextual.
• Tier SdC - compound, specialized.

core being common shared functions, the FBP pseudo code

contextual being specific to domains, e.g. API's, IoT

specialized being a catch all. The Other option!

Each tier may also have stable and proposed layers. FBComponents A and B can be united under AB. FBP flow/graph can test both A + B and AB. I would think AB would outperform A+B, but who knows until we try it eh!

CeS need a benign dictator to oversee them, much like Linus Torvalds and Linux, I would imagine, so I would volunteer you for that bit.

ClS ought to have their own teams

SdC and all the *C tiers could be open to all.

OK. with all these potential sources of FBCs anywhere on the internet plus I have my own locally I would suspect a selection mechanism for which sources I want available for this flow/graph would be in order. DrawFBP would need to be able to rebuild the components list on the fly. The same checklist would be needed by JavaFBP obviously as it will need to know what to run, but it is not a good idea to load everything available.

Perhaps you can see me getting bogged down again, so I'll rest and return.

ANeo,

 

FYI. re Matrix. Elon Musk, Stephen Hawking and Michio Kaku have all express the idea that we do actually live in a Matrix. I thought I heard Kaku can actually prove it.

 

Sorry not to have gotten back to you earlier - things are a bit hectic at home right now as we have to help out our daughter and grandson for a few weeks. 

I really found your note very thought-provoking, and would like to get more of your thinking on these topics - myself, I'll have to grab time as I can get it!

Wasn't too clear what the Tank is - something in Matrix (which I haven't seen)?  Re your last paragraph, it would be great if you could expand on this.  As you may have gathered from the book, I am very keen on componentry and reuse, but I haven't really figured out how we will integrate libraries of components coming from different vendors - maybe in different countries...?  If you look athttps://github.com/jpaulm/javafbp/blob/master/README.md , you will see a section called "Building/viewing Component Attributes List" - maybe I should include a sample section of its output in the Readme...   This is an attempt to automatically generate a searchable list of components, and their attributes, from selected JavaFBP component libraries.  This in turn told me that I needed to add a bunch more component descriptions.   I think this has potential... what do you think?

I look forward to looking at Node Red, but I confess I was put off by the name - still, your example is pretty impressive!   I'll dig into it a bit deeper as time permits.

Best regards - let's talk some more!

Paul M.

PS You forgot Jill Stein!

[Paul Morrison]

Hi Dan,

Looks like David had trouble with his diagram - the next post is OK.  Could you therefore delete this one?

Thanks in advance,

Paul M.

[Henri Bergius]

Hi,

On Thu, Nov 3, 2016, 06:22 David Ryman <goo...@davidryman.com> wrote:

 I have been mulling over components, and got very bogged down from the start, and it may come down to our visions of FB*.

On this diagram you have Flowhub shown in two places: in the "IDE land" on top, and on the "FBP runtime land" below.

I'm assuming you intended to have NoFlo in the same region as javaFBP and JSFBP, not Flowhub. Flowhub is an IDE and can talk to multiple FBP environments.

Also, the Flow/graph box might want to mention the FBP protocol, which is how we achieve these cross-runtime communications:

https://flowbased.github.io/fbp-protocol/

In addition to IDEs, it allows other cross-runtime tooling, like the fbp-spec unit testing tool and Flowtrace (which could be boxes on the top as well).

And in runtime land there are some other potential FBP runtimes to list:

* http://microflo.org/

* https://github.com/msgflo/msgflo#readme

* http://www.imgflo.org/

/Henri

[Paul Morrison]

Hi ANeo,

You get a classification problem, even when you don't mix environments.

Bert Madeira has done some thinking about this, and we did some repackaging of JavaFBP based on his ideas.  Bert, would you care to comment?

Re sockets, my personal view is that you build socket writers and socket readers for each environment, and they can talk to each other using strings.  This means that you will probably need serialization and deserialization components as well for various non-string formats.  You can see an example of this kind of thing in https://github.com/jpaulm/javafbp-websockets , where the two languages involved are Java and HTML5 (?) - JS and HTML anyway!

I seem to remember Bert advocating using XML compressed in some way - Bert, could you remind me?  TIA

And yes, DrawFBP has the potential to be helpful in this area - it already can access multiple component jar files - might be a smidgeon toughe to incorporate C# conventions, but I'm willing to learn! :-)

Best regards,

Paul

[Paul Morrison]

FYI

I built a simple javadoc doclet to generate component names and attributes from the JavaFBP components folder, based on Bert's suggestions  - this doclet can be run any time there are changes to the components list.   Here is part of its output:

ReadFromSocket

ComponentDescription

value - "Generate stream of packets from socket"

OutPort

value - "OUT"

description - "Read packets"

type - java.lang.String.class

InPort

value - "PORT"

description - "Port name"

type - java.lang.String.class

Regards,

Paul M.

[Paul Morrison]

Lets face it, Morrison, nobody trusts your software!

And of course it's much more fun to concoct an FBP-like implementation in your favorite language, and spend a good few months playing with it, testing performance, talking it up, and generally putting off the dread moment when you have to build a functioning system for real live customers.

Of course your own system is much more reliable and user-friendly than any of mine... because you wrote it!  Of course that means you can't test it... for the same reason!  See 3rd  party testing.

So, a challenge to all you Laputans:  build an Update application including a generalized Collate and an app-specific component to process the merged stream... in your favorite language and software.

And leave us a link so we can try it out!  The JavaFBP version is at: https://github.com/jpaulm/javafbp/blob/master/src/main/java/com/jpaulmorrison/fbp/resourcekit/examples/networks/Update.java  - in case you need to look!

Go to it!  May the best sentient win!  And NO brain-f**k!

Paul

--

[Matthew Lai]

A Brobdingnagian task :)

Matt

[Paul Morrison]

Bert Madeira sent the following (I am forwarding his post as he seems to be having problems posting to the Group):

| Hi ANeo,
|
| You get a classification problem, even when you don't mix environments.
|
| Bert Madeira has done some thinking about this, and we did some repackaging of JavaFBP based on his ideas.  Bert, would you care to comment?

So the idea with the packaging of components based on type....

Yes, there should be some sort of generic/reusable core. 
Typically these components should be those that are likely to be used in most typical applications, with high reusability (and well tested).
They should be included/shipped with the engine as basic building blocks, and categorized for discovery according to basic function (basic connections, file IO, network io etc.)
Documentation should be excellent and the components should be able to be used black-box style without need to look at the source code
(trying for a good out-of-the-box/beginner experience).

Then there can be collections of components that are focused on something specific, like audio processing, or mail handling, or some other such
Each collection would essentially be a separate module, according to an area of interest.   Reusability/testing is still high, albeit in a more limited sense than the core components.
You would include these modules only when needed.  They should get reasonable testing and be cohesive in design within each module.
They could be implemented by 3rd parties (even commercial ones) where it makes sense (even have competing alternatives).
Documentation should be pretty good and the components should generally be able to used black box style without need to look at the source code (unless you want to extend something).

The next classification of components would be sample/example/demo code used for tutorials and demos.
Reusability/testing can be low - just need to make sure the examples work out of the box.
This can be packaged separately as a kind of tutorial/cookbook/SDK kind of thing.
Component documentation can be strictly in the code or the tutorial and it would be expected to look at the code so as to understand/extend/modify it.

By having these 3 kinds of packages, you ideally allow someone to be able to build something useful strictly from stock components
(so they can try to compose something and learn FBP without having to go read all the component source code)
and you avoid requiring someone to include demo code in their application (only core and engine in the basic package to keep the application package small).

As new components come in, they will tend to accumulate in demo/samples or in modules
until they are improved to be more generic/robust at which point it may be reasonable to include them in the core.

The idea is to give a new user a way to get a successful taste for FBP with minimal time investment
and then allow them to progress into more depth (such as coding their own components) by investing further as needed/desired.

Re sockets, my personal view is that you build socket writers and socket readers for each environment, and they can talk to each other using strings.  This means that you will probably need serialization and deserialization components as well for various non-string formats. 

I tend to agree, although some specialty environments might need binary.  In which case, serializing to byte arrays will work well enough (in the case of most CPUs, little-endian won out).

You can see an example of this kind of thing in https://github.com/jpaulm/javafbp-websockets , where the two languages involved are Java and HTML5 (?) - JS and HTML anyway!

I seem to remember Bert advocating using XML compressed in some way - Bert, could you remind me?  TIA

So my comment was in the comparison/contrast of different methods of complex data passing.

The basic idea is that data should be passed in a compressed (gzip) format generally.

All modern browsers can handle this and it is part of one or more RFC's. 
This is something that has been taken advantage of since the early days of Google search and should be something that should just generally be done by the server.
(not sure if it is being done in the browser/client yet, but that will likely come up at some point.

So my thesis (based on some prior experience with early compression algorithms) is that generally,
the size of the compressed data stream is related to the complexity of its uncompressed data and not its unpacked size. 
RLE (run length encoding) is generally a component of most compression formats,
and will by itself  go a long way to achieve this effect even before applying  further compression techniques.

As such, encoding a data stream in XML vs encoding it in in binary or in Json will tend to result in similarly sized stream lengths after compression.

The point is that (when using compression) the performance of data passing
is largely the result of the size of the compressed stream (the number of bytes to be passed via the network connection)
and not on the size of the uncompressed data streams on either side of the connection.

This is similar in concept to on-disk file compression.  You gain performance by compressing files to be stored
The CPU processing delay is small enough  that the savings in transfer size way more than offsets it.

Another issue is the parsing of the received data.  Using an XML DOM-based parser is quite a performance hog (which is probably what set off the original Json guys).
But using a SAX-based parser is much more performant as long as you cache/reuse the parser between requests.

In contrast, a typical Json parser tends to create a lot of hashmaps which, while better than DOM-style parsing, are still not very efficient.

On the browser side, this difference in parsing cost is usually a non-issue, since the cost of parsing is paid by only a single user at a time
and generally the browser side has spare CPU to burn (plus it has special code to deal with both Json and XML).

But if your client is not a browser...

In any case, Json is so entrenched in mainstream JavaScript programming culture these days that this sort of argument will likely not make any inroads.
On the other hand - I also don't feel any burning need to follow the rest of the pack.  And if I really have to, I can always pump out Json or whatever else readily enough.

And yes, DrawFBP has the potential to be helpful in this area - it already can access multiple component jar files - might be a smidgeon toughe to incorporate C# conventions, but I'm willing to learn! :-)

Just as a side note, I've just come off of having to deal with a large chunk of C# build automation and deployment.

We did manage to somehow to work it out in the end, but let's just say that the C# tool environment is in a bit of a mess at the present time
(a more choice description omitted in the interest of minimizing offence to casual readers)
A lot of experimentation was required because the tool documentation is worse than useless...
we had to do a lot of digging around in blogs and such, and most of those were wrong or out of date too.

Note that this is not the sort of thing that would affect a typical C# application developer....

In any case, my fervent hope is that the next version of Visual Studio will clean up these issues as they can deter teams less determined to figure it out.

In the meantime. Java works well enough for my needs.  And both Swift and Rust seem to be deserving of investigation (when/if I can find the time).

Best Regards,
--Bert

[David Ryman]

Your assumption is correct.

Both the runtime and FBEngine lands should have a catchall, to be open for expansion.

Thanks for the input.

--
You received this message because you are subscribed to a topic in the Google Groups "Flow Based Programming" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/flow-based-programming/hfYw1gP7FVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to flow-based-programming+unsub...@googlegroups.com.

[Paul Morrison]

Hi Bert,

As you may have seen, I have posted your response - thanks so much for your very complete answer.  The present JavaFBP packaging basically follows your suggestions, except that all the binary is in one jar file - it's only 250 KB, so I didn't see much point in chopping it up!  Alternatively, there is a major division (as per your suggestions) between 'core' and 'resourcekit', so the jar file could be split this way... although 'resourcekit' also contains all the test networks for the core components! 

In addition, there are some components in 'core', e.g. 'audio', which will not be of interest to most users, so they could be split off as well - although the 'audio' components only add up to 13K.  Still we don't want to make the mental model too complex, as we already have a fairly complex directory structure, and if it is chopped up among multiple jar files, that could make it even more complex!

Other vendors will of course provide their own jar files (or multiple) and I believe DrawFBP can handle that - maybe with some minor modifications.

Good conversation - let's keep it going!

Best regards,

Paul

[David Ryman]

Last update: I added bidirectional links between runtimes and generalized a little.

The network in runtime land warrants its own study.

Another penny dropped. In my normal use scenario, all the runtimes I will use are local, so are all the runnable components.

Pass the component info back up the chain !

With that mechanism, the onus would be on the C# runtime developer to interact under FBProtocol with any other runtime that already interact with any of the FBEngines. Bang, the GUI accesses a library of single component flows/graphs. The Circle of life.

I also have direct links between FBEngines and FBRuntimes, which would provide potential performance improvements. Having a FBPremium package could provide a financial leg to FBP, whilst retaining full operability for everyone. For personal use, I do not need it to be fast, just to work. For business, I want micro trading potential.

I bet Bert already said this, as I have not read it yet, I want to give it justice, and I had to get this out of my head.

If I ever come up with an original idea, let me know, I'd like to claim it.

ANeo

​PS the more I think about this, the more I think we are creating The Matrix. You really should watch them, or The Rise of The Machines, I am not sure which.

Funny: 2 keyboards, because 1 has not G, the other no FKeys. Google fibre router wifi out so I'll run a Linksys hardwired to phone, basement system and chromecast. I think I may be a High Tech Redneck. (ref: Jeff Foxworthy)

ANeo

--
You received this message because you are subscribed to a topic in the Google Groups "Flow Based Programming" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/flow-based-programming/hfYw1gP7FVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to flow-based-programming+unsub...@googlegroups.com.

[David Ryman]

Oops I did it again!

[David Ryman]

The idea is to give a new user a way to get a successful taste for FBP with minimal time investment 
and then allow them to progress into more depth (such as coding their own components) by investing further as needed/desired. 

That's me.

Artists rendition of backward propagation of components.

So my thesis (based on some prior experience with early compression algorithms) is that generally, 
the size of the compressed data stream is related to the complexity of its uncompressed data and not its unpacked size.  
RLE (run length encoding) is generally a component of most compression formats, 
and will by itself  go a long way to achieve this effect even before applying  further compression techniques.

Made me think of encryption. At some point an IP may have to travel distance and require encryption. I would think encryption would add a layer of complexity so which is better encrypt-compress or compress-encrypt? or is this not a thing?

[Humberto Madeira]

Hi David,

I must admit I have been too busy lately to take a good look at NodeRed (I am assuming that is your main focus?)

Some personal background: I am working on my own flow engine platform (not out yet and not open source).
My current working code name for this platform is "Chicle"

Chicle does share some concepts with classic FBP but is dissimilar enough that I have trouble putting it into a specific category.
It is different both in the type of flow it supports and in other requirements not related to flow, but which impact how flow networks are specified and how they are run.
The flow engine runs on the server side, and let's just say for now that the client side (currently implemented in browser JavaScript) has an interesting way of interacting with the server.

I am also attempting to help Paul in furthering the cause of classic FBP (when I can afford the time - lately in very short supply - but working to change that)
I happen to have some exposure to product management and I believe that promoting the use of classic FBP
by improving its "out-of-box experience", can only help to improve the acceptance and popularity of the FBP community and FBP-like techniques in general.

As such, I am also interested in interoperability between flow frameworks (unfortunately, this is where things can get messy).

On Friday, 4 November 2016 15:10:29 UTC-4, David Ryman wrote:

Last update: I added bidirectional links between runtimes and generalized a little.

The network in runtime land warrants its own study.

Another penny dropped. In my normal use scenario, all the runtimes I will use are local, so are all the runnable components.

Pass the component info back up the chain !

With that mechanism, the onus would be on the C# runtime developer to interact under FBProtocol with any other runtime that already interact with any of the FBEngines. Bang, the GUI accesses a library of single component flows/graphs. The Circle of life.

From a diagramming perspective, (based on your introductory post in a previous topic) your example of a NodeRed flow network appears to share an element of Chicle that classic FBP does not.
Specifically, the notion that one output port can service multiple listeners

Depending on the underlying cause, this type of thing can have an impact on whether two networks can be easily connected or not.

There are multiple ways to interoperate between (let's call them "flow engine regimes"), starting with data streams and working up to flow network definition and registration.
There are also issues of data definition and of security to be addressed.

Chicle comes with its own network graphical designer/editor and its own interoperation protocol but there are multiple design issues to overcome before it could be used to connect to FBP.
 

I also have direct links between FBEngines and FBRuntimes, which would provide potential performance improvements.

I am not sure what you mean by "direct links" and how that impacts performance.  Can you elaborate?
 

Having a FBPremium package could provide a financial leg to FBP, whilst retaining full operability for everyone. For personal use, I do not need it to be fast, just to work. For business, I want micro trading potential.

I bet Bert already said this, as I have not read it yet, I want to give it justice, and I had to get this out of my head.

If I ever come up with an original idea, let me know, I'd like to claim it.

 
I believe that a healthy product needs a healthy community.

For now (and for the foreseeable future), dollars seem to be able to act as the lubricant for attention to accumulate and for progress to occur.

Modules are one way to participate - premium modules bring $$ and 3rd party modules bring attention
Interoperability is the other way to participate.

​PS the more I think about this, the more I think we are creating The Matrix. You really should watch them, or The Rise of The Machines, I am not sure which.

Yay, Matrix.  When do I get to meet the Lady in Red? ;-)
 

Best Regards,

--Bert

[Humberto Madeira]

Hi David,

On Saturday, 5 November 2016 14:23:39 UTC-4, David Ryman wrote:

The idea is to give a new user a way to get a successful taste for FBP with minimal time investment 
and then allow them to progress into more depth (such as coding their own components) by investing further as needed/desired. 

That's me.

 
That is excellent - we definitely need this sort of thing to happen. 
If you notice something that could be done better, please let us know.

So my thesis (based on some prior experience with early compression algorithms) is that generally, 
the size of the compressed data stream is related to the complexity of its uncompressed data and not its unpacked size.  
RLE (run length encoding) is generally a component of most compression formats, 
and will by itself  go a long way to achieve this effect even before applying  further compression techniques.

Made me think of encryption. At some point an IP may have to travel distance and require encryption. I would think encryption would add a layer of complexity so which is better encrypt-compress or compress-encrypt? or is this not a thing?

I tend to think of compression and encryption as platform issues.  Unfortunately, FBP was not designed with platform in mind.
In FBP, you would need to make compression and encryption as separate components, and you would also need components for communicating with the web..

However, my own platform (Chicle) doesn't do it that way.

Instead, Chicle has GZip compression built-in to the platform's messaging sub-system and triggered according to configured policy.

As for encryption, I currently rely on a reverse proxy server to provide SSL for any messages sent or received.
In that case, compression comes first in the platform, followed by TLS encryption on the reverse proxy.
(and yes, my platform could also support SSL directly, but it is better for now to do the reverse proxy thing)

In any case, I would also have to admit that messaging via SSL does not fully address end-to-end (pass-through) encryption.
So yes, additional encryption is still a thing much to be considered.

In light of that, I would also like to note that there is some effort on the browser side already going in that direction,
so I would like to leverage those design efforts before settling on my own design.

In the meantime, SSL/TLS should be good enough for direct node-to-node links - unless you are feeling extra-paranoid =:-O

Best Regards,
--Bert

[David Ryman]

Interspersed responses:

On Saturday, November 5, 2016 at 1:47:16 PM UTC-5, Humberto Madeira wrote:

Hi David,

I must admit I have been too busy lately to take a good look at NodeRed (I am assuming that is your main focus?)

My path led to Node Red. My main interest is AI, not programmed AI, but real AI. The Grid piqued my interest, having played with many website frameworks, which led me to NoFlo, which led to FBP.  I installed software and read half of Paul's book, made little progress, but am hooked on the idea of FBP. My focus is on seeing FBP develop into a revolution. As Node Red is in the FBP family and quick to get started with, I use it for flow diagrams at the same time as learning the product. The knowledge required to start being productive is very small, and that is what FBP should aim for.

 For that I highly recommend a look. 

  

Some personal background: I am working on my own flow engine platform (not out yet and not open source).
My current working code name for this platform is "Chicle"

Chicle does share some concepts with classic FBP but is dissimilar enough that I have trouble putting it into a specific category.
It is different both in the type of flow it supports and in other requirements not related to flow, but which impact how flow networks are specified and how they are run.

I would argue that a flow is a flow. A stream can be solid, liquid, gas, plasma, or digital, the same principals can apply. 

The flow engine runs on the server side, and let's just say for now that the client side (currently implemented in browser JavaScript) has an interesting way of interacting with the server.

I am also attempting to help Paul in furthering the cause of classic FBP (when I can afford the time - lately in very short supply - but working to change that)
I happen to have some exposure to product management and I believe that promoting the use of classic FBP
by improving its "out-of-box experience", can only help to improve the acceptance and popularity of the FBP community and FBP-like techniques in general.

totally. see Node Red about Out-of-box, I think the available components experience could be better, but not a bad starting model.   

 

As such, I am also interested in interoperability between flow frameworks (unfortunately, this is where things can get messy).

If Chicle conformed to the FBProtocol then it would fit naturally, (ish) 

On Friday, 4 November 2016 15:10:29 UTC-4, David Ryman wrote:

Last update: I added bidirectional links between runtimes and generalized a little.

The network in runtime land warrants its own study.

Another penny dropped. In my normal use scenario, all the runtimes I will use are local, so are all the runnable components.

Pass the component info back up the chain !

With that mechanism, the onus would be on the C# runtime developer to interact under FBProtocol with any other runtime that already interact with any of the FBEngines. Bang, the GUI accesses a library of single component flows/graphs. The Circle of life.

From a diagramming perspective, (based on your introductory post in a previous topic) your example of a NodeRed flow network appears to share an element of Chicle that classic FBP does not.
Specifically, the notion that one output port can service multiple listeners

Interesting. An FBP clone component could emulate the multiple outputs. 

 

Depending on the underlying cause, this type of thing can have an impact on whether two networks can be easily connected or not.

There are multiple ways to interoperate between (let's call them "flow engine regimes"), starting with data streams and working up to flow network definition and registration.
There are also issues of data definition and of security to be addressed.

I think all IPs should be encrypted as soon as possible, hence my query about compress/encrypt. Great feature to sell to the business world.

 

Chicle comes with its own network graphical designer/editor and its own interoperation protocol but there are multiple design issues to overcome before it could be used to connect to FBP.
 

I also have direct links between FBEngines and FBRuntimes, which would provide potential performance improvements.

I am not sure what you mean by "direct links" and how that impacts performance.  Can you elaborate?
 

FBP is agnostic. 

A flow/graph can contain components written in different languages.

A FBEngine has to manage the IPs through multiple FBRuntimes, either, as already suggested by Paul et al, using sockets as FBRuntime to FBRuntime messaging. With Direct-Links, as far as I know unimplemented, we have the FBEngine call the appropriate FBRuntime thereby removing use of sockets. Assuming this produces a performance gain, which would be the FBPremium package, again hooks into business needs. 

Having a FBPremium package could provide a financial leg to FBP, whilst retaining full operability for everyone. For personal use, I do not need it to be fast, just to work. For business, I want micro trading potential.

I bet Bert already said this, as I have not read it yet, I want to give it justice, and I had to get this out of my head.

If I ever come up with an original idea, let me know, I'd like to claim it.

 
I believe that a healthy product needs a healthy community.

For now (and for the foreseeable future), dollars seem to be able to act as the lubricant for attention to accumulate and for progress to occur.

I can see FBP changing that.

[David Ryman]

interspersed comments:

On Saturday, November 5, 2016 at 2:30:04 PM UTC-5, Humberto Madeira wrote:

Hi David,

On Saturday, 5 November 2016 14:23:39 UTC-4, David Ryman wrote:

The idea is to give a new user a way to get a successful taste for FBP with minimal time investment 
and then allow them to progress into more depth (such as coding their own components) by investing further as needed/desired. 

That's me.

 
That is excellent - we definitely need this sort of thing to happen. 
If you notice something that could be done better, please let us know.

Count on it :) 

So my thesis (based on some prior experience with early compression algorithms) is that generally, 
the size of the compressed data stream is related to the complexity of its uncompressed data and not its unpacked size.  
RLE (run length encoding) is generally a component of most compression formats, 
and will by itself  go a long way to achieve this effect even before applying  further compression techniques.

Made me think of encryption. At some point an IP may have to travel distance and require encryption. I would think encryption would add a layer of complexity so which is better encrypt-compress or compress-encrypt? or is this not a thing?

I tend to think of compression and encryption as platform issues.  Unfortunately, FBP was not designed with platform in mind.
In FBP, you would need to make compression and encryption as separate components, and you would also need components for communicating with the web..

I say Fortunately FBP was not designed with platform in mind.

Great idea. Compressed IPs within the network is an internal performance issue. Zen FBPhilosophy would say encryption/decryption components is the way to go.

However, my own platform (Chicle) doesn't do it that way.

Instead, Chicle has GZip compression built-in to the platform's messaging sub-system and triggered according to configured policy.

As for encryption, I currently rely on a reverse proxy server to provide SSL for any messages sent or received.
In that case, compression comes first in the platform, followed by TLS encryption on the reverse proxy.
(and yes, my platform could also support SSL directly, but it is better for now to do the reverse proxy thing)

In any case, I would also have to admit that messaging via SSL does not fully address end-to-end (pass-through) encryption.
So yes, additional encryption is still a thing much to be considered.

In light of that, I would also like to note that there is some effort on the browser side already going in that direction,
so I would like to leverage those design efforts before settling on my own design.

In the meantime, SSL/TLS should be good enough for direct node-to-node links - unless you are feeling extra-paranoid =:-O

I'm not paranoid, all my fears are real. :) 

Best Regards,
--Bert

[Humberto Madeira]

Hi David,

On Monday, 7 November 2016 15:20:01 UTC-5, David Ryman (ANeo) wrote:

I would argue that a flow is a flow. A stream can be solid, liquid, gas, plasma, or digital, the same principals can apply. 

I'm going to suggest that Chicle and FBP are very different beasts that came from very different backgrounds.

And while all streams are flows, not all flows are streams.

If Chicle conformed to the FBProtocol then it would fit naturally, (ish) 

That isn't really possible.  Neither the FBProtocol or the FBP Language (as they are currently specified) are expressive enough to be able to describe a Chicle flow network with all its features.

To be honest, at present I'm not even sure if the reverse (Chicle reading an FBP specification and emulating an FBP network) would ever become possible (although I do have some ideas under investigation).
 

From a diagramming perspective, (based on your introductory post in a previous topic) your example of a NodeRed flow network appears to share an element of Chicle that classic FBP does not.
Specifically, the notion that one output port can service multiple listeners

Interesting. An FBP clone component could emulate the multiple outputs.

Or just provide an array output port with cloned data to each element of the array.
Or  just have the first listener component have a pass-through port that you can use to chain an additional component.
(we've had discussions on this previously)

Personally however, I don't find any of these solutions very elegant.

 

 

Depending on the underlying cause, this type of thing can have an impact on whether two networks can be easily connected or not.

There are multiple ways to interoperate between (let's call them "flow engine regimes"), starting with data streams and working up to flow network definition and registration.
There are also issues of data definition and of security to be addressed.

I think all IPs should be encrypted as soon as possible, hence my query about compress/encrypt. Great feature to sell to the business world.

If the IP is only traveling to another component in the same engine, there is no need to encrypt. (and performance reasons not to)
If you are using a web socket to deliver an IP, you just need to use SSL and you get encryption for free.

 

FBP is agnostic. 

Um, what do you mean by that?  Agnostic to what?  Programming Languages?
If so, not really.  There are some languages and some environments that are missing the required elements to properly implement FBP.

Then there is the issue of getting agreement on the data types transferred between them.

A FBEngine has to manage the IPs through multiple FBRuntimes, either, as already suggested by Paul et al, using sockets as FBRuntime to FBRuntime messaging. With Direct-Links, as far as I know unimplemented, we have the FBEngine call the appropriate FBRuntime thereby removing use of sockets.

I am not sure what you mean by this.  What do you mean by "call the appropriate FBRuntime"?  How would such a call be made?
 

Assuming this produces a performance gain, which would be the FBPremium package, again hooks into business needs. 

 
Performance isn't the only criteria for a premium package or for business needs.

Generally, you only tend to notice performance when it is not good enough.
And once performance is good enough, you tend not to notice any further improvements.

For now (and for the foreseeable future), dollars seem to be able to act as the lubricant for attention to accumulate and for progress to occur.

I can see FBP changing that.

Ok.  We will agree to disagree on that one. :-)

Best regards,
--Bert

[Paul Morrison]

Interesting conversation!

Just wanted to say that people keep talking rather casually about "cloning" IPs, but FBP also has the concept of IP "trees" - see http://www.jpaulmorrison.com/fbp/tree.shtml - where a tree of connected IPs can travel through the network as a single IP.  See attachment.


Theoretically it is possible to wire up the tree so it is self-describing, but my current FBP implementations don't do this.  Given that IP trees can be quite large and complex, my feeling was that building a cloning function into the infrastructure wasn't worth the trouble - why would you need to make an exact copy of a (potentially quite large) IP tree? 

A stronger argument could be made that serialization and deserialization would be easier if the tree were self-describing, so we might consider modifying the current "chain" API calls, or adding new ones to achieve this.  Feedback would be appreciated.

[Paul Morrison]

Hi David and Bert,

Zeroing in on one paragraph:

"A FBEngine has to manage the IPs through multiple FBRuntimes, either, as already suggested by Paul et al, using sockets as FBRuntime to FBRuntime messaging. With Direct-Links, as far as I know unimplemented, we have the FBEngine call the appropriate FBRuntime thereby removing use of sockets. Assuming this produces a performance gain, which would be the FBPremium package, again hooks into business needs."

What are Direct-Links?  When you said "call the appropriate FBRuntime", you just have to get the run times started - surely a bat file would be enough!  :-)

I learn best by doing!  Do you have a prototype you can show us?   I have had a JavaFBP (partial) network talk to a C#FBP (partial) network using sockets - worked like a charm!  The problem that this solves is that of multiple concurrent VMs - I do understand that VMs help the compiler builder, but IMO they don't help developers much, and in fact actually get in the way. 

If you don't use sockets, I guess you could use some kind of native code to jump back and forth between the Java VM and the C# CLR - sockets seem cleaner!

Just as FBP is agnostic, one language should not assume it controls the whole world.  FBP makes that unnecessary.  In the old days of PL/I (remember that?) you could start up concurrent multiple PL/I environments, but then they converted to Language Environment 370 (LE/370) and it became impossible (programmer shortsightedness, not conceptual problems - they actually provided that for CICS, but not for batch).

I'd like to stress this: you don't have to do everything with one language!  There are features in some languages which you young whipper-snappers have never heard of, and are probably mutually incompatible!  :-)  Besides it is not difficult to learn multiple programming languages - I'm sure most of you speak at least 3 or 4: FORTRAN, COBOL, Assembler, REXX, JCL... just joking!

This is why in the past I have talked about building on a lower infrastructure, and is partly why I developed the C++ implementation (CppFBP) using Boost for multithreading.  You could then build components in any language which talks to C++.  If you particularly like scripting languages (I prefer Java personally), you can look at my C++/Lua interface - see https://github.com/jpaulm/cppfbp - do a find on TryLua.

One other idea was to use Parrot - https://en.wikipedia.org/wiki/Parrot_virtual_machine - might be interesting to play with that.

What do you have in mind for FBPremium?

Regards,

Paul M.

[Humberto Madeira]

Hi Paul,

On Tuesday, 8 November 2016 09:21:33 UTC-5, Paul Morrison wrote:

Interesting conversation!

Just wanted to say that people keep talking rather casually about "cloning" IPs, but FBP also has the concept of IP "trees" - see http://www.jpaulmorrison.com/fbp/tree.shtml - where a tree of connected IPs can travel through the network as a single IP.  See attachment.

Chicle has a similar concept.  You can encapsulate a data item into an attribute of another data item.

Theoretically it is possible to wire up the tree so it is self-describing, but my current FBP implementations don't do this.  Given that IP trees can be quite large and complex, my feeling was that building a cloning function into the infrastructure wasn't worth the trouble - why would you need to make an exact copy of a (potentially quite large) IP tree? 

Since all transferable Chicle data items are deeply immutable, no cloning function is necessary.  You can just pass additional copies of the reference without fear of concurrency issues.

Actually, that's one of the main reasons Chicle limits the the flows to immutable data items (but not the only reason).

 

A stronger argument could be made that serialization and deserialization would be easier if the tree were self-describing, so we might consider modifying the current "chain" API calls, or adding new ones to achieve this.  Feedback would be appreciated.

Chicle data items are self describing, precisely to make serialization/deserialization and conversions between type systems easier.

 

regards,

--Bert

 

[Humberto Madeira]

Hi Paul,

On Tuesday, 8 November 2016 11:34:44 UTC-5, Paul Morrison wrote:

What are Direct-Links?  When you said "call the appropriate FBRuntime", you just have to get the run times started - surely a bat file would be enough!  :-)

I didn't understand this one either.

 

The problem that this solves is that of multiple concurrent VMs - I do understand that VMs help the compiler builder, but IMO they don't help developers much, and in fact actually get in the way. 

Ok.  Now I don't understand what you mean by this one.

-- Problem of multiple concurrent VMs...  

What problems?  

--VMs help the compiler builder but get in the way of the developer...  

How does it get in the way?

 

If you don't use sockets, I guess you could use some kind of native code to jump back and forth between the Java VM and the C# CLR - sockets seem cleaner!

There are a few ways other than sockets to pass data between processes.

Shared memory, The file system, A command line.  Serial ports.  USB ports.  HDMI ports.  Thunderbolt ports. (you get the idea)

None of these are particularly efficient as compared to sockets. 

Some bytecode based environments (such as the JVM and the CLR) can tolerate multiple languages calling each other directly within their process.

But I would consider that they are all really running in the same language - the bytecode.

You can also link between languages using DLL calls, but some inter-language interfaces are worse than others and can cause various problems with garbage collection.

To call Java from within the same process as a CLR, you would need a C/C++ bridging DLL of some sort.

 

Just as FBP is agnostic, one language should not assume it controls the whole world.  FBP makes that unnecessary. 

Actually, just plain sockets make it unnecessary.  Most Webapps already consist of multiple languages.

 

In the old days of PL/I (remember that?) 

Just barely...  mostly successful in trying to forget :-)

 

I'd like to stress this: you don't have to do everything with one language!  There are features in some languages which you young whipper-snappers have never heard of, and are probably mutually incompatible!  :-)  Besides it is not difficult to learn multiple programming languages - I'm sure most of you speak at least 3 or 4: FORTRAN, COBOL, Assembler, REXX, JCL... just joking!

I will never admit to knowing COBOL or Visual Basic :-)

Never had to deal with JCL (yay!).  Just barely touched REXX.  

I grew up on FORTRAN and  Assembler.  

 

This is why in the past I have talked about building on a lower infrastructure, and is partly why I developed the C++ implementation (CppFBP) using Boost for multithreading.  You could then build components in any language which talks to C++.  If you particularly like scripting languages (I prefer Java personally), you can look at my C++/Lua interface - see https://github.com/jpaulm/cppfbp - do a find on TryLua.

I still prefer sockets to inter-language calls.  Granted that every application talks to the underlying OS in this way. 

But that tends to be hidden from you by your programming language or VM runtime.

 

One other idea was to use Parrot - https://en.wikipedia.org/wiki/Parrot_virtual_machine - might be interesting to play with that.

Oh yes. There are several cross-compilers available for that.  I've been waiting for them to mature a bit.

 

What do you have in mind for FBPremium?

 

I think that's primarily a David question, but I should think a premium package should be more oriented towards business use.

For example -  the ability to read, manipulate and send email.

Best Regards,

--Bert

[David Ryman (ANeo)]

It states on my resume that I have "good communications skills", that's a lie. :)

comments interspersed.

ANeo

On Tuesday, November 8, 2016 at 10:34:44 AM UTC-6, Paul Morrison wrote:

Hi David and Bert,

Zeroing in on one paragraph:

"A FBEngine has to manage the IPs through multiple FBRuntimes, either, as already suggested by Paul et al, using sockets as FBRuntime to FBRuntime messaging. With Direct-Links, as far as I know unimplemented, we have the FBEngine call the appropriate FBRuntime thereby removing use of sockets. Assuming this produces a performance gain, which would be the FBPremium package, again hooks into business needs."

Direct Links clarification.

Example. network consists of A and B, coded in different languages. At the FBEngine you get a choice. 1. Run A and B through FBRuntime A, using sockets to harvest the results of B run on FBRuntime B. 2. Run A through FBRuntime A, harvest the results and run B through FBRuntime B.

I am suggesting that both can exist. Option 2 (Direct links) is an extra. 

What are Direct-Links?  When you said "call the appropriate FBRuntime", you just have to get the run times started - surely a bat file would be enough!  :-)

In my vision of FBP, both individuals and corporation would be running essentially the same systems. Individuals probably do not want 50 runtimes active. Corporations probably do want them all active 24/7. IF (big if) there is a performance gain by adding option 2 then I feel it would be a selling point to Industry. The idea of FBPremium is to have a financial wing to the whole thing. There is an overhead in starting/stopping a runtime, which is fine for me, but can add up in a production environment where everything is done by FBP.

 

I learn best by doing!  Do you have a prototype you can show us?   I have had a JavaFBP (partial) network talk to a C#FBP (partial) network using sockets - worked like a charm!  The problem that this solves is that of multiple concurrent VMs - I do understand that VMs help the compiler builder, but IMO they don't help developers much, and in fact actually get in the way.

I wish.

What happens when you add more components in more languages?

Ah, sockets connect FBEngines not FBRuntimes?

 

 

If you don't use sockets, I guess you could use some kind of native code to jump back and forth between the Java VM and the C# CLR - sockets seem cleaner!

Sockets are great, we used them to get horse race results directly from the track into the Sporting Life Database in the 90's.

 

Just as FBP is agnostic, one language should not assume it controls the whole world.  FBP makes that unnecessary.  In the old days of PL/I (remember that?) you could start up concurrent multiple PL/I environments, but then they converted to Language Environment 370 (LE/370) and it became impossible (programmer shortsightedness, not conceptual problems - they actually provided that for CICS, but not for batch).

The one 'thing' that controls the whole world is FBNetwork. One ought to be able to create a FBNetwork using a text editor, run it through any FBEngine and obtain results.

 

I'd like to stress this: you don't have to do everything with one language!  There are features in some languages which you young whipper-snappers have never heard of, and are probably mutually incompatible!  :-)  Besides it is not difficult to learn multiple programming languages - I'm sure most of you speak at least 3 or 4: FORTRAN, COBOL, Assembler, REXX, JCL... just joking!

Exactly, but you can do everything with FBP!

One should be able to pipe IPs into any piece of executable code, harvest the output, and feed it into any other piece of executable code until you are done.

The code is a black box that does what it says on the box.

Information packets are blobs of whatever the architect/user/designer/?? needs to manipulate.

Totally agnostic.

Back in the day when I was a young whipper-snapper, I wrote some RPG II to analyse RPG II programs to create 'horses mouth' documentation (remember RemDoc?, I developed our own), 'cause no one liked doing documentation. Using similar techniques previously, I had an analysis of an entire package to identify dates for an anglicization of a US system. Then RPG II to update every program. 2 days at each client instead of weeks. FBP could do that.

My point being that the IPs, to me, are also black.

This is why in the past I have talked about building on a lower infrastructure, and is partly why I developed the C++ implementation (CppFBP) using Boost for multithreading.  You could then build components in any language which talks to C++.  If you particularly like scripting languages (I prefer Java personally), you can look at my C++/Lua interface - see https://github.com/jpaulm/cppfbp - do a find on TryLua.

One other idea was to use Parrot - https://en.wikipedia.org/wiki/Parrot_virtual_machine - might be interesting to play with that.

What do you have in mind for FBPremium?

Money. I am pro open source, I hate sales and marketing 'tricks'. The uptake of Linux is hampered by being 'free', no one is selling it. so no salespeople, so business ignores it. There is no such thing as a free lunch mentality. FBPremium is an enhanced package specifically for business, without removing any functionality from the 'home' edition. As I think I had it wrong as to where the sockets lived, FBPremium is now an empty set.

You mentioned the C++/Lua set before, for some reason C++ scares me, but I'll force myself :)

[David Ryman (ANeo)]

On Tuesday, November 8, 2016 at 1:26:05 AM UTC-6, Humberto Madeira wrote:

Hi David,

On Monday, 7 November 2016 15:20:01 UTC-5, David Ryman (ANeo) wrote:

I would argue that a flow is a flow. A stream can be solid, liquid, gas, plasma, or digital, the same principals can apply. 

I'm going to suggest that Chicle and FBP are very different beasts that came from very different backgrounds.

And while all streams are flows, not all flows are streams.

:) I was thinking Eastern philosophy. please replace streams with flows. 

If Chicle conformed to the FBProtocol then it would fit naturally, (ish) 

That isn't really possible.  Neither the FBProtocol or the FBP Language (as they are currently specified) are expressive enough to be able to describe a Chicle flow network with all its features.

Can the FBProtocol be modified to include Chicle?

To be honest, at present I'm not even sure if the reverse (Chicle reading an FBP specification and emulating an FBP network) would ever become possible (although I do have some ideas under investigation).
 

From a diagramming perspective, (based on your introductory post in a previous topic) your example of a NodeRed flow network appears to share an element of Chicle that classic FBP does not.
Specifically, the notion that one output port can service multiple listeners

Interesting. An FBP clone component could emulate the multiple outputs.

Or just provide an array output port with cloned data to each element of the array.
Or  just have the first listener component have a pass-through port that you can use to chain an additional component.
(we've had discussions on this previously)

Personally however, I don't find any of these solutions very elegant.

There is always a way.

 

 

 

Depending on the underlying cause, this type of thing can have an impact on whether two networks can be easily connected or not.

There are multiple ways to interoperate between (let's call them "flow engine regimes"), starting with data streams and working up to flow network definition and registration.
There are also issues of data definition and of security to be addressed.

I think all IPs should be encrypted as soon as possible, hence my query about compress/encrypt. Great feature to sell to the business world.

If the IP is only traveling to another component in the same engine, there is no need to encrypt. (and performance reasons not to)
If you are using a web socket to deliver an IP, you just need to use SSL and you get encryption for free.

Love simplicity, encryption/decryption components works for me.  

 

FBP is agnostic. 

Um, what do you mean by that?  Agnostic to what?  Programming Languages?
If so, not really.  There are some languages and some environments that are missing the required elements to properly implement FBP.

Agnostic as far as language used to transform the IP, what I called FBRuntimes (probably what Paul calls VMs). The FBEngine that processes the FBNetwork would and should the written in the best language for the task. FBPhilosphy speaks to logic and black boxes

 

Then there is the issue of getting agreement on the data types transferred between them.

A FBEngine has to manage the IPs through multiple FBRuntimes, either, as already suggested by Paul et al, using sockets as FBRuntime to FBRuntime messaging. With Direct-Links, as far as I know unimplemented, we have the FBEngine call the appropriate FBRuntime thereby removing use of sockets.

 

I am not sure what you mean by this.  What do you mean by "call the appropriate FBRuntime"?  How would such a call be made?

 

FBEngine processes the FBNetwork, The component runs in a runtime, I think. I may have this mixed up. 

What actually runs the component? the FBEngine or FBRuntime or are they one and the same?

Flowhub can call several runtimes all JS I guess. 

 

Assuming this produces a performance gain, which would be the FBPremium package, again hooks into business needs. 

 
Performance isn't the only criteria for a premium package or for business needs.

Possibly. Not sure what else can be included without taking functionality away from the 'Personal' edition. 

Generally, you only tend to notice performance when it is not good enough.
And once performance is good enough, you tend not to notice any further improvements.

Not all decision makers know what the inside of a server room looks like, but mention speed and security and you have a foot in the door. 

For now (and for the foreseeable future), dollars seem to be able to act as the lubricant for attention to accumulate and for progress to occur.

I can see FBP changing that.

Ok.  We will agree to disagree on that one. :-)

I, perhaps, have more faith in the open community. Push then. 

Best regards,
--Bert

[Humberto Madeira]

Hi David,

On Tuesday, 8 November 2016 19:37:04 UTC-5, David Ryman (ANeo) wrote:

If Chicle conformed to the FBProtocol then it would fit naturally, (ish) 

That isn't really possible.  Neither the FBProtocol or the FBP Language (as they are currently specified) are expressive enough to be able to describe a Chicle flow network with all its features.

Can the FBProtocol be modified to include Chicle?

Unlikely.  Adding Chicle features to the FBProtocol would most likely end up causing resistance to change in FBProtocol users before very long.

FBP doesn't model anything beyond flow network itself. This minimizes the complexity of the protocol.

Chicle models involve a lot more than that.

Interesting. An FBP clone component could emulate the multiple outputs.

Or just provide an array output port with cloned data to each element of the array.
Or  just have the first listener component have a pass-through port that you can use to chain an additional component.
(we've had discussions on this previously)

Personally however, I don't find any of these solutions very elegant.

There is always a way.

 

Didn't say otherwise.  Chicle's approach doesn't require any such components. All Chicle output ports directly accept multiple connections.

But Chicle also requires that all transferable data items be immutable.  As such, no cloning is required, just send a reference of the same item as many times as you want.

I have spoken to Paul about this issue and he has indicated an unwillingness to make FBP require IP immutability (because of backward compatibility).

Since I am not yet saddled with the burden of backward compatibility, I have taken the alternate approach.

Surprisingly, this apparently small design difference actually has major downstream repercussions in how things work.

And it is hardly the only difference between Chicle and FBP.

 

FBP is agnostic. 

Um, what do you mean by that?  Agnostic to what?  Programming Languages?
If so, not really.  There are some languages and some environments that are missing the required elements to properly implement FBP.

Agnostic as far as language used to transform the IP, what I called FBRuntimes (probably what Paul calls VMs). The FBEngine that processes the FBNetwork would and should the written in the best language for the task. FBPhilosphy speaks to logic and black boxes

 

Mmm.  Not all languages have support for multi-threading.  And that is a minimum requirement for classic FBP.  Chicle further requires that a language have support for immutability.

This does not strike me as fully language-agnostic.

 

A FBEngine has to manage the IPs through multiple FBRuntimes, either, as already suggested by Paul et al, using sockets as FBRuntime to FBRuntime messaging. With Direct-Links, as far as I know unimplemented, we have the FBEngine call the appropriate FBRuntime thereby removing use of sockets.

 

I am not sure what you mean by this.  What do you mean by "call the appropriate FBRuntime"?  How would such a call be made?

 

FBEngine processes the FBNetwork, The component runs in a runtime, I think. I may have this mixed up. 

What actually runs the component? the FBEngine or FBRuntime or are they one and the same?

Flowhub can call several runtimes all JS I guess. 

But those are all called through sockets, AFAIK.  I don't see directness of any sort.

 

 

Assuming this produces a performance gain, which would be the FBPremium package, again hooks into business needs. 

 
Performance isn't the only criteria for a premium package or for business needs.

Possibly. Not sure what else can be included without taking functionality away from the 'Personal' edition. 

There are plenty of business systems that require specific interface adapters to work with.

Those adapters will also need to be maintained to remain current with those business systems.

 

Generally, you only tend to notice performance when it is not good enough.
And once performance is good enough, you tend not to notice any further improvements.

Not all decision makers know what the inside of a server room looks like, but mention speed and security and you have a foot in the door. 

The problem is that the performance gambit has been practiced so much to the exclusion of all else that it has made a mess of everything else.

In a lot of cases, any such "performance gains" merely allow sloppier and sloppier designs to prevail to the point that all of the performance gained has been lost (and even worse).

Once that happens, a different foot eventually gets placed in a different spot and the cycle continues.  

 

The security gambit, as it is currently practiced, IMHO doesn't seem to be working very well.  The number and volume of data breaches keeps rising all the time.

In both cases, the perception of what is considered "better" has been twisted by marketing effort to the point of resulting in negative outcomes.

Personally I would rather prefer to stop playing these types of marketing games and produce something with real benefits instead.

Who knows, people might even come to appreciate that sort of thing someday... :-)

Best Regards,

--Bert