GoFBP

[Paul Morrison]

Hi FBPers!  Wanted to start a conversation with this title - I could have put it under an existing conversation, but I wanted it to have its own heading, as I feel it could potentially be important for the future...!

I have started a "classical" FBP implementation using the Go language, and I feel it is just about ready for people to give it a test run!  The externals seem to be fairly firm... with a lot of help from Egon Elbre and Emil Valeev...  thanks, guys!  It can be found on https://github.com/jpaulm/gofbp , and follows the architecture of the other "classical" FBP implementations on https://github.com/jpaulm pretty closely.  For the distinction between "classical" FBP and "FBP-like", see https://jpaulm.github.io/fbp/fbp-inspired-vs-real-fbp.html .

Part of my motivation for this is that I am interested in migrating to WebAssembly at some point, but for now I am working in Go...  I am not enthused about the Go language, as I feel that some of the things I had to do were somewhat unnatural :-)  , but I think we are mostly there, and I would like to see some of you Go-enthusiasts give it a try!

GoFBP follows the other FBP implementations on https://github.com/jpaulm in its architecture, but one difference you will find is in its treatment of FBP deadlocks: JavaFBP and C#FBP detect deadlocks and generate useful information about them.  Go, on the other hand, detects deadlocks itself before GoFBP can generate its own debugging info, and instead generates a series of stack traces.  I have therefore written a deadlock analyzer program, which analyzes the stack traces, and prints out diagnostic info similar to what JavaFBP generates.  Maybe one of you can come up with a smarter solution, but I will run with this for now!

Good luck trying out GoFBP - I look forward to your feedback!

Paul M.

[Paul Morrison]

By "externals", I mean network and subnet definitions, and the component API - see https://jpaulm.github.io/fbp/gosyntax.htm (under construction)...

[Paul Morrison]

I gather that Go supports Lua - C++FBP also supports Lua - see https://jpaulm.github.io/fbp/thlua.html -  so the interfaces might be transferable!  This looks like a pretty interesting combination!  I'll add it to my list of todos!

[Mathieu Ledru]

Thanks for sharing. I will be glad to see FBP on WebAssembly too ! For now I mainly use Javascript. But with the use of WebAssembly, I go a lot of idea to use FBP by using a runtime and there is lot here : https://github.com/appcypher/awesome-wasm-runtimes

I think about using https://wasmer.io/ which has a lot of porting for almost every object oriented langages.
So may be I should have more interest to contribute when talking about webassembly.

[Niclas Hedhman]

Paul,

I am a bit confused, and perhaps I am greatly mistaken somewhere; Wasn't each node supposed to be isolated from all other nodes to be considered "true FBP"? If so, then how can you end up with deadlock issues?

Cheers

Niclas

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flow-based-programming/ea60189c-f85e-437f-a93b-aac959fe616an%40googlegroups.com.

[Paul Morrison]

Hi Niclas, No, you are not confused!  In FBP, we have always found deadlock to be a design issue - if you look at https://jpaulmorrison.com/fbp/deadlock.shtml , do a find on diagrams 16.5 and 16.8 - the latter results in a deadlock condition, and in fact, https://github.com/jpaulm/gofbp/blob/master/force_deadlock_test.go implements this diagram, and is designed to crash due to this condition.  You can of course still get deadlocks due to database access conflicts, but these have to be handled just as you would expect in any database application...  Hope this helps!

To view this discussion on the web visit https://groups.google.com/d/msgid/flow-based-programming/CADmm%2BKczQTTYwoso9wk5uPDQgr2kfB1X_3cP3G9rhLib%2BFfTyA%40mail.gmail.com.

[Niclas Hedhman]

Alright, I think I get what I have misunderstood all along;

I have assumed that IPs were sent to inputs fully asynchronously (with an active/ordered read on the receiver), and the downstream component was "triggered" into action when the inputs were satisfied. (Perhaps polluted by this being the case in a tool from Sun Microsystems back in 1998/99 or so, which did exactly that and didn't fit my purposes at all (I work with discrete numerical values only and order is irrelevant)).

Cheers

Niclas

To view this discussion on the web visit https://groups.google.com/d/msgid/flow-based-programming/CAHttgx4G2FaeCgfZGKY6W1jWF_tPU4eYOeeS8zcb%3DP6DEug9mA%40mail.gmail.com.

[Ged Byrne]

Hi Nicolas,

The difference is that in classical FBP the downstream component  is not "triggered" into action.  That is how event driven implementations like NoFlo work.

Components share IPs over a bounded buffer.  That is a FIFO queue that sits between the two components.

When upstream component sends an IP to downstream component it adds it's component to the back of the queue.  The queue can only take a fixed number of entries so it could be full.  In the event that the queue is full the upstream component will block waiting for a space to become available.  This creates an effect called back pressure: https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data-through-software-2350b3e77ce7

As the reactive manifesto explains: " back-pressure is an important feedback mechanism that allows systems to gracefully respond to load rather than collapse under it.": https://www.reactivemanifesto.org/glossary#Back-Pressure  

As far as I'm aware event driven FBP does not have back pressure.

The downstream component receives an IP from an upstream component by reading from it's queue.  Every time it does this a space is made available in the queue so that any component blocked by back pressure will then be able to proceed.

A deadlock occurs then the flow graph contains a loop.  This means that one component is both upstream and downstream from another.  If that component is blocked by back pressure then it will never clear because it is blocking itself.  Rather like grid lock.

https://en.wikipedia.org/wiki/Gridlock

Unlike city streets FBP's connecting buffers are not limited by physical space and this allows us to fix the deadlock problem by introducing self expanding buffers.  When a loop exists one of the connections can be made into a buffer that will grow so that back pressure never occurs.  This means that a deadlock will not happen.

This comes at a cost because you lose the important feedback mechanism provided by back pressure.  With back pressure you quickly find out that there is a problem because your graph locks up.  Without grid lock you discover the issue months after deployment when peak load causes your server to run out of memory.

Loops can't always be avoided, so the self expanding buffer is there when you need it.

Regards, 

Ged

To view this discussion on the web visit https://groups.google.com/d/msgid/flow-based-programming/CADmm%2BKc9ODy_O0QGvdxUnFEf-ybzLb%2BnSDhEv_TP-aO1Rxg7zA%40mail.gmail.com.

[Paul Morrison]

Hi Ged, you're correct when you say, " A deadlock occurs then the flow graph contains a loop. ", but I just wanted to point out that you can also have what I called a "divergent-convergent" topology - both this one and the loop topology should act as warning signs of potential problems.  The divergent-convergent topology is shown in Fig. 16.8 of  https://jpaulmorrison.com/fbp/deadlock.shtml ...  Actually, both Figs. 16.5 and 16.8 are divergent-convergent, but 16.8 is the one that deadlocks, because CONCAT is waiting on one port while most of the data is coming in on the other one! 

PS Glad to see that the Google Group is still active!

[Paul Morrison]

Re GoFBP: This has also been posted to the Discord group, but I suspect there are lots of people who are on this Google Group, but who may not have signed on to the Discord group yet, so I thought I should post here as well.  The Discord group can be joined at https://discord.gg/YBQj6UsD5H - thanks, Matt!

Just wanted to say that I think I got rid of a race condition from  https://github.com/jpaulm/gofbp ... and it was due to some diagnostic code that wasn't really necessary anyway!   GoFBP is now at v2.6.0 , and I am feeling pretty comfortable with the code. I just ran the test suite a hundred times, with no problems, and it seems pretty fast!

Please download the latest release of GoFBP, and give it a try! There are examples of networks and components in https://jpaulm.github.io/fbp/gosyntax.htm , which can be used as models - and there will be more complete documentation following soon! Let me know how you manage!

Good luck, and thanks in advance!

Paul M.

[Paul Morrison]

We have changed the Release numbers for GoFBP over to tags, and we are now at v1.0.3 .  GoFBP seems to be working pretty well (IMHO) - so feedback would be appreciated!

I would also like to announce (!) that DrawFBP can now generate runnable GoFBP network and subnet definitions. 

There is one quirk (as far as we know, right now):  if a subnet is in the same folder as the network that invokes it, the invoking statement in the calling network will probably have to be corrected!   I may figure out how to avoid this in a future release of DrawFBP... or you can just make sure they are in different folders!

Cheers!