Dealing with large graphs

[Stefan Sauer]

Hi,

I am lurking here for some time and playing with noflojs (mostly on my lego mindstorms ev3: https://github.com/ensonic/noflo-legoev3). When talking to people about FPB a common reaction I get is that they believe it is good for some rapid prototyping, but not sustainable. People are afraid that in the end for 'real' applications the graphs are too big to be useful as a means to look at and understand what the code would do when run.

Now there are means to structure the graph like:

- choosing the right granularity for the components (like instead of doing math/Add -> math/Multiply -> math/Floor, write a math/MapRange component)

- using sub-graphs with exports for some of the ports

Any other things people can add to this list. I'd love to hear your experience if you had similar discussions? How did you reply? Can anyone share their success story?

Stefan

[Paul Morrison]

Yes, granularity and subnetting are key here.  If you are truly thinking asynchronously, you will usually not split IPs into individual fields, as you know how much trouble it is to recombine them!  Imagine taking your car to be serviced and having them send every part to a different department - some fast and some slow!  So your granularity is coarser, and your networks tend to be smaller.

The biggest network we ever built, using "classical" FBP, was about 200 nodes, built up hierarchically.  Very easy to understand and very easy to maintain!

Hope this helps,

Paul

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alfredo Sistema]

I will add that another option is loading graphs dynamically from
within components. This is a specialization of sub graphs, but it can
be very handy to generate pipelines of subgraphs on the fly.
Comments are very helpful too, as I rarely see any when visual
programming examples are shown.

[Paul Morrison]

I agree, Alfredo, dynamically loading subnets helps to unclutter the graph - and you know you can defer looking at parts of the total network until you need to understand them.

By the way, DrawFPB has a useful "legend" block - basically a box without a border - Ged in particular has made constructive use of them.

Regards,

Paul

[Humberto Madeira]

Hi Stefan,

On Wednesday, 18 February 2015 11:53:25 UTC-5, Stefan Sauer wrote:

I am lurking here for some time and playing with noflojs (mostly on my lego mindstorms ev3: https://github.com/ensonic/noflo-legoev3). When talking to people about FPB a common reaction I get is that they believe it is good for some rapid prototyping, but not sustainable. People are afraid that in the end for 'real' applications the graphs are too big to be useful as a means to look at and understand what the code would do when run.

Perhaps one approach to reassure the people you are talking to might go something more or less along these lines: 

Actually, in regards to program size, "writing" a flow-based app is a lot like writing regular code.  

A 5000 line method is not maintainable/manageable/understandable/reasonable.  Why should this be any different for a 5000 component network?

A text editor's vertical height is limited - that limitation is a necessity born of the limitations of the display technology, but it also helps to guide the author in limiting the size of a method.

If the method doesn't fit into the available vertical space, then the irritation you feel when constantly scrolling it is a strong clue that you should probably break it up into smaller methods.

Similarly a diagram editor has a limited display area - if a network doesn't fit comfortably into it, then you should think about breaking it up into subnets.

As a side benefit, once the program is reorganized that way, reuse becomes possible, and then the rate at which you can create additional functionality improves.

And that is how "real applications" should be written, regardless of technology.

Hope this helps,

--Bert

[Kenneth Kan]

Alfredo, do you have some pointers to how dynamic loading would look like? I'm curious of what does that entail from the programmer's perspective. Thanks!

>> email to flow-based-programming+unsub...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Flow Based Programming" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to flow-based-programming+unsub...@googlegroups.com.

[Paul Tarvydas]

IMO, FBP is the only way to scale to large projects.

It is difficult to describe the power of FBP to people who haven't used it.

I told someone that it's SOA done right.

That seemed to make sense to them.

pt

[Stefan Sauer]

Thanks for all the replies! 

One of my arguments that people seem to understand also is that one does not necessarily have to use a graphical editor to write FBP code, but being able to use one gives the opportunity to see the graph and the eventual mess we developers sometimes create. If one writes a app using an event model in a OO style, one usually does not have that visibility.

I definitely need to explore dynamic subgraph loading. This is important for many cases. One case I am looking at is running the same graph on different platforms and being able to use a platform specific component. Now sure though if I am too much tied in classic OO thinking here, with n components implementing the same interface and plugin one of them as fits.

Stefan

[Ged Byrne]

Hi Stefan,

Have you read Paul's book yet?

Here is what he wrote in the first edition about dynamically loading modules:

'... a better way to develop these networks was to use the decomposition techniques of Structured Analysis, but that, unlike conventional programming, there was no change in viewpoint as you moved from design to implementation. In conventional program development there is a "gap" between the data flow approach used during design and the control flow viewpoint required during programming, which is extraordinarily difficult to get across. When building DFDM we therefore provided a way for developers to grow their systems by stepwise decomposition, but at structure build time the hierarchy got "flattened" into the conventional AMPS-type network. This approach turned out to be very successful - I alluded earlier to a colleague who built a 200-node network without once drawing a picture of a network!'

http://www.jpaulmorrison.com/fbp/compos.shtml

The techniques of Structured Analysis have been around for decades and show how to deal with complexity:

http://yourdon.com/strucanalysis/wiki/index.php?title=Chapter_9#Avoid_Overly_Complex_DFDs

The chapter also describes the approach of dynamically linking sub-graphs:

'This approach naturally coordinated the hierarchy of processes with the stream hierarchy. In addition, since the monitoring process's other job is to stitch the composite into the main network, we could now have "black box" composite components. This facility would allow subnets to be packaged as separate load modules for distribution, which could later be linked with other components by a developer to form the full application network.'

http://www.jpaulmorrison.com/fbp/compos.shtml

The chapter goes on into implementation detail that you might find useful.

Regards, 

Ged

--

You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.

[Alfredo Sistema]

The original idea of graph loading is from Paul, the Screen Manager
chapter, where a component driven by a table loads ( or instantiates )
a component or graph , connects to it, sends data, waits for a
response and then disconnects. Here's where using brackets and
substreams shines, because you can mix components with one input and
one output for this purpose. From the programmers perspective it would
look like instantiating a component and using it as a subroutine.
Another thing I haven't mentioned is that using a DSL can help a lot
with graphs, specially very long pipelines that are unwieldy in
traditional data flow editors.