The Annealer

[Max Yaffe]

I've been interested in FBP for a long time, sadly though from a distance.   However, that hasn't stopped me from having ideas.

One reason I haven't leapt into it, is that I work in fast data acquisition systems, say with message/package sizes of 20ish bytes running at up to 1 million messages /sec.   Most of my operations are along a linear flow, eg Make a measurement, pass it to an overload detector, pass that to a filter,  pass that to something else...   I start to worry about the overhead of process/thread/fiber switching between different operations.  Running at a MHz, this overhead can add up compared to writing one stage that encompasses several operations.   At the same time, having a formalism that allows someone to string together and rearrange them makes a lot of sense.

So my idea is to create an "annealer" that would take a FBP diagram, and collapse those operations which can be collapsed together into a single operation, oblivious to the designer, i.e. anneal them together.  The signaling/switching/queuing between the annealed operations would be eliminated.  The annealer would be a separate optimizing step in the compilation from diagram to final system.

Is there some lovely wheel already out there that I'm reinventing?  Or is the overhead with a well chosen cooperative multitasker so minor that I am worring about a nothing? 

Your thoughts?

Max

[Paul Tarvydas]

The reason I backed into FBP-like techniques was that I needed blazing speed.

I’d been building RTOS’es and compilers and embedded systems and could see where the inefficiencies laid.

FBP-think allowed me to throw away most of the O/S!

A "kernel" that supports the bare minimum for an FBP system is about 1 page of code.

After some 25+ years refining these ideas, my most current thinking is on my blog (dedicated only to this topic, read from bottom to top) and github:

https://bittarvydas.wordpress.com

https://github.com/guitarvydas/collate-fbp-classic


I'm currently reading Matt Carkci's book (Dataflow & Reactive Programming Systems). What you describe sounds like Chapter 8 "Synchronous Dataflow Implementation". He gives an algorithm (referencing an IEEE paper) for compiling (is this the same as annealing???) such a program.

I'd be happy to discuss further...

pt

[Alfredo Sistema]

Cooperative multitasking can be blazing fast, it is in essence a while loop that calls a sequence of update functions.

Communication can be blazing fast too, writing to a shared array in memory.

There really isn't a requirement to support all the FBP ideas at once, there's not even a need to load graphs at runtime, you might as well write them as function calls that wire the nodes to the shared arrays/stacks.

Why don't you spend an afternoon writing a small prototype and test it? Paul T is right about the one page of code, you don't need to make a huge system for the basics.

FBP is 100% pragmatism, there are no high brow abstractions required.

I look forward to see what you'll build.

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sam Watkins]

On Thu, Jun 25, 2015 at 12:51:15PM -0700, Max Yaffe wrote:
> So my idea is to create an "annealer" that would take a FBP diagram, and
> collapse those operations which can be collapsed together into a single
> operation, oblivious to the designer, i.e. anneal them together.

I think this is a good idea.

I don't know any FBP system that can optimize in this way.
It seems similar to function inlining in C, with similar trade-offs.

It might be especially helpful to do this optimization if we are using
small components for arithmetic and logic operations.
In general it would be best not to combine several large components.

Our "ideal FBP compiler" might combine adjacent components if they are
small, coded in the same language, and their I/O modes are compatible.

[Paul Morrison]

Seems like a good idea, provided the seams are fairly obvious (for debugging).  Or maybe you only do this in production.

It seems to me that the only constraint is that you can only anneal 1-in-1-out processes to their neighbours. IMO size doesn't matter, and I don't know what I/O modes are - as long as it is a working network when you start, I would think you could anneal any 1:1 process to its neighbour(s).  Of course, it probably only makes sense to do this when there is a high volume of data IPs going across the relevant connection...