Demo of DrawFBP on YouTube
Paul Morrison
The URL is https://t.co/E5ZPYjyXGS - feedback would be appreciated!
Paul M.
stewart mackenzie
Quite informative, though would have been good to see the entire network of components properly hooked up and displaying the actual word count.
I recall struggling with DrawFBP a long time ago, this video would have been quite helpful.
IMHO, I encourage you to make the video again, but polish it off by
* removing the fbp motivation - most folks at the stage of DrawFBP research already understand the benefits of fbp.
* make an example that's complex enough to make a non-fbp programmer umm and ahh a bit yet simple enough to get the point of fbp across in a fast manner. Show how the graph abstracts away all the code by (maybe) doing a line count on all the src files of all the components. Ie show how fbp makes flow a first class citizen.
* complete the app; how does one execute the final product? How does one deploy/distribute the code? How does one execute the distributed code, is DrawFBP a required dependency for deployed code? How can one do rapid prototyping (I did get a feel for it by you deleting the ReCompose component)?
* How do I navigate a complex graph?
* How do I create and navigate subnets?
* What do each of the radio buttons on the bottom do?
My 2c worth!
/sjm
Paul Morrison
--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
stewart mackenzie
I wouldn't worry about diving in too quickly, FBP is now on the tongues of many programmers, the concept is generally understood. The main issue is how to do fbp.
Strike a balance by writing up a page on your website going into as much detail as you want and linking in each perinent video as needed. You could break up the videos in 5~10 min special focus chunks. Optimum length? Make 10~20 small videos if you want, each with its own title and strictly keeping to topic. Order them too. I personally prefer a good 25 mins with few repeats of basic yet important concepts and capping it off with a hardcore demonstration.
I'll watch the video again and again if it's a serious knowledge drop.
If Eclipse is the normal tool to use then go for it, but i'd also encourage you to demonstrate on linux using the shell. Personally I feel more comfortable there.
Lastly, it might help to do the video a number of takes, each without cuts and only allow satisfaction once you complete the video in one smooth run. Delete it and start from scratch if you find you're ahhing and creating then deleting components unecessarily.
For example, I really like this video: https://m.youtube.com/watch?v=dP1xVpMPn8M
I feel like I can use this ACME application, it starts out simple enough, then towards the end he gets hardcore and blows me away in awe. Now I know what I could achieve given enough determination. If you're able to communicate the same feeling then IMHO you have succeeded.
/sjm
Paul Morrison
stewart mackenzie
Important concepts like schedulers, IIPs, components etc is crucial knowledge for using FBP.
Might want to make a separate video describing the difference between classical and non-classical fbp.
But there is more to DrawFBP than FBP concepts, I'm highlighting the need for experienced developers to understand how to 1) build 2) execute 3) debug 4) package and distribute FBP graphs.
Option 1 : multiple short videos describing each concept in detail
Option 2 : one full length video describing everything together, if you're worried about charges then go for option 1.
kr
/sjm
P.S. we've gone to great lengths to implement classical fbp in Fractalide, if you see something dodgy in the scheduler or other places do let us know!
stewart mackenzie
1) lack of linux packages
2) sparse documentation on the tool
3) ship example functional component src with the tool, which is needed to demonstrate the component src structure. (I cannot recall if this is the case, but I do recall having a graph that 'did nothing' before me, my incompetence reigns supreme yet again.)
A webpage detailing a hands on approach to developing classical FBP in DrawFBP, which highlights each concept and how to use it with short video examples interwoven would solve problem 2. That would have smoothed my path to using DrawFBP.
Regarding me making the videos, my time is tied up between beekeeping and Fractalide, so I shan't take on this responsibility! What you could do is highlight in the video that it's crossplatform.
Besides one of the FBP godfathers should do that!
Paul Morrison
Rob Halff
Paul Morrison
Paul Tarvydas
> ...
> so easy to add a display process that I didn't bother.
to debug.
It's like using a voltmeter to probe an electric circuit.
Yes, it's obvious, but not "obvious enough" :-).
pt
Matthew Lai
Paul Morrison
https://www.youtube.com/watch?v=9NXYNxDjFWY
I did say it was simple! :-)
Regards,
Paul M.
Paul Morrison
Cheers,
Paul M.
Paul Morrison
https://youtu.be/-AmzfhV2hIU
Paul Morrison
I only know at this point that the xml is valid, but I haven't validated the output of DrawFBP against it, but I hope to try it shortly. Also, for any enhancements, attributes, etc., that people would like to see, please feel free to raise pull requests.
Regards,
Paul
Paul Morrison
To accommodate the new schema, DrawFBP had to change the way a number of boolean fields were generated, so there is a new release (v2.12.1) which does this. This release can still display diagrams encoded using the old format, but will generate the new format when the diagram file is rebuilt.
I would appreciate feedback on the tool (DrawFBP) and on the schema as well, as I am pretty green with this technology!
Cheers,
Paul M.
PS I miswrote - DrawFBP currently generates NoFlo-style JSON, not JSFBP networks (in addition to JavaFBP and C#FBP). I hope to add generation of JSFBP networks in a later release.
PPS Stewart, I have put up a sample .drw diagram on GitHub - https://github.com/jpaulm/drawfbp/blob/master/lib/ConcordR.drw . Again, comments would be appreciated!
Henri Bergius
PS I miswrote - DrawFBP currently generates NoFlo-style JSON, not JSFBP networks (in addition to JavaFBP and C#FBP). I hope to add generation of JSFBP networks in a later release.
stewart mackenzie
Hello Paul,
Yes, we've just moved to something called D3js for our front end, it's coming in slowly. We've developed a technique that will allow us to use fbp to construct a user interface. How to tie this with what Paul T refers to as a factbase, probably this .drw file is a similar concept. We're still a few steps away from that.
DrawFBP is starting to look a lot better, would love to see it published into maven though. With fractalide it was one of the first things we did, published it into the community package repo https://crates.io/crates/rustfbp
Now what in particular do you want me to comment on? This is the solution, what was the original problem?
Kind regards
Stewart
Paul Morrison
https://github.com/flowbased/fbp/blob/master/schemata/graph.yaml seems to be yet another schema! Are they basically the same? At what point in development do they come into play?
/Henri--
Paul Morrison
As you suggested, DrawFBP is up on Maven - it took a while, but I got there in the end! I am not sure why it took me so long to realize the importance of doing this, but of course it makes DrawFBP a lot more accessible!
Thanks for the suggestion!
Paul M.
stewart mackenzie
Hi Paul,
Ah yes, so I noticed! Really happy about that!
Next major thing for DrawFBP: when deploying/sharing a subnet it needs to be as simple as possible to get running for the end user. Can I receive a subnet from someone and run it immediately without having to click on each component and load up all the component binaries or jar files?
Are each of the component binaries/jars stored on maven? Is the DrawFBP vm smart enough to pull down component jars and link them into a graph-to-be-run?
Is this possible to do?
IMHO this is critical for operations. FBP might be good at reducing cost of the maintenance phase but it must also be good at operations.
So far in Fractalide we're able to achieve this:
a => a nand(${maths_boolean_nand}) output => output
b => b nand()
The above subnet undergoes a compilation step and ${maths_boolean_nand} resolves into a path `... a nand(/nix/store/<hash>-maths_boolean_nand) output ...`
containing a subnet or shared object, the fractalide vm is smart enough to determine if /nix/store/<hash>-maths_boolean_nand is a subnet or shared object.
When using the ${...} method, evaluation is lazy, therefore, one could have many 100,000s of components in your git repo and only a smaller subset of dependent components + subnets will be compiled when requested.... indeed our entire vm is built in these classical fbp components, so we're very meta! Thankfully nix immediately catches infinite recursions and component naming clashes!
We use this ${...} method for components that are to be committed into the fractalide repository, typically these components are generic. If users just want to slap and tickle (experiment) components i.e. pass in different IIPs, then just use `a nand(maths_boolean_nand) output` this file can be passed into the fractalide vm/shell called `lain`.
I.e. `lain /path/to/fbp.subnet`
(Still vapourware as I'm battling with parser combinators, but much of it already works)
Lain then will compile this script and run it accordingly. Obviously if you don't have that component it won't work.
BUT! Passing in dynamic IIPs via fbp flow script is still too verbose for most users, particularly operations! I'm currently working on a shell which compiles standard unix/gnu tool naming conventions i.e. `cd /directory` etc to fbp flowscript, quickly updating IIPs and creating new subnet hierarchies, again at the speed of short bash like commands... well a tad slower.
In other words a shell is a typical way to quickly run and execute program, it should be emulated! Besides the shell is the king of fbp, it kinda makes sense to implement a shell in classical fbp.
It should be possible to deploy and run subnets of 1000s of components with a single simple bash like command. So this will be a shell on steroids.
Probably a bit early to announce but I'm nearly done with this, I hope someone can test it out.
kr/sjm
Paul Morrison
Hi Paul,
Ah yes, so I noticed! Really happy about that!
Next major thing for DrawFBP: when deploying/sharing a subnet it needs to be as simple as possible to get running for the end user. Can I receive a subnet from someone and run it immediately without having to click on each component and load up all the component binaries or jar files?
Alternatively, you can use the double-line convention: in this case the subnet block references another .drw file, and the subnet is more of a white box (light grey).... Double-clicking on the double-line block brings up the subnet's diagram in another panel, so you can work with both the master diagram and the subnet diagram independently. The class associations can be stored in the .drw files, so, yes, these could be provided by a 3rd party, as long as the relevant jar files are also provided.
Are each of the component binaries/jars stored on maven? Is the DrawFBP vm smart enough to pull down component jars and link them into a graph-to-be-run?
Is this possible to do?
stewart mackenzie
> would be useful.
without needing a garbage collector. Which means our scheduler is
quite different from DrawFBP's scheduler.
Secondly Fractalide's IPs are encoded using Capnproto contracts
(capnproto.org) which allows us to send IPs to any language that
supports capnproto bindings outside of the our vm.
Thirdly fractalide's components support strong contracts. Components
only accept precisely defined data and reject IPs/connections
otherwise (unless you explicitly accept `any` contract). This is a
strong security mechanism to prevent weird machines or strange loops
as prescribed by the langsec.org group. I'm uncertain (longterm) if
strong contract support will be a bad choice. So far it hasn't been a
restricting factor at all, indeed quite the opposite, in the sense we
have strong confidence of data traversing our system. This seems to
have removed an entire debugging layer, that of needing to introspect
or probing IPs flowing through a system. One just needs to see the
capnproto schema to understand. I could be wrong but if it becomes a
problem it'll be addressed.
Refactoring DrawFBP to support such an environment would 1) require a
lot of work using the Java/C++ programming language, which means
Fractalide doesn't advance. 2) My suspicion is: the overhead of
manually managing the versions/compilations of all the components and
coordinating the build makes DrawFBP hostile to deployment, compared
to today's programming languages with language level package managers.
That's why Fractalide uses http://nixos.org/nix/ as its package
manager. Nix comes with it's own expression language and thus handles
the coordination and building of all our components automatically.
Shimming that into DrawFBP would be hard. I just don't see how it
would happen.
One serious advantage when using Nix: Nix isn't just a language level
package manager it also OS level, which means a component might have
third party dependencies such as openssl, and iff that component is
called into scope of a subnet compilation then openssl will be
downloaded and made available on the system. This means a component is
powerful enough to setup its own environment which might include
dependencies such as python or libffi or any library presented here:
http://nixos.org/nixos/packages.html, or indeed even closed src
components if needed. This is real power to developers and operations
teams, it removes an entire class of headaches from both parties.
Given that Nix has its own CI (http://nixos.org/hydra/) and CD
(http://nixos.org/nixops) solutions it means those problems are also
solved. Most importantly we rely on the very great nix community which
as you can see get shit done: https://octoverse.github.com/ (search
for "nixpkgs"), (Rust is also on that list too). So from a popularity
point of view this is the beginning of a very popular wave that's
about to hit. I think Fractalide is nicely placed to take advantage of
this "trend" wave. [1]
I'm also suspect companies that build their infrastructure off the
combination of nix+rust+fbp = fractalide will gain a good advantage
over other stacks.
example: https://github.com/fractalide/fractalide/blob/c2c918dee5999303b11ba426226accc4bf34a8d4/components/nucleus/find/component/default.nix#L10
this component calls nix into scope.
This is the implementation of the nucleus_find_component component
https://github.com/fractalide/fractalide/blob/c2c918dee5999303b11ba426226accc4bf34a8d4/components/nucleus/find/component/src/lib.rs
3) I'm not convinced having a flashy front ends increase FBP adoption.
I think it's eye candy that is very useful to the developer, but, and
this is crucial in my view, a developer will not chose something that
makes life difficult for people to deploy code, hence FBP remains a
developer's dream. Secondly, how does a developer debug remote
subnets? The graph editor should operate over ssh. So for us it would
seem we're taking a completely different path, that of using
https://github.com/Ticki/termion. Programmers must be able to remotely
access a machine and administer a subnet. Erlang with its shell sets a
good example. Hence we are going the direction of a shell cum graph
editor over ssh.
If you'd like to assist then it'll need to be on Fractalide I'm
afraid, too much infrastructure revolves around coordination of
dependencies and subnets and finding ways to leverage the best of nix
to liberate us from this inherent problem that arises due to separate
components in FBP. Nix really is a secret weapon here. Noflo gets away
with dissemination issues because their components are centralized and
their servers will disseminate the rendered frontend. I'm looking at
disseminating the components not just the user interface. But it means
one has to code in javascript || derivatives. Personally I prefer a
nice systems language. A mechanism which automates dependency
versioning and building IMHO is the needed ingredient for FBP to be
adopted en mass. Without this, FBP is dead in the water. The flashy
editor has already been shown not to be the essential ingredient to
FBP adoption, in other words a simple terminal based graph editor
should suffice!
> One question: is there a role for both DrawFBP and Fractalide in the FBP
> universe?
Portraying Fractalide as an implementation of FBP only would be a mistake.
Fractalide uses FBP a subset of Dataflow, to solve a problem, namely
number 2 of https://github.com/fractalide/fractalide/blob/master/README.md.
Just pushing a programming concept won't get Fractalide anywhere. The
project needs to solve insanely expensive problems, we're operating on
new levels of abstractions which make it cheap for us! Hence we're
reorienting Fractalide to solve the same problem docker solves... only
done much better, with an add-on of allowing programs to compose and
pipe nicely thanks to FBP. The hopes are to create a streaming layer
whereby folks can use components to obtain decent insights into large
amounts of data. This would have to be exposed as a shell. In other
words... deploying code to your system means you _just_ type the
subnet name and press enter. An entire subnet of possibly thousands of
components would arrive and be executed on your machine nicely piping
data to other components you typed in the shell.
i.e. $ cat earthquakes.txt | stats count mean(mag), stdev(mag),
var(mag) BY magType
Keep in mind all these commands would be implemented in Fractalide
FBP/Flowscript
Granted the CLI is a limited in that full blown FBP explicit input
ports and output ports are made implicit, hidden and defaulted (named
pipes). If you wanted to implement the stats command you could press a
mode switch button and your terminal editor would switch to a textual
graph editor. This would allow you to connect the full FBP port names.
This is where we're going now and is unimplemented yet.
I'm experimenting with the name Flowscript over FBP. That way we won't
have an identity crisis (viz get embroiled in the whole classical FBP
vs non-classical FBP). Secondly, we'll be dealing with people who use
names like "bashscript", so something the is an incremental step up
from there might work. i.e. "flowscript" which would express the CLI
interface language and the full blown FBP .... I don't know maybe call
it Flowscript FBP would be a good balance? We have enough added
features that deviate from classical FBP in that we have this concept
of a teleportation portal called "portal" which like the Valve game
Portal (see https://www.youtube.com/watch?v=TluRVBhmf8w) allows one to
easily keep state around by simply throw the state into the portal and
it pops out on the next function run, to be sure state doesn't leak
outside the component. (see
https://github.com/fractalide/fractalide/blob/a480665034cb863c5e0d35e636d5e912ead0627a/components/nucleus/flow/scheduler/src/lib.rs#L54
for an example usage). Makes things super convenient with things like
hashmaps etc.
It would be really excellent having more people take Fractalide
seriously and contribute code that succinctly solves real world
problems.
[1] I really dislike these trend waves as it means our discipline
isn't firmly based in the scientific process but on the herd mentality
of people. But it's a fact of life at the moment.
Paul Morrison
Paul Morrison
Also, the combination of DrawFBP and a powerful IDE produces a very straightforward natural interface IMHO! See my 3 videos describing this interaction, especially the 2nd one: https://www.youtube.com/watch?v=9NXYNxDjFWY .
stewart mackenzie
Okay, I see.
I'm very curious, in the 70ies,
* A typical installation consisted of how many components?
* How did you setup and install each component?
In distributed Erlang it's normal to have a shell you can connect to and debug a running erlang runtime system. Indeed it has a graphical frontend giving the programmer statistics of the system. It's even possible to hotswap a component.
My main point is not so much about the graph editor but more so how to easily pull dependencies into a graph, link them up and execute the graph. I'm very curious to know if you automated this process in the 70ies, if so how did you do it? Keeping in mind, if fbp wants to go global, it needs to globally share components which require a proper naming convention. See Zooko's triangle.
stewart mackenzie
Interesting, thank you!
Regarding the naming, it's important to also factor in the compenent version. I dislike semantic versioning but many appreciate it. I think there are better ways of respecting ABI contracts when linking libraries. iirc java's naming scheme is just the reverse of your domain name n levels deep
I'm aware that Erlang and FBP occupy different spaces, I'm refering to drawing inspiration from the management and administration layers. Secondly FBP is actually very good for long lived server type environments (so is Erlang!), and servers are spun up automatically by the 100s in some cases. (This is because we have data dissemination built in at the application level, when it should me built in at the networking level. )
Paul M.Regards,I would guess from 50 to 200 processes. On the IBM mainframe systems we used, an FBP application network was specified using macros, compiled, and linked, which would then automatically combine the network with the components ( https://en.wikipedia.org/wiki/Linker_(computing) ), resulting in what was called a load module. As we controlled all the code, we merely needed a company-internal naming convention. Program names were restricted to 8 characters max, so we had to be creative!No hot-swapping - these were batch jobs, so you simply modified the network if necessary, relinked, retested, promoted and restarted! Job submission was typically under the control of job scheduling software.Also, remember Erlang is a different application space: as I understand it, it supports multitudes of short-running processes, whereas classical FBP tends to lean to a smaller number of long-running processes.As you suggested earlier, since you can have multiple component jar files, then that gives us basically a 2-level naming convention: jar file name/component name. DrawFBP supports this currently, but I'm wondering if we should try to extend it (and the other classical FBP run-time implementations) to Maven 3-level artifact names. That sounds like something worth doing...?