I think a new programming language needed...

[mark taylor]

Hi,

I've said this before, but can I say it again... I think a new
programming language is needed. Is anyone here interested in helping?

I've made a start writing down some ideas (not quite finished enough
to show others, but hopefully will be soon), and would very much like
to get some feedback.

NB - please note the change of account name - I was forced to create a
new google account due to forces beyond my control. anyone that has
emailed me before can use the old email address (preferred), I will
try to read the new email address but can't guarantee this. It's best
just to reply here.

Mark Taylor.

[David Neil Murphy]

Don't agree we need yet another language. What is needed is effective plumbing that will allow corporates to use the language of choice.  Properly modularised even Cobol would do for this type of work, provided the plumbing is there.

[Dan]

I fully agree. For me it is quite strange why so many programmers (FBP
fans) still think that this is not so important. After all we can
simulate any programming language in any other programming language
but this does not bring to us any efficiency.

The main problem that I foresee is that doing FBP in a "classic"
programming language stops you to think in FBP style. All the time you
have to solve problems of "implementation". Optimizing a FBP program
is very important. You can do optimizations only when you have global
information available. That's why FBP is so good because you have data
flow explicitly available (just an easy example: a temporary data used
to transport information from one component to another could be reused
many times without reallocation wherever that type is used for an IP.
It should be the equivalent of the classical "call stack" in this case
but with a stackless implementation).

Imagine the simplest FBP program ever written:
A -> B;
How many lines of code you have to write in C++ in order to accomplish
the FBP code above? Well, I am telling you, quite many (listeners,
updaters, changes monitors, dirty bits, message system, parameter's
memory allocation management etc. depending on the situation).

At first glance the code says something very simple: the information
in component A (value of A) is continually connected to the
information in B (value of B). This is not trivial in a classical
programming language. In a classical language you have to explicitly
allocate / declare variables to accomplish this and pay attention to
them. What it is interesting is that in FBP you don't have to do that.
You don't have to explicitly declare variables just to move the
information around but you have to declare a connection instead. This
is a huge difference.

I will end with another very important reason why a new language is
necessary: (real world) modeling. It is so hard to do modeling in a
classical programming language. You might think that many languages
deal with the concept of "class", some others bring "templates" and
"prototypes" into place. It's not enough. I haven't seen any
programming language that satisfies me with respect to this aspect
yet.

This is also an answer to J. Paul M. regarding the subject: why FBP
did not launch itself yet in the programming world.

If I want to use FBP I want to have a language now and write in
seconds a small program that could integrate other components
(wrappers) written in different languages. Yes, integration with other
languages is very important but FBP is not just a coordination system,
it is much more. OTOH, in my view, a "coordination language" is not
enough to implement truly powerful FBP programs. FBP should be a
complete platform and this includes a FBP language on its own.

Regards,
Dan

On Sep 5, 3:48 am, mark taylor <the-mark-tay...@blueyonder.co.uk>
wrote:

[mark taylor]

Hi,

On Sep 5, 10:28 am, Dan <dpolo...@gmail.com> wrote:
>
> Imagine the simplest FBP program ever written:
> A -> B;
> How many lines of code you have to write in C++ in order to accomplish
> the FBP code above?

This is my point exactly... using traditional control-flow languages
forces the programmer to write enormous amounts of clutter that only
serves to (a) vastly increase the risk of bugs, and (b) obscure the
benefits & clarity of the new processing paradigm.

In fact, your example pseudo-code above could be a complete working
program in my proposed language. The only difference would be that in
my proposed syntax, it would be slightly simpler due to the fact that
lines/pipes/connectors (or whatever you what to call them) would
always flow downwards with an (implicit) vertical line between them.
So in my imaginary language, your example program would look like
this:

--start--
A
B
--end--

And if the components were "off-the-shelf" or library components (e.g.
a sort or filter or input/output component), then *no further code
would be necessary*.

In my imaginary language, what would flow along the (implicit) pipes
would be self-describing data objects, which would have attached to
them (implicitly/transparently) the code snippets needed for (most)
manipulation by the components - i.e. in many cases the components
would not need to have any knowledge of the types of data objects
flowing through them. The classic example of this would be a sort
component - it would "call" methods/subroutines attached to the data
object to decide the sequential order of any set of objects it was
looking at, then emit them downstream in correct sequence.

regards,
Mark.

[Danny Ayers]

I'm new here, so please forgive my butting in, but there seem to be
two fairly orthogonal aims in this thread: one is a notation to
express flow, the other to be able to optimize processing.

For a notation, I'm curious, what's needed beyond traditional Unix |
and tee with variables and/or files for buffers? I'm certainly not
proposing bash or whatever as the language (bit too ugly!) but don't
shells have all the necessary constructs for describing flows?

I'm not sure, but presumably a precursor of optimization would be
monitoring, so you'd want to be able to attach some kind of hook to
the | or whatever to see how it was using resources.

re. operating transparently over different types of data - I agree
self-describing objects are probably the best way to go to support
polymorphism while staying loosely-coupled. (I've some HTTP-related
thoughts on this, will post later in a fresh thread)

Cheers,
Danny.

--
http://dannyayers.com

[John Cowan]

Danny Ayers scripsit:

> For a notation, I'm curious, what's needed beyond traditional Unix |
> and tee with variables and/or files for buffers? I'm certainly not
> proposing bash or whatever as the language (bit too ugly!) but don't
> shells have all the necessary constructs for describing flows?

Technically, if you make use of /dev/fdn, you can construct all the
loops you want. You need some conventions about components that aren't
one-input/one-output, though, and someone has to write those components,
like a program that writes some inputs to fd 1 (standard output) and
the others to fd 3 (fd 2 is standard error) depending on some criterion
expressed in the arguments.

The art of shell programming is slowly dying, partly because bash is such
a terrible programming language. I have a design in mind for a new shell
based on rc (the Plan 9 shell) and Lua, called plwsh (pronounced "ploosh",
meaning Powerful LightWeight SHell or Processes+Lua+Wiring SHell, take
your pick). Lua is a powerful dynamically typed programming language
with coroutines, with a small implementation designed to be embedded
in C programs, and rc ("run commands", the same origin as .bashrc,
.mailrc, etc.) breaks Posix compatibility (but not basic usage) to get
a shell right.

Put these two together and I think you'd really have something powerful,
with low enough overhead to run scads of them on a modest modern Unix
(Windows would work, but it would be an issue because Windows processes
are incredibly slow to start.) I haven't got any further than fiddling
about with unifying the grammar of the two languages, though. Eventually,
when I get time (or some help with the project) ....

--
John Cowan http://www.ccil.org/~cowan co...@ccil.org
Please leave your values Check your assumptions. In fact,
at the front desk. check your assumptions at the door.
--sign in Paris hotel --Cordelia Vorkosigan

[Danny Ayers]

On 5 September 2011 22:18, John Cowan <co...@mercury.ccil.org> wrote:
> Danny Ayers scripsit:
>
>> For a notation, I'm curious, what's needed beyond traditional Unix |
>> and tee with variables and/or files for buffers? I'm certainly not
>> proposing bash or whatever as the language (bit too ugly!) but don't
>> shells have all the necessary constructs for describing flows?
>
> Technically, if you make use of /dev/fdn, you can construct all the
> loops you want.  You need some conventions about components that aren't
> one-input/one-output, though,

tee is one-input, two-output, no? And couldn't you do two input with x
| somewhere; y | somewhere (though that might not be the convention
you'd want :)

and someone has to write those components,
> like a program that writes some inputs to fd 1 (standard output) and
> the others to fd 3 (fd 2 is standard error) depending on some criterion
> expressed in the arguments.
>
> The art of shell programming is slowly dying, partly because bash is such
> a terrible programming language.

But how much is just due to the horrible syntax? Joining commands
together seems a pretty powerful approach (essentially flow-based).

 I have a design in mind for a new shell
> based on rc (the Plan 9 shell) and Lua, called plwsh (pronounced "ploosh",
> meaning Powerful LightWeight SHell or Processes+Lua+Wiring SHell, take
> your pick).  Lua is a powerful dynamically typed programming language
> with coroutines, with a small implementation designed to be embedded
> in C programs, and rc ("run commands", the same origin as .bashrc,
> .mailrc, etc.)  breaks Posix compatibility (but not basic usage) to get
> a shell right.

interesting, all new to me

> Put these two together and I think you'd really have something powerful,
> with low enough overhead to run scads of them on a modest modern Unix
> (Windows would work, but it would be an issue because Windows processes
> are incredibly slow to start.)  I haven't got any further than fiddling
> about with unifying the grammar of the two languages, though.  Eventually,
> when I get time (or some help with the project) ....

Good luck!

[John Cowan]

Danny Ayers scripsit:

> tee is one-input, two-output, no?

Well, yes, if you use a named pipe or a non-bash shell that can pipe
stuff using descriptors other than 0 or 1, as rc can.

> And couldn't you do two input with x
>
> | somewhere; y | somewhere (though that might not be the convention
>
> you'd want :)

That would give you two separate instantiations of "somewhere".

> But how much is just due to the horrible syntax? Joining commands
> together seems a pretty powerful approach (essentially flow-based).

There was also the problem of knowing what components would be available
on a random system. Posix.2 and the GNU core-utils package help there.

> Good luck!

That's what everyone says, as in "good luck finding anyone to help". :-)

[Paul Morrison]

On Sep 5, 5:28 am, Dan <dpolo...@gmail.com> wrote:
>
> If I want to use FBP I want to have a language now and write in
> seconds a small program that could integrate other components
> (wrappers) written in different languages.

If you take the idea of componentry to an extreme, you shouldn't care
what language the component is written in. And, in this case, the
language a component is written in should be the coder's choice. A
principle of FBP is that no one language can do everything - and
shouldn't try to. By the way, I am assuming here that we are just
talking about the language(s) for components - I don't see a need for
a connection language, as the network is just a list of connections.
Unfortunately modern languages seem to exist in their own little
worlds, and they are not friendly to other languages.

As I said earlier, there has been a tendency to put more and more
infrastructure in the language environment, resulting in separate VMs
for Java and C#, so it's very hard to mix and match languages.
However, as part of this trend, modern languages are now basically
lists of subroutines, with a few decisions thrown in, and FBP makes a
number of present-day language features unnecessary, so IMO what we
need is a range of languages that all compile to a pretty low-level
FBP-friendly VM. Actually, one VM per hardware architecture...
Unless we can influence Parrot...

[John Cowan]

Paul Morrison scripsit:

> By the way, I am assuming here that we are just talking about the
> language(s) for components - I don't see a need for a connection
> language, as the network is just a list of connections.

Well, that works if you always use a graphical layout tool. But a nice
pipeline like "ls | wc -l" is a lot more perspicuous than

comp1 = new ls
comp2 = new wc
connect(comp1.out, comp2.in)
connect("-l", comp2.options)
run

And the difference only gets more drastic as the pipeline gets longer
and hairier.

--
You're a brave man! Go and break through the John Cowan
lines, and remember while you're out there co...@ccil.org
risking life and limb through shot and shell, http://ccil.org/~cowan
we'll be in here thinking what a sucker you are!
--Rufus T. Firefly

[Ralf Westphal]

I agree a small language to describe flows would be helpfull.
A designer of any kind might not be at hand at all times.
Also a designer would need to persist what has been drawn to some
format, preferrably text.

However this language (let me call it fbp.txt) should be focused on
just flows. It should not try to mimic 3GLs.
Also it should start small, and then be incrementally developed.
Thinking up first a grand scheme likely will not lead to success.

I have experimented with the following language. The sample code
describes a solution to a small problem:
Reformat a text file so that each line is only at most n chars long.

The top level of the FBP program looks like this:

(run) -(args:string[])-> (Parse command line).Filename
-(filename:string)-> (Analyse source file)
-(word:string*)-> Words.(Assemble destination file);

(Parse command line).Filename -(filename:string)-> Filename.(Assemble
destination file);
(Parse command line).LineLength -(int)-> LineLength.(Assemble
destination file);


(Analyse source file) {
(this) -(filename:string)-> (Read lines from text file)
-(line:string*)-> (Split into words)
-(word:string*)-> (this)
}


(Assemble destination file) {
(this.Words) -(word:string*)-> Words.(Assemble lines from words)
-(line:string*)-> Lines.(Write lines to file);

(this.LineLength) -(int)-> LineLength.(Assemble lines from words);

(this.Filename) -(string)-> Filename.(Write lines to file)
}

Nodes like (Write lines to file) are singeltons by default.
Nodes can be composites like (Assemble destination file). That way
flows can be described on any number of strata (as in Stratified
Design, see Abelson & Sussman).
Connections are typed.
Nodes can have any number of named input/output "pins".
To denote lists of values put a * after he type of a connection.

I have used this notation for a couple of months and it works very
well.

Any details of the leaf nodes are left to whatever implementation
language is used.
All of he notation can be translated into plain C# or Java if you
like.
But of course a VM could be targeted as well. In fact that's what I'm
working on currently.
It is supposed to make writing leaf nodes as easy as writing a single
function, e.g.

IEnumerable<string> Split_into_words(IEnumerable<string> words) {
...
}

Despite all this, though, I've found that much less seems to be
sufficient. The above fbp.txt syntax is quite expressive.
But priming a VM to execute leaf nodes like above composed into flows
can be made easier.
Simple tables are sufficient, e.g.

* // name of root flow
.run, Parse-command-line
Parse-command-line.Filename, Analyse-source-file.Filename
Analyse-source-file.Words, Assemble-destination-file.Words

Analyse-source-file // name of composit
.Filename, Read-lines-from-text-file
Read-lines-from-text-file, Split-into-words
Split-into-words, .Words

...

So what I'm thinking right now is, whether we really need an elaborate
language.

[Henri Bergius]

On Tue, Sep 6, 2011 at 9:38 AM, Ralf Westphal <ral...@googlemail.com> wrote:
> I agree a small language to describe flows would be helpfull.

It certainly would. Here are my current stabs at it:

NoFlo graph API:

graph = noflo.graph.createGraph "My new flow"
graph.addNode "read", "ReadFile"
graph.addNode "show", "Output"
graph.addEdge "read", "out", "show", "in"
graph.addInitial "somefile.txt", "read", "source"

NoFlo interactive shell:

$ noflo -i
NoFlo>> new My new flow
My new flow>> add read ReadFile
My new flow>> add show Output
My new flow>> connect read out show in
My new flow>> send read source somefile.txt

I'm not entirely happy with either. Maybe switching from component and
port as separate arguments to component:port syntax would help?

--
Henri Bergius
Motorcycle Adventures and Free Software
http://bergie.iki.fi/

Jabber: henri....@gmail.com
Microblogs: @bergie

[Paul Morrison]

On Sep 6, 2:38 am, Ralf Westphal <ralf...@googlemail.com> wrote:

> So what I'm thinking right now is, whether we really need an elaborate
> language.
>

Exactly, Ralf. You can take a really simple notation - linear or
tables - and generate the "connect" calls. The only FBP
implementation by IBM that ever got on the market, DFDM, used the
following notation:

'data.fil'->OPT Reader(THFILERD) OUT -> IN Selector(THSPLIT) MATCH -
> ...,
Selector NOMATCH -> ...

where "Reader" is the process name, and THFILERD the component name.
You could either interpret it at run time, or compile it into an
executable.

Wayne Stevens felt it should be an industry standard and described it
in one of his books on application development. As Ralf says, that's
basically all you need. I never got around to implementing it for
JavaFBP or C#FBP because the "connect" calls seem pretty simple
themselves (and also because I like the diagramming tool approach)...
but it would be very easy!

[Dan]

I don't think that any FBP language should contain any "connect" calls
or code like:

graph = noflo.graph.createGraph "My new flow"
graph.addNode "read", "ReadFile"
graph.addNode "show", "Output"
graph.addEdge "read", "out", "show", "in"

I think an FBP language should not contain any calls in the first
place. It is exactly what FBP eliminates. The FBP program is, at first
glance, a snapshot / description of the network of components (i.e.
current state).

FBP is declarative programming not imperative and also not functional.
The "functions" should be used only in the mathematical sense, never
as they are used in imperative programming (i.e. using "calls").

By the way: who does the calls? A special parent component? A
component does not call anything and if it does when it calls and
what?

It's a paradigm shift. Well, somebody does a kind of
"calls" (i.e.system) as a reaction of IPs flow, but the FBP system
implementation is not really necessary based on a stack (with function
calls using a stack).

Dan

[John Cowan]

Dan scripsit:

> I don't think that any FBP language should contain any "connect"
> calls or code like:
> graph = noflo.graph.createGraph "My new flow"
> graph.addNode "read", "ReadFile"
> graph.addNode "show", "Output"
> graph.addEdge "read", "out", "show", "in"

[snip]

> FBP is declarative programming not imperative and also not functional.

The language above *is* declarative: don't be fooled by its form.
It makes no provision for dynamic connections or disconnections.
It just defines a specific graph for the engine to execute.

> The "functions" should be used only in the mathematical sense, never >
as they are used in imperative programming (i.e. using "calls").

That's what it means to be a (pure) functional language.

> By the way: who does the calls? A special parent component? A
> component does not call anything and if it does when it calls and what?

The initialization phase of the overall engine: in shell programming,
the shell. Posix shell language is not declarative/functional as a whole,
but the subset of it excluding assignments and backquotes (but including
conditionals and loops) is.

--
I now introduce Professor Smullyan, John Cowan
who will prove to you that either co...@ccil.org
he doesn't exist or you don't exist, http://www.ccil.org/~cowan
but you won't know which. --Melvin Fitting

[Ralf Westphal]

On 6 Sep., 15:38, Paul Morrison <paul.morri...@rogers.com> wrote:
> Wayne Stevens felt it should be an industry standard and described it
> in one of his books on application development.  As Ralf says, that's
> basically all you need.  

You only need to relate "ports" to each other. Which port of a process
is connected to which other port?
That´s simply a list:

*.Run, ParseCommandLine.Args
ParseCommandLine.Filename, Read.Filename
ParseCommandLine.LineLength, ReformatLines.LineLength
Read.Lines, SplitIntoWords.Lines
SplitIntoWords.Words, ReformatLines.Words
ReformatLines.Lines, Write.Lines

From this you can build a graph, if you like, that you can visualize.
Or you take this as an execution plan: an interpreter can go over
these mappings and pull processes from a container to schedule them
for execution. That´s what I´m working on right now.

The results are promising. A prototype can execute such flows in three
modes: sync, async, and parallel.
Also it can automatically log trace messages so you can later on check
how the execution went.
And it can do slow-motion execution so you actually can watch the
program during execution.
Last but not least I´m planning to stop the execution for changing the
flow and/or the processes at runtime.
That would make live evolution of flows possible.

>I never got around to implementing it for
> JavaFBP or C#FBP because the "connect" calls seem pretty simple
> themselves (and also because I like the diagramming tool approach)...
> but it would be very easy!

Diagrams are nice for explaining flows. But after having worked with a
couple of diagramming tools I have come to the conclusion there is a
quicker way: an IDE for flows should have three panes:

-a text pane for editing some text based flow description like above
-a graph pane which is updated automatically showing a graphical
representation of the textual flow description
-a tree pane also updated automatically showing the same flow as a
hierarchy of nodes which can be rearranged. This is for easy
refactoring of flows.

Hope I´ll get around building such an IDE later this year.

[John Cowan]

Ralf Westphal scripsit:

> You only need to relate "ports" to each other. Which port of a process
> is connected to which other port?

You also have to name processes, because a process may appear more than
once in a pipeline or flow graph.

--
Samuel Johnson on playing the violin: John Cowan
"Difficult do you call it, Sir? co...@ccil.org
I wish it were impossible." http://www.ccil.org/~cowan

[mark taylor]

Hi,

NB this post contains pseudo-code - it will not makes sense unless a
fixed width font is used!

On Sep 6, 7:38 am, Ralf Westphal <ralf...@googlemail.com> wrote:
>
> ...The sample code

> describes a solution to a small problem:

> Reformat a text file so that each line is only at most n chars long...
>

In my imaginary language this would like this:

--start--
split_lines_into_words
join_words_into_lines(max_line_len=$1)
--end--

split_lines_into_words and join_words_into_lines would standard
library components, or could be written easily in the procedural code
my language would allow or any other language.

the flow of data is downwards. there are implicit vertical lines
between the two components, before the first component & after the
second component, e.g. like this:

-> split_lines_into_words -> join_words_into_lines(max_line_len=$1) ->

if reformat_text was executed on a unix box, it could called in bash
shell like this:

<bash_shell_prompt>$ reformat_text_file < input_file > output_file

the language runtime would automatically deal with joining the
get_text_line component to the input file.

in my language, split_into_words could be coded something like this:

--start--
get_text_line; --getline is a library component that reads from
--the input stream and writes lines to the output
stream
--one line at a time
split_text_line(' '); --split is a library component that splits a
line
--into chunks delimited by the parameter
character pattern
--end--

split_text_line would probably also be a library component, but could
be coded something like this (below), in the procedural code that my
imaginary language would allow. this is standard procedural code, not
the streams + components code of previous examples. the code of the
component deals with a single object on the input stream (in this case
a line), and the runtime would automatically call the component in a
loop for each object of the input stream. each time it is called the
component outputs 1 or more strings to the output stream (each
containing either a word or string of delimiting characters) for the
input object it is passed.

--start--
component split_text_line(pattern:string)
streams
input line:string; --the input stream consists of string objects;
output word:string; --the output stream consists of string objects;
variables
n:number;
w:string;
begin
n:=1;
while (n <= length)
begin
while ( (n <= length(line))
&& (! line[n] in pattern)
)
begin
w + line[n];
n += 1;
end;
word:=w; --outputs the word itself
w:='';
while ( (n <= length(line))
&& (line[n] in pattern)
)
begin
w + line[n];
n += 1;
end;
word:=w; --outputs the whitespace delimiting the word
w:='';
end;
end;
--end--

So my language would have two "flavours"... one at the component
+streams level, and a very simple procedural language for building
components.

the only detailed code is to split a single line of text into words.
all the buffering, looping, input & output to files, error handling,
would all be handled by the runtime system. because much of the work
would be handled by the runtime, the procedural language would be very
simple.

component+streams level code would be inherently parallel - objects on
input streams could be processed in parallel by the runtime system
calling multiple instances of the components to read the stream; and
it would also be inherently pipelined... meaning that objects
downstream could be processed in parallel with objects upstream.

I hope the above makes sense... please feel free to pick holes in
this.

Mark.

[mark taylor]

Hi,

I'm sorry folks, even when viewed in fixed-text, my post is still
scrambled in parts :(

here's the worst bits done again...

in my language, split_into_words could be coded something like this:

--start--
get_text_line;

split_text_line(' ');
--end--

getline is a library component that reads from the input stream and
writes lines to the output stream one line at a time

split_text_line is a library component that splits a line into chunks

delimited by the parameter character pattern

And later on the start of the definition of the component
split_text_line...

--start--
component split_text_line(pattern:string)
streams
input line:string;

output word:string;

variables
n:number;
w:string;
begin

<etc>...

the input & output stream consist of string objects.

Sorry about this. Maybe I should write this all up & post it as a pdf
or something.

Mark.

[Paul Morrison]

On Sep 6, 6:15 pm, Ralf Westphal <ralf...@googlemail.com> wrote:

> From this you can build a graph, if you like, that you can visualize.

I don't see why people keep talking about turning text into diagrams.
It's so much easier to go the other way. The text has less info than
the diagram, so it makes sense to work with the diagrams. Of course,
you have to have an FBP-friendly diagramming tool, which is what I am
trying to do with DrawFBP.

Consider the diagram in http:www.jpaulmorrison.com/fbp/image010.jpg -
this is about 1/3 of a long-running batch program, which was running
every day for at least 35 years (and may still be)... You certainly
can generate a text-type network description from this, but I
challenge you to write a diagram-generator program that can go the
other way, unless you code up the x-y coordinates of every block,
arrow and bend. Or are willing to accept an almost unreadable mess.
And why would you want to go to that trouble when you can simply
position a block by dragging it?! Text was fine when all we had was
teletypes, but we now have screens, touch screens, teraflops of
computing power, etc., etc.

Sorry, Ralf and others - I just don't see the big appeal of text!!!

> Or you take this as an execution plan: an interpreter can go over
> these mappings and pull processes from a container to schedule them
> for execution. That´s what I´m working on right now.

Isn't that what JavaFBP and C#FBP do already? John, what am I
missing?

>
> The results are promising. A prototype can execute such flows in three
> modes: sync, async, and parallel.

Not sure what the distinction is... unless it's green vs. red
processes...?

> Also it can automatically log trace messages so you can later on check
> how the execution went.

Yes,JavaFBP and C#FBP do that too...

> And it can do slow-motion execution so you actually can watch the
> program during execution.

That's neat - and could easily be added - just slow down all the sends
and receives, and add some displays...

> Last but not least I´m planning to stop the execution for changing the
> flow and/or the processes at runtime.
> That would make live evolution of flows possible.

IMO that one's dangerous, but there are some special cases which have
turned out to be useful... See http://www.jpaulmorrison.com/fbp/compos.shtml
.
>
<snip>

>
>an IDE for flows should have three panes:
>
> -a text pane for editing some text based flow description like above
> -a graph pane which is updated automatically showing a graphical
> representation of the textual flow description

See comments above...

> -a tree pane also updated automatically showing the same flow as a
> hierarchy of nodes which can be rearranged. This is for easy
> refactoring of flows.

If by hierarchy you mean the subnet hierarchy generated by stepwise
decomposition, DrawFBP can walk that tree, and could easily be
enhanced to show the whole hierarchy on a single page.

>
> Hope I´ll get around building such an IDE later this year.

Start with DrawFBP, and enhance it :-) It's all open source!

[Tom Young]

On Tue, Sep 6, 2011 at 9:38 AM, Paul Morrison <paul.m...@rogers.com> wrote:

, DFDM, used the
following notation:

'data.fil'->OPT Reader(THFILERD) OUT -> IN Selector(THSPLIT) MATCH -
> ...,
Selector NOMATCH -> ...

This is very close to  basic DFD syntax which would look something like:

"data.fil"->(Reader THFILERD)OUT -> IN( Selector THSPLIT)MATCH - > ...

--
"...the shell syntax required to initiate a coprocess and connect its input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

[John Cowan]

Paul Morrison scripsit:

> I don't see why people keep talking about turning text into diagrams.
> It's so much easier to go the other way.

It's a matter of taste: some of us prefer to write things down (textually)
and present them to others as pictures.

> > Or you take this as an execution plan: an interpreter can go over
> > these mappings and pull processes from a container to schedule them
> > for execution. That�s what I�m working on right now.
>
> Isn't that what JavaFBP and C#FBP do already? John, what am I
> missing?

The point is that an FBP mini-language can be read two ways: as a description
of the flow graph, or as a set of instructions to a flow graph interpreter.
In some cases, it is possible to replace the interpreter with a compiler:
the Faust comnpiler renders the whole graph as a single-threaded C++
program, because Faust is synchronous.

--
Do I contradict myself? John Cowan
Very well then, I contradict myself. co...@ccil.org
I am large, I contain multitudes. http://www.ccil.org/~cowan
--Walt Whitman, Leaves of Grass

[Aristophanes]

On Sep 6, 8:46 pm, Paul Morrison <paul.morri...@rogers.com> wrote:
> On Sep 6, 6:15 pm, Ralf Westphal <ralf...@googlemail.com> wrote:
>
> > From this you can build a graph, if you like, that you can visualize.
>
> I don't see why people keep talking about turning text into diagrams.
> It's so much easier to go the other way.  The text has less info than
> the diagram, so it makes sense to work with the diagrams.  Of course,
> you have to have an FBP-friendly diagramming tool, which is what I am
> trying to do with DrawFBP.
>
> Consider the diagram in http:www.jpaulmorrison.com/fbp/image010.jpg-
> this is about 1/3 of a long-running batch program, which was running
> every day for at least 35 years (and may still be)...    You certainly
> can generate a text-type network description from this, but I
> challenge you to write a diagram-generator program that can go the
> other way, unless you code up the x-y coordinates of every block,
> arrow and bend.  Or are willing to accept an almost unreadable mess.
> And why would you want to go to that trouble when you can simply
> position a block by dragging it?!  Text was fine when all we had was
> teletypes, but we now have screens, touch screens, teraflops of
> computing power, etc., etc.

Bad link. It should be http://www.jpaulmorrison.com/fbp/image010.jpg

It's not all that hard to generate good pictures from a text network
description. I'm not a graphics layout guy, but once upon a time I
had people working for me who did automatic layout that was a lot
more complicated than that. I wouldn't be surprised if there are
some good open source layout programs out there.

Being able to move back and forth between text description and visual
description is a good thing. Some people are text oriented, some are
visual oriented.

[Henri Bergius]

Hi,

On Tue, Sep 6, 2011 at 4:38 PM, Paul Morrison <paul.m...@rogers.com> wrote:
> Exactly, Ralf.  You can take a really simple notation - linear or
> tables - and generate the "connect" calls.  The only FBP
> implementation by IBM that ever got on the market, DFDM, used the
> following notation:
>
> 'data.fil'->OPT Reader(THFILERD) OUT -> IN Selector(THSPLIT) MATCH -

I like it. Very concise description of a FBP graph. Last night I wrote
a quick-and-dirty parser for this format, and now NoFlo can run it.
For example:

https://github.com/bergie/noflo/blob/master/examples/linecount/count.fbp

Can be run with:

$ noflo examples/linecount/count.fbp

Note that I "standardized" on the .fbp file suffix for this language.
Is that OK?

There is also a bit of documentation on the format in the NoFlo README:
https://github.com/bergie/noflo#readme

[Henri Bergius]

On Tue, Sep 6, 2011 at 10:55 PM, Dan <dpol...@gmail.com> wrote:
> By the way: who does the calls? A special parent component? A
> component does not call anything and if it does when it calls and
> what?

This is the graph definition API that NoFlo has. It is used internally
when loading graphs from JSON or FBP files, but Node.js developers can
also use it to "embed" flow-based parts into their traditional
JavaScript applications.

What this code does is building a FBP graph, that can then be run by
turning it to a network and sending it some initial data. The graph
(and the components inside it) will not have any interfaces with the
external world, except by the means of input and output ports.

I think having a reasonably simple API that allows embedding FBP into
regular programs is an important step for adoption of the paradigm.
This way users can turn to FBP when they have tricky concurrency or
flow control problems to solve without having to reimplement their
whole application first. Later they can then expand the use of FBP,
until optimally the whole application is flowing.

> Dan

/Henri

[Henri Bergius]

Hi,

On Wed, Sep 7, 2011 at 4:46 AM, Paul Morrison <paul.m...@rogers.com> wrote:
> I don't see why people keep talking about turning text into diagrams.
> It's so much easier to go the other way.  The text has less info than
> the diagram, so it makes sense to work with the diagrams.  Of course,
> you have to have an FBP-friendly diagramming tool, which is what I am
> trying to do with DrawFBP.

Agreed that a graphical tool is much clearer for making sense of
complicated flows. And depending how the tool is done, it may even be
a more efficient way of defining them than text. But lacking these
tools for now means we should also focus on a nice textual flow
format, which the syntax you described earlier gets very close to.

And even if we had a nice graphical editor that everybody could use,
there would still be situations where text is just more convenient. So
I think a good editor would allow you to seamlessly jump between the
graphical representation and the textual representation.

One aspect that hasn't been mentioned yet is that a graphical FBP tool
might be just the key needed for making tablets useful for developers.
Instead of using a mouse you'd just touch and drag the components you
want to connect or rearrange, and you could easily pinch-zoom to see
details of a subgraph. This is why I'm making the editor in NoFlo
web-based, as then it would work also on all these iPads and Androids.

> Consider the diagram in http:www.jpaulmorrison.com/fbp/image010.jpg -
> this is about 1/3 of a long-running batch program, which was running
> every day for at least 35 years (and may still be)...    You certainly
> can generate a text-type network description from this, but I
> challenge you to write a diagram-generator program that can go the
> other way, unless you code up the x-y coordinates of every block,
> arrow and bend.

There are some algorithms for automatically positioning graphs. For
example, GraphViz seems to do a fine job of visualizing my existing
FBP graphs.

Other than that, we also need a clear and standardized text format for
graphs for version control and such.

>> Last but not least I´m planning to stop the execution for changing the
>> flow and/or the processes at runtime.

> IMO that one's dangerous, but there are some special cases which have
> turned out to be useful...  See http://www.jpaulmorrison.com/fbp/compos.shtml

NoFlo also allows disconnecting ports, and adding or removing nodes at
runtime. But for now you either have to use the NoFlo interactive
shell for this, or start NoFlo together with its web-based API server.

> Start with DrawFBP, and enhance it :-)  It's all open source!

Unfortunately it doesn't run on Linux, or the tablets I'm hoping to target :-(

[Ralf Westphal]

On 7 Sep., 01:09, John Cowan <co...@mercury.ccil.org> wrote:
> Ralf Westphal scripsit:
>
> > You only need to relate "ports" to each other. Which port of a process
> > is connected to which other port?
>
> You also have to name processes, because a process may appear more than
> once in a pipeline or flow graph.

Processes are named:

*.Run, ParseCommandLine.Args
ParseCommandLine.Filename, Read.Filename
...

Their names are *, ParseCommandLine, Read, ...

The default is, they are singletons. But if you want multipe copies
you can add a label to tell them apart, e.g.

A.Output, B#1.Input
B#1.Output, C.Input
C.Output, B#foo.Input
...

[Ralf Westphal]

On 7 Sep., 03:29, mark taylor <the-mark-tay...@blueyonder.co.uk>
wrote:

> So my language would have two "flavours"... one at the component
> +streams level, and a very simple procedural language for building
> components.

I don´t think yet another 3GL would entice people to use FBP.
Also yet another 3GL would likely be behind existing languages.
Plus it would cost much effort to just duplicate what other languages
already offer which rather should go into evolving the true FBP
aspects.

If need be for a 3GL "scripting language" then why not integrate
Python or JavaScript and provide a binding to FBP through libs?

[Ralf Westphal]

On 7 Sep., 03:46, Paul Morrison <paul.morri...@rogers.com> wrote:
> On Sep 6, 6:15 pm, Ralf Westphal <ralf...@googlemail.com> wrote:
>
> > From this you can build a graph, if you like, that you can visualize.
>
> I don't see why people keep talking about turning text into diagrams.
> It's so much easier to go the other way.  The text has less info than
> the diagram, so it makes sense to work with the diagrams.  Of course,
> you have to have an FBP-friendly diagramming tool, which is what I am
> trying to do with DrawFBP.
>
> Consider the diagram in http:www.jpaulmorrison.com/fbp/image010.jpg-
> this is about 1/3 of a long-running batch program, which was running
> every day for at least 35 years (and may still be)...    You certainly
> can generate a text-type network description from this, but I

Firstly I´d never design such a FBP network on just one level. So
writing text to describe it is not necessary. Sure so much text would
be hard to read. But with smaller networks on different levels of
abstraction, text is a quite readable option.

Secondly text is often much faster to enter than drawing pretty
pictures. So if I can type and at the same time the diagram is
updated, why shouldn´t I use text?

Thirdly it´s a matter of personal taste. Some people like console UIs
more than colorful windows :-) Some people like text more than a
designer.

> challenge you to write a diagram-generator program that can go the
> other way, unless you code up the x-y coordinates of every block,

Why not go for auto-layout? If you do stratified design each network
is small enough to be layouted automatically and be easy to read.

> > Or you take this as an execution plan: an interpreter can go over
> > these mappings and pull processes from a container to schedule them
> > for execution. That´s what I´m working on right now.
>
> Isn't that what JavaFBP and C#FBP do already?  John, what am I
> missing?

I thought all the processes used their own threads.
And if not, is there a guarantee messages are processed sequentially
by each process?

[John Cowan]

Ralf Westphal scripsit:

> > > Or you take this as an execution plan: an interpreter can go over
> > > these mappings and pull processes from a container to schedule
> > > them for execution. That�s what I�m working on right now.
> >
> > Isn't that what JavaFBP and C#FBP do already? �John, what am I
> > missing?
>
> I thought all the processes used their own threads.

Yes (the simplest possible execution plan). When writing JavaFBP, I
considered checking for a "last process" (a unique process that does
not output to any other process) and running it in the main thread, but
in the end decided that the extra work was not worth it. This is the
pipeline equivalent of tail-recursion optimization.

If it's important to reduce the number of threads, any number of
optimizations can be undertaken; as I mentioned, the Faust compiler only
uses the main thread. Threads are cheap nowadays, however.

--
John Cowan http://ccil.org/~cowan co...@ccil.org
[T]here is a Darwinian explanation for the refusal to accept Darwin.
Given the very pessimistic conclusions about moral purpose to which his
theory drives us, and given the importance of a sense of moral purpose
in helping us cope with life, a refusal to believe Darwin's theory may
have important survival value. --Ian Johnston

[Aristophanes]

Below is a sample network description file. Here are some points
about it that I want to make. The file has four sections, not
counting comments. They are:

(1) Sources:
This is an OS dependent section (might have platform sections but that
is for another time) that specify where things can be found.

(2) Options:
This list available options and their default values:

(3) Processors:
Processors are the acting components - the boxes in the flow charts.
In my view it is essential, and I repeat, essential that every
processor, i.e., every box in the flow chart, have its own label. For
a processor we need three kinds of information - the component type,
its list of inports, and its list of outports. (note - in my work
ports are implicit; I actually use typed connectors).

(4) Connections.
My view here is that each connection gets its own line. This
simplifies life for tools. The function of the network description
file is to provide a reference point from which both builds and visual
layout can be generated.

I like the format below but I could live with an xml type format if I
had to (yuch). A useful feature is to always permit parentheses
around a processor name. When a processor takes options just put
parentheses around the whole mess.

# Illustrative network description file.
#
# The program is a version of the telegraph problem.
# The main flow path is:
#
# telegraph =>getfile =>getline
# =>make_words =>(make_line width) =>print
#
# The subsididary paths are:
#
# (a) getfile opens input files and passes them to getline.
# when there are no more input files it sends a message
# to term.
# (b) getline reads the current input file and extracts one
# line at a time. When the current input file is
# exhausted it sends a message to getfile asking for a
# new file. Start and stop codes are included in the
# getline output.
# (c) make_line repackages the words into a new line of
# specified length. The line is sent to the print
# processor. When a stop code appears it sends a
# message to getline asking for a new line.


sources:
default:
end

options:
width: 30
file: stdin
end

processors:
telegraph: default
inputs:
outputs: out
getfile: default
inputs: in
outputs: out,done
getline: default
inputs: in,next
outputs: out,more
make_words: default
inputs: in
outputs: out
make_line: default
inputs: in
outputs: out,next
print: default
inputs: in
outputs:
term: default
inputs: in
outputs:
end

connections:
(telegraph -file).out => getfile.in
getfile.out => getline.in
getfile.done => term.in
getline.out => (make_words width).in
getline.more => getfile.in
make_words.out => print.in
make_words.more => getline.next
make_line.out => print.in
make_line.next => getline.next
end

[Henri Bergius]

On Wed, Sep 7, 2011 at 10:59 AM, Henri Bergius <henri....@iki.fi> wrote:
> I like it. Very concise description of a FBP graph. Last night I wrote
> a quick-and-dirty parser for this format, and now NoFlo can run it.

Here is a bit more complicated example of extracting data from spreadsheets:
https://github.com/bergie/noflo/blob/master/examples/spreadsheet/parse.fbp

To help explaining this I added support for comments. Anything on a
line after a # gets ignored.

[Henri Bergius]

On Fri, Sep 9, 2011 at 3:22 PM, Henri Bergius <henri....@iki.fi> wrote:
> Here is a bit more complicated example of extracting data from spreadsheets:
> https://github.com/bergie/noflo/blob/master/examples/spreadsheet/parse.fbp
>
> To help explaining this I added support for comments. Anything on a
> line after a # gets ignored.

Also, my fork of Docco adds support for generating documentation from
these files:
https://github.com/bergie/docco

You'll need the associated Pygments lexer as well:
https://bitbucket.org/birkenfeld/pygments-main/issue/698/add-lexer-for-fbp-workflows

Looks pretty nice!
http://static.qaiku.com/b7b/ef5/16d/b7bef516dae311e09ada75210ed4998a998a.png

[Paul Morrison]

Hi Henri,

I really liked this example! Being a visual sort of person, I popped
it into my diagramming tool, and the result is at
http://www.jpaulmorrison.com/graphicsstuff/HenriB.png . I noticed a
couple of things:
it is highly linear - almost one-dimensional - I found that
interesting! And also that the names of the processes often match the
parameters (IIPs). This suggests that even more compact shorthands
might be possible.

You will notice that I haven't put in the component names, although
the tool supports this, but at an early design stage that wouldn't be
necessary, and even the parameters wouldn't be needed initially.

For your interest, it took me 10 minutes to draw the picture with just
the process names, plus another 15 to add in the IIP values, and
pretty it up a bit (including adding in one process that I had
missed). I suspect keying in text isn't *that* much faster...?

I was talking to an ex-colleague recently, and he suggested that the
majority of application developers probably prefer the visual
notation, but that the members of this group (at least the ones who
participate in these discussions) are in fact the ones who don't seem
to mind writing text! Perhaps we could hear from some of the people
who haven't weighed in recently (the "lurkers")?


On Sep 9, 8:22 am, Henri Bergius <henri.berg...@iki.fi> wrote:

> On Wed, Sep 7, 2011 at 10:59 AM, Henri Bergius <henri.berg...@iki.fi> wrote:
> > I like it. Very concise description of a FBP graph. Last night I wrote
> > a quick-and-dirty parser for this format, and now NoFlo can run it.
>

> Here is a bit more complicated example of extracting data from spreadsheets:https://github.com/bergie/noflo/blob/master/examples/spreadsheet/pars...

>
> To help explaining this I added support for comments. Anything on a
> line after a # gets ignored.
>
> --
> Henri Bergius
> Motorcycle Adventures and Free Softwarehttp://bergie.iki.fi/
>

> Jabber: henri.berg...@gmail.com
> Microblogs: @bergie

[John Cowan]

Paul Morrison scripsit:

> For your interest, it took me 10 minutes to draw the picture with just
> the process names, plus another 15 to add in the IIP values, and
> pretty it up a bit (including adding in one process that I had
> missed). I suspect keying in text isn't *that* much faster...?

25 minutes? I should hope I could do better than that.

> I was talking to an ex-colleague recently, and he suggested that the
> majority of application developers probably prefer the visual
> notation, but that the members of this group (at least the ones who
> participate in these discussions) are in fact the ones who don't seem
> to mind writing text!

Inevitably so, since you have to "not mind writing text" to participate
in the group at all.

--
A rabbi whose congregation doesn't want John Cowan
to drive him out of town isn't a rabbi, http://www.ccil.org/~cowan
and a rabbi who lets them do it co...@ccil.org
isn't a man. --Jewish saying

[Paul Morrison]

There is an interesting discussion going on at the Od Geek Registry on
LinkedIn - especially the last few days' posts. I think you can
access it at

http://www.linkedin.com/groups/Best-versus-worst-programming-language-1768367%2ES%2E65617734

(I'll see if this works)

[Paul Morrison]

Of course that should be "Old" - but "Odd" might be appropriate
too :-)

On Oct 10, 9:20 am, Paul Morrison <paul.morri...@rogers.com> wrote:
> There is an interesting discussion going on at the Od Geek Registry on
> LinkedIn - especially the last few days' posts.  I think you can
> access it  at
>

> http://www.linkedin.com/groups/Best-versus-worst-programming-language...