Unix and FBP

[Paul Tarvydas]

====== Paul T =====

For me, understanding Unix was the first step to understanding FBP.

Unix is *very* much closer to FBP, than is Windows, CMS, VAX/VMS, etc.  In fact, one could strip out a bunch of noise from Unix and be left with an implementation of FBP.  I’ve been toying with doing just that with Linux...

I’m trying to remember.  I *think* that the best overview of a Unix-like systems is

http://www.amazon.com/Concurrent-Euclid-Addison-Wesley-computer-science/dp/0201106949/ref=sr_1_1?ie=UTF8&qid=1449861664&sr=8-1&keywords=concurrent+euclid+tunis

It was written for people who already understand computing and don’t need a hand-holding “now press the left-arrow key” type of namby pamby.  297 pages of discussion about Unix-like operating systems, including an assembler listing for a bare O/S.  Holt’s claim to fame is his PhD thesis which brought understanding of deadlocks to the world.

The next best book is

http://www.amazon.com/Software-Tools-Brian-W-Kernighan/dp/020103669X/ref=sr_1_1?ie=UTF8&qid=1449861830&sr=8-1&keywords=software+tools

which shows how to turn FORTRAN(!) into FBP.  I used it (RATFOR) to get Unix-like behaviour on VAX/VMS at Mitel.  (There’s also a version of the book that uses Pascal).

These days, one can run multiple operating systems on the same computer using VirtualBox.

pt

====== Paul M =====

This whole Unix topic is fascinating though, and I'm starting to see why FBP had such appeal to Unix-trained people - and presumably vice versa!

What I don't understand is why so few people picked up on the strong similarities - unless it was just those people who were using Unix in an FBP-like style...  In other words, they (you) had already made the paradigm shift, but in the Unix world!

====== Paul T =====

The Unix story is varied.  When it started to become popular I was in University.  There was no open-source at the time and the source licence for Unix was like $50k.  The only way you could see the source, or even get to use Unix, was to be in a university that had a source licence (cheaper (free?) for universities).

At the same time, I did the O/S course and had to read "Concurrent-Euclid Unix and Tunis" by Ric Holt.  So, I got to understand intimately what was under the hood.

Then Tannenbaum came out with his book and it didn’t really take off. 

Then Torvalds open-sourced linux and that is what started to gain popularity.

At the time, linux was ridiculously hard to install - you had to build it from source code, tweaking parameters to match your own system.  I think I gave it a try and gave up.

At the same time, the competing O/S’es were VAX/VMS and VM/CMS and Windows 3.11, well and OS/2 (which was ridiculous to install/use and promoted by IBM, which people hated as much as Microsoft is hated nowadays).

The simplicity of bolting components together via Unix pipes was essentially unknown except to the small number of grad students who were fortunate enough to have used Unix in grad school. 

I lucked out, without being a grad, because my buddy was a grad student and Small C had just been published in Dr. Dobb’s.  It was written in C.  The only C compilers available at the time were on Unix V6 and V7.  My buddy and I went into his grad lab and typed the whole compiler in by hand into a PDP11 running Unix V7.  At 2am that morning I got to see the first output from the compiler and instantly understood what a compiler “does”.  I decided to devote my 4th year EE thesis to writing a C compiler myself.  My thesis prof was a member of CSRI (Computer Systems Research Institute) and I got to use Unix for that whole year.

(At the end of the year, my prof called me in and grinned at me - he told me that just about every other student in 4th year couldn’t write one page of code without it containing a fatal error, yet I had produced a whole compiler.  Then, he offered me a job in Halifax with the military, and I turned it down.  Sigh.)

I got 7 job offers that year.  I turned most of them down because they didn’t involve Unix.  I accepted Mitel’s offer, even though they used VAX/VMS (1 minute, literally, to spawn a process!!!).

I ported the RATFOR tools to VAX/VMS, so that I could have a more productive environment (still 1 minute to start each process in a pipeline, sigh).

When Linux began to gain traction, it was taken over by various competing distributions (distros).  They competed for market share based on feature sets and windowing and completely lost the original simplicity and light-weightedness of Unix.  So, again, most people never got to see what Unix was about.

What *you* did with FBP was to discover the simplicity of Unix - bolted onto a clunky operating system.

Note that many of the progenitors of Unix came from CSRI at UofT.  Kernighan did his undergrad at UofT.  I rubbed shoulders with Rob Pike at CSRI while I was doing my C compiler (I doubt that he remembers me).  Holt taught me compilers and O/S’es.  CSRI was populated with minds who relished simplicity and worked hard to achieve it.  Henry Spencer (most famous for regex) and several others whose names don’t come to mind at the moment.   I believe that Berklee 4.1 Unix was mostly developed at UofT, then released, without attribution, by Berkley.  [Another floor of the CSRI building housed Marshal McLuhan’s office - where he tore a strip off of John Lennon for being a tool of the establishment].  The UofT Comp Sci department (different than CSRI) most famously developed 3D realism (Alain Fournier) and lots of Lisp A.I. (“Planner”, I think).

I think that I backed into FBP because I already knew the beauty of Unix and that I studied the Software Tools book and because I became one of the few experts in porting the Concurrent Euclid language (which caused me to become expert in multi-processing and concurrency, because it was built-in to the Concurrent Euclid spec).  [Euclid was designed to compete for the US Military standard language, but Ada won out].

So, I knew the beauty of bolting components together because of Unix and because of multi-processing.  I wanted to use processes for everything, but, I couldn’t because Concurrent Euclid’s Hoare Monitors weren’t fast enough to use for building RS232 drivers.  It could only handle 1200 baud, whereas I needed 9600 baud (on a MC6809).

And *that* set me on a quest to figure out how to make processes “cheaper”, so that I could use them to build device drivers.  [Unix device drivers could not use processes, either, and resorted to very hoary ad-hoc C code - what a mess].  I wanted to bolt components together to make device drivers, where processes could *not* be used.

Interesting!  So only you (and your co-workers) and early Unix users understood how simple things could be.  Linux has actually destroyed Unix.  Steam Engine Time came and went in a short number of years.  It’s coming back now because of multiple core programming.

Wow, one could even use a full-screen editor (Vi) on Unix over 300 baud.  [My C compiler was mostly written at home in the evenings over 300 baud].  I remember seeing the first revision of Berkley 4.2 - a grad student was editing in 2D, typing many commands well ahead of the system - what he typed would appear on a blank screen, then seconds later the rest of the screen would fill in, edits already accomplished.

I would love to see a project which strips Linux back to its Unix roots, then bolts visual FBP onto it instead of the Bourne shell…

What does this say about popularizing O/S’es and languages???  Something we need to think about.

The current mentality is that “functional programming” will solve all of the problems in the world - eventually (just like OOP was supposed to :-).

Hmm, looking at Functional Jobs reveals interesting things

https://functionaljobs.com

The “top” functional languages today are Clojure, Haskell (risky, hard), Erlang, (Scala, OCaml).  Clojure is functional Lisp on the JVM.  Erlang has its own VM (Beam) which claims to easily support 60,000 simultaneous processes.

Go was developed at Google and supports the concept of “ports” and pipes.  (Rob Pike was one of the developers of Go).

Hmmmm
 

====== Paul T =====

I was going to mention Unix “cat” but forgot.

Wildly simple - it takes at most one argument. 

In Unix shell, everything is a “file descriptor” (think port).  Every program gets an array of 16 standard file descriptors (ports).  #0 is called stdin, #1 stdout, #2 stderr.

The shell syntax allows these file descriptors to be plumbed together, e.g. the stdout of one program can be joined to the stdin of the next program (think wires, but no feedback allowed).

Every program also gets an array of strings representing the parsed out command line (the shell does the parsing, not the program).  The 0th argument is always the name of the program (in Unix you can name the same program different ways, and the program will change its behaviour depending on it’s invocation name).

If there’s an argument on the command line, cat opens that file and pours its contents into the “stdout” file descriptor.

If there are no arguments, cat accepts input from stdin and pours it to stdout.

You can imagine how small the code is for this, even in C. From the top of my head:

#include <stdio.h>
int main (int argc, char **argv) {
  FILE *fd;
  char buff[1024];
  if (argc == 1) {  // no arguments, except program name
    fd = stdin;
  } else {
    fd = fopen(argv[1], “r”);
  }
  if (fd == NULL) {  // file not found
    exit(1);
  }
  n = fread(fd, sizeof(buff));
  while (n > 0) {
    fwrite(stdout, buff, sizeof(buff));
    n = fread(fd, sizeof(buff));
  }
  exit(0);
}

(give or take some error checking and sending an error message to stderr).

This is small and simple and wildly useful.  Unix doesn’t need a “type” nor “print” command.  If you want to display a file, just cat it.  If you want to paginate the file, “cat" it to the “more” command.  If you want to print it, “cat” it to the “lpr” (line printer) command.

> cat file.txt
> cat file.txt | more
> cat file.txt | lpr

GNU and Linux have corrupted this simplicity.

In Unix V7, you could print off the man page for *every* command in the system on fan-fold paper, and keep it in your briefcase with enough room left over to hold your lunch.  Most commands required only one page to describe their function and all parameters.  Of course, to print a man page on the printer, one would say something like: (where troff is the command to text format a troff text file)

man 0 cat | troff | lpr

And, with the standard library, any program could act like the shell by calling fork(), dup2() and exec(), so you could plumb programs together in C (or in the shell).  My light-weight example “grash” implements an interpreter  in 200 lines of C that plumbs Unix (FBP) components together at run time - only 8 instructions for the interpreter.

https://github.com/guitarvydas/vsh/blob/master/grash/grash.c


Woah!  The source for v7 is browsable on the web…

http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7/usr/src/cmd/cat.c

http://minnie.tuhs.org/cgi-bin/utree.pl?file=V7

pt

[Kenneth Kan]

There's too much information here for me to tackle... but, for those of us who lack first-hand historical context, I'm curious of this statement:

> GNU and Linux have corrupted this simplicity.

By GNU/Linux do you mean basically all descendants of UNIX? *BSD line of OS seems to be as complex as GNU/Linux, at least to a person like me who lives mostly at the application level. Or are they really categorically different?

I'm clearly looking for a history lesson. :)

[Paul Tarvydas]

Do "man cat".

If it shows any command line options (parameters) for cat, other than -u or -, then the author(s) "lost" the idea of Unix.

The main idea of Unix was to bolt components together.  Each component does one thing well.

Synchronous (call/return) software promotes API's which provide parameters to the routines, thereby making the routines very complicated (and bug-full).

The Unix philosophy was, instead of parameterizing things, to bolt components together.  Some command line parameters were required, e.g. pr, which produced printed listings of files, needed to know about columns, page widths, etc.  These options, I think, are like IIP's.

But, if you look at modern-day 'cat', you'll see options that could easily have been broken out into separate components - number the lines, display $ at the end of each line, show tabs, etc, etc.  Each of those actions could have been accomplished with separate components, e.g. a line-numbering component takes stdin and outputs to stdout with a line-number prepended to each line.  Someone began to start worrying about "efficiency" and deviated from the Uinx model.

It appears that there were only about 137 commands in V7 Unix (incl. addenda), but, it could accomplish amazing things.  Grep could filter input based on regex's, sed could regex-replace input and send the edited output to stdout, join acted like a DB join operator, awk was amazingly powerful, etc, etc.

I note that vi is not in the V7 command set, so I guess I first saw that in BSD4.1 (.2?).  V7 had 'ed', a command-line editor, that was wonderfully productive (I wrote a whole C compiler using ed at 300 baud).

BSD4.x also introduced the notion of suspending the foreground process.  ^Z and 'jobs'.

At that point, I went into industry, where no one was using Unix and I lost track of the developments.  I don't know exactly where the shift came in - from components to parameterized apps.  My perception is that the Unix mentality started to lose ground when Linux came out, esp. when it adopted windowing and tried to compete with Windows / Mac.

I owned a B&W 19" Sun box at one point and it still felt Unix-y, despite having a windowing system.

pt

[John Cowan]

Paul Tarvydas scripsit:

> If it shows any command line options (parameters) for cat, other than -u or
> -, then the author(s) "lost" the idea of Unix.

You are grossly overstating the case. The 6th Edition Research Unix
manual had 81 commands in section 1. 35, or 43%, of those had options.
If what you say were true, Unix would have already lost the plot by 1975.

> The main idea of Unix was to bolt components together. Each component does
> one thing well.

Or sometimes a few related things well.

> The Unix philosophy was, instead of parameterizing things, to bolt
> components together. Some command line parameters were required, e.g. pr,
> which produced printed listings of files, needed to know about columns,
> page widths, etc. These options, I think, are like IIP's.

Consider stty. Do you think it should have been divided into a zillion
smaller commands? Rm -r and -f go back at least to 6th Edition, though
there was a separate command for -i, and -r could have been emulated
by find. Nm enables built-in sorting and reverse sorting using options.
Roff (the precedessor of nroff/troff) had options to regulate which
pages were output, certainly not necessary if you wrote a filter to pass
through only certain pages. I could go on.

--
John Cowan http://www.ccil.org/~cowan co...@ccil.org
When I wrote it I was more than a little febrile with foodpoisoning
from an antique carrot that I foolishly ate out of an illjudged faith
in the benignancy of vegetables. --And Rosta

[Paul Tarvydas]

>
>
...

>You are grossly overstating the case.

Correct.

Hyperbole is often necessary when explaining how to think in a new paradigm.

I gave 'pr' as an example of something that must be parameterized, you
gave 'stty' as an example of something that must be parameterized.

My point is that, when learning to think in the paradigm of composition
of components, one should eschew the use of parameters, until there is
no way out other than to use a parameter.

The obfuscators of 'cat' fell into a trap. -A, -b, -e, -E, -n, -s, -t
and -T should all have been done as separate components. The
programmers allowed old habits to creep in, and, I would guess, thought
that it would be "more efficient" to parameterize the code for cat
rather than to "duplicate" the code for specific purposes.

After attempting, hard, to eschew parameters for a while, one learns
where the local minima are and what the trade-offs are.

I am, also, on record as saying that there is a lower limit on how
"small" an FBP component should be, e.g. +, -, *, / should not be
components. 'Expr' gets this right.

My prescription to those who grew up when simple Unix was long-gone and
have known only Linux, Windows and MacOS is to try to build things
without parameters. Then, see what happens.

pt

[Alfredo Sistema]

This is an interesting proposition, I might be digressing but making scripts ( graphs) that are versions of cat with certain flag set is a way to fix this. In many distributions I've found this to be the case for certain commands.
The complexity of using a unix system to make programs comes from debugging and the artistry needed to orchestrate processes with multiple inputs and outputs. The obsession with minimalistic tools facilitates the construction of serial pipelines, which are largely unattractive ( the average programmer sees it as simply nested functions/procedures).
Isn't the main innovation of FBP treating multiple inputs and outputs as first class citizens?

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Tarvydas]

On 15-12-22 04:29 PM, Alfredo Sistema wrote:
>
> ...Isn't the main innovation of FBP treating multiple inputs and

> outputs as first class citizens?
>

Unix already supports multiple inputs and outputs (IIRC, 16 fd's for a
process in early Unix).

It is the shell 'sh' which constrained one to using only one input
(stdin) and two outputs (stdout, stderr) per command.

Multiple inputs and outputs are available at the C level.

This is only a problem of syntax.

The cult of text couldn't figure out how to usefully show more inputs
and outputs using only text.

IMO, FBP's innovation is that of showing that diagrams can act as a
useful syntax for multiple inputs and outputs.

pt

[Paul Morrison]

I agree - I find diagrams like http://www.jpaulmorrison.com/fbp/image010.jpg  much more understandable that a textual representation of the same diagram, although the latter can be very easily generated from the diagram.

As I said in my book, we had one programmer on a project who built a 200-node network (using multi-level subnets) without ever drawing a picture, but I remain convinced that the majority of programmers think visually, and the latter mode suits them much better!

pt

[Tom Young]

On Tue, Dec 22, 2015 at 4:42 PM, Paul Tarvydas <paulta...@gmail.com> wrote:
>

> It is the shell 'sh' which constrained one to using only one input (stdin)
> and two outputs (stdout, stderr) per command.

A common misconception.
The following command:
sh -c "echo abc" 3> /tmp/a 4> /tmp/b 5>/tmp/c 6>/tmp/d
writes abc on STDOUT and creates four empty files.
The groundbreaking Unix advance -- creating standard streams (STDIN,
STDOUT, and STDERR) --- did not preclude the use of additional
streams defined by file descriptors greater than 3. It just requires
a bit more effort to use them.

>
> Multiple inputs and outputs are available at the C level.
>
>

> pt
>s://groups.google.com/d/optout.
"...the shell syntax required to initiate a coprocess and connect its
input and output to other processes is quite contorted..."
W. Richard Stevens [Advanced Programming in the Unix Environment, p441]

[John Cowan]

Tom Young scripsit:

> The groundbreaking Unix advance -- creating standard streams (STDIN,
> STDOUT, and STDERR) --- did not preclude the use of additional
> streams defined by file descriptors greater than 3. It just requires
> a bit more effort to use them.

Indeed, and consequently few components were written to exploit them.
That can be blamed on the design of the shell, which treats neither
processes nor ports as first-class values (indeed, the only first-class
values in the shell are strings). In scsh, the Scheme schell, where
shell-like behavior is embedded in the Scheme programming language,
it is straightforward to construct arbitrary networks.

This is quite independent of text vs. drawings. Drawings are more
intuitive than text up to a point, but eventually they become limiting
as the program grows more complex, with lines crossing one another in
unintelligible ways. There's a reason that writing flowcharts and then
constructing programs from them has basically been abandoned as a
programming technique.

--
John Cowan http://www.ccil.org/~cowan co...@ccil.org

Almost all theorems are true, but almost all proofs have bugs.
--Paul Pedersen

[Paul Tarvydas]

On 15-12-23 02:30 PM, John Cowan wrote:
> ...

> This is quite independent of text vs. drawings. Drawings are more
> intuitive than text up to a point, but eventually they become limiting
> as the program grows more complex, with lines crossing one another in
> unintelligible ways. There's a reason that writing flowcharts and then
> constructing programs from them has basically been abandoned as a
> programming technique.

I very much disagree with this characterization of diagrams. I have
routinely drawn diagrams with 300+ components on them and have had no
problem keeping them understandable (in one case, we built an IDE with
20k components (although many of them were "too small" in retrospect)).

The secrets are:

- Hierarchy. Compose diagrams in a hierarchical manner, each level
being easy-to-understand. The original Harel paper on StateCharts is an
interesting inspiration (although he does get some things wrong).

- Concurrency. Each box on the diagram is concurrent, hence,
independent and encapsulated (as the OO people would like to claim).

- Limit oneself to diagrammatic items that can clearly be compiled to
executable code (except comments).

- Allow diagrams to be annotated with text snippets (hybrid diagrams).
Certain things (e.g. 'expr') are better expressed as text, but
architecture is better expressed as diagrams.

Yes, flowcharts of way-back-when failed, as did full-blown UML. But,
they didn't obey the above secret rules...

pt

[Paul Morrison]

Paul, I totally agree!  Our diagrams were very understandable.  As you say, multiple levels, simplicity, concurrency, annotations, are key!  I didn't like the UML diagramming tool, or any of the others I came across, so I wrote my own!

[Ged Byrne]

I think this is where the real advantage of FBP.

What I would like to develop is an approach for FBPing existing code, so the complex structures can be extracted into a flow graph for long term maintenance and future development.

Software product line (SPL) development has the concept of mining software for existing assets: http://www.sei.cmu.edu/productlines/frame_report/miningEAs.htm

By this approach a high quality production quality component sets can be quickly compiled.

Regards,


Ged

[Paul Morrison]

Given the strong similarities that have started to emerge recently between ("classical") Unix and "classical" FBP, I thought I would post two presentations from 2006 on Unix, kindly contributed by No. 1 son and Unix guru, Joe Morrison:

http://www.jpaulmorrison.com/fbp/intro-to-unix-part1.ppt   and 
http://www.jpaulmorrison.com/fbp/intro-to-unix-part2.ppt

IMO these presentations not only clearly describe the history of, and philosophy behind, Unix, but suggest possible directions for FBP in the future.

Enjoy!

[Alfredo Sistema]

Developing strategies for integrating fbp into existing software is  good start, that's my main motivation for making a javascript library.
Looking at the limitations of sh as a user interface for the unix os can be eye opening for us.
A certain level of ugly parametrization for graphs ( shell scripts ) can be achieved by setting environment variables and other 'unclean' techniques. In my DSL design I considered this situation by making graphs parametrizable.