NoFlo: distributing components via NPM

[Henri Bergius]

Hi,

This week I've been working on a new feature for NoFlo: allowing
components (and graphs) to be distributed via Node's NPM package
manager. The tracking issue for this is
https://github.com/bergie/noflo/issues/49

This makes sense, since most FBP components are anyway meant to be
reusable. Now developers can simply publish their own components
independently without having to send pull requests to the main NoFlo
repository. The same mechanism can also be used when publishing more
traditional Node.js libraries to include the NoFlo components
utilizing that library.

Here is the documentation for publishing NPM packages:
https://npmjs.org/doc/developers.html

To publish NoFlo components, you simply need to amend the package.json
file with the NoFlo metadata to register the components you're
providing, like this:

"noflo": {
"components": {
"ParseYaml": "./components/ParseYaml.coffee",
"ToYaml": "./components/ToYaml.coffee"
}
},

If you're publishing FBP graphs in either .json or .fbp format, just
provide them similarly with the "graphs" key. Components can be either
in CoffeeScript or JavaScript, the NoFlo ComponentLoader will handle
the coffee compilation transparently.

Here are some examples:
https://github.com/bergie/noflo-yaml
https://github.com/bergie/noflo-webserver
https://github.com/bergie/noflo-mq

In addition to the components and the package.json, the repository
ought to also include unit tests for each component, and would
preferably be using Travis for continuous integration. This way every
change to the repository will cause the components to be tested
against relevant Node.js (and NoFlo) versions. For example:
https://travis-ci.org/bergie/noflo-yaml/builds/3112024

I'm still in process of migrating some components away from the NoFlo
repository. Once all of them are out I'll publish NoFlo 0.2 with these
changes. If you're using NoFlo in your applications, the main change
you need to make is to add the appropriate component-providing
packages to your list of dependencies and install them.

An additional benefit for this new structure is that now the NoFlo
ComponentLoader can provide a list of installed components, which is
also available on the noflo command-line. For example:

$ noflo list ../noflo-webserver
GroupByPacket
(/home/bergie/Projects/noflo/src/components/GroupByPacket.coffee)
Inports: in
Outports: out
...
Server (/home/bergie/Projects/noflo-webserver/components/Server.coffee)
This component receives a port and host, and initializes a HTTP
server for that combination. It sends out a request/response pairfor
each HTTP request it receives
Inports: listen
Outports: request

This same information will also be exposed to the web-based NoFlo GUI.

--
Henri Bergius
Motorcycle Adventures and Free Software
http://bergie.iki.fi/

Jabber: henri....@gmail.com
Microblogs: @bergie

[Paul Morrison]

Hi Henri,

Do you need any changes made to the NoFlo generator in DrawFBP?

Regards,

Paul

http://jpaulmorrison.blogspot.ca/

[Henri Bergius]

Hi,

On Fri, Nov 9, 2012 at 4:09 PM, Paul Morrison <paul.m...@rogers.com> wrote:
> Do you need any changes made to the NoFlo generator in DrawFBP?

With the new structure it would be best to refer to components just by
their names instead of full paths, i.e.
"ReadFile" instead of "/home/bergie/Projects/noflo/components/ReadFile.js"

The trickier part is that now the components can be in multiple
places. We could consider having some sort of generated "component
registry" file that you could read to get a full list of them. But for
now, just using the name instead of file path can suffice.

> Paul

/Henri

[Paul Morrison]

I can send you an experimental version over the week-end. I assume I
drop the ".js" as well...?

On 09/11/2012 10:15 AM, Henri Bergius wrote:
> Hi,
>
> On Fri, Nov 9, 2012 at 4:09 PM, Paul Morrison <paul.m...@rogers.com> wrote:
>> Do you need any changes made to the NoFlo generator in DrawFBP?
> With the new structure it would be best to refer to components just by
> their names instead of full paths, i.e.
> "ReadFile" instead of "/home/bergie/Projects/noflo/components/ReadFile.js"
>
> The trickier part is that now the components can be in multiple
> places. We could consider having some sort of generated "component
> registry" file that you could read to get a full list of them. But for
> now, just using the name instead of file path can suffice.
>
>> Paul
> /Henri
>


--

http://jpaulmorrison.blogspot.ca/

[Henri Bergius]

On Fri, Nov 9, 2012 at 4:24 PM, Paul Morrison <paul.m...@rogers.com> wrote:
> I can send you an experimental version over the week-end.

That'd be great! I just published the new NoFlo 0.2 that supports this.

> I assume I drop the ".js" as well...?

Correct, the new ComponentLoader will support both .coffee and .js
files (which is something the filepicker probably also has to be aware
of)

[Kenneth Kan]

Henri,

What about the "core" set of components that every project needs like Group and others? Will there be a "noflo-core"?

Either way, what is considered core and what is not? I have some utility components that do not really belong to a category but I may still find necessary while others may think otherwise.

- Ken

[Henri Bergius]

Hi,

On Fri, Nov 9, 2012 at 9:54 PM, Kenneth Kan <ken...@gmail.com> wrote:
> What about the "core" set of components that every project needs like Group
> and others? Will there be a "noflo-core"?

Right now they're still in the main noflo package, but I guess
something like noflo-core / noflo-base would make sense in the long
run.

I'd even go a bit further and move the file system handling components
to their own noflo-fs. Easier to keep track of :-)

> Either way, what is considered core and what is not? I have some utility
> components that do not really belong to a category but I may still find
> necessary while others may think otherwise.

Hard question, but I think these two guidelines would be a start:

* Core components shouldn't have dependencies outside of NoFlo and
Node.js itself
* Core components should be things you could see yourself using in
70-80% of NoFlo projects

> - Ken

/Henri

[Kenneth Kan]

Right now they're still in the main noflo package, but I guess
something like noflo-core / noflo-base would make sense in the long
run.
I'd even go a bit further and move the file system handling components
to their own noflo-fs. Easier to keep track of :-)

That would be so cool! I like the modularity of FBP in general already. Sometimes I think of some additions to these "core" components but don't think they apply to everyone so I modify them and use them privately. I would be cool if I can fork any part of noflo and use a modified derivative of it.

* Core components shouldn't have dependencies outside of NoFlo and
Node.js itself
* Core components should be things you could see yourself using in
70-80% of NoFlo projects

Hmmm... yea, except the second point is hard to define. Like you said above though, if we make every part, even the filesystem part pluggable, it should be easy for anyone to switch out the core part (or any other part for that matter) and use her own. Sounds like a fantasy though...

[Henri Bergius]

Hi,

On Sat, Nov 17, 2012 at 2:16 AM, Kenneth Kan <ken...@gmail.com> wrote:
> That would be so cool! I like the modularity of FBP in general already.
> Sometimes I think of some additions to these "core" components but don't
> think they apply to everyone so I modify them and use them privately. I
> would be cool if I can fork any part of noflo and use a modified derivative
> of it.

Yep, that is definitely one benefit. We should figure out a nice
mechanism for "subclassing" components as well.

BTW, yesterday I split the UI parts of NoFlo to a separate project.
You can watch the progress in https://github.com/bergie/noflo-ui

[Kenneth Kan]

Yep, that is definitely one benefit. We should figure out a nice 
mechanism for "subclassing" components as well. 

Hmmm... Is subclassing a good thing in FBP? I thought that we extend functionality of a "program" by means of adding different types (or more of the same) of components. If a component lacks a functionality that cannot or should not be made into its own component, should we improve on that component instead? Coming from OO I can see the value of subclassing but I don't see how it fits in FBP.

[Paul Morrison]

Hi Henri,

Do you have examples in mind of "subclassing"?

Regards,

Paul

[Paul Tarvydas]

On 12-11-19 09:33 PM, Kenneth Kan wrote:

Yep, that is definitely one benefit. We should figure out a nice 
mechanism for "subclassing" components as well. 

Hmmm... Is subclassing a good thing in FBP? I thought that we extend functionality of a "program" by means of adding different types (or more of the same) of components. If a component lacks a functionality that cannot or should not be made into its own component, should we improve on that component instead? Coming from OO I can see the value of subclassing but I don't see how it fits in FBP.

I agree that subclassing (inheritance) and FBP don't mix.

FBP uses "composition" to create networks of components.

Inheritance is just fancy cut-and-paste of code and creates hidden dependency chains.

A non-hierarchical component (a leaf node) contains some kind of code written in some kind of language.  It would be reasonable to use a language that supports inheritance at that level (leaf nodes).

In practice (over a decade of using fbp-like ideas), I have never needed inheritance or subclassing at the FBP level.  When I want to "reuse" a particular combination of components, I wrap them into new single component and reuse / multiply-instantiate that.

pt

[Dan]

Inheritance is composition.
The only problem is that it is implicit and hidden. In the end inheritance is just composition. This is why I do not understand some OOP programmers that advocate that "multiple inheritance" is bad. The bad part is not multiple inheritance (i.e. composition) but the fact it is hidden (not explicit) and it has side effects. One of them is the fact you can not get rid of it when sub-classing if you want. The continuous relation ship between the parent and the child creates brittle software. When you change the parent the child will change too. This is related to the fact the system is not keeping the values of classes at different times.
The problem with multiple inheritance in OOP imperative languages is interfacing.
There are also programmers that think that a class and an instance are different animals as the data and the functions are also two different animals. They are not.

Regards,
Dan

[David Barbour]

In general, and especially in mainstream programming languages, inheritance is NOT compositional.

A compositional system has operands and composition operators, the property of algebraic closure (every composite is suitable as an operand for further composition), and compositional properties. Compositional properties are a set of properties that can be computed just from the same properties of each operand and the composition operator (i.e. exists F . forall X,*,Y. P(X*Y) = F(P(X),'*',P(Y))).

There are a few approaches to inheritance that are compositional or nearly so, most notably use of traits [1]. But even use of traits is only suitable at a small scale, since the set of composition properties is weak.

Inheritance and FBP can be mixed. If I were to tackle this, I'd consider use of traits to represent FBP subnets. FBP composition would allow subnets to grow into larger networks, while traits composition would allow overrides and tweaks within a subnet. Traits would serve as an alternative to use of functions to parameterize subnets and abstract components. Whether this is "better" than use of functional abstraction of FBP subnets would be an open question.

[1] http://scg.unibe.ch/archive/papers/Scha03aTraits.pdf

--
bringing s-words to a pen fight

[Dan]

dmbarbour: "In general, and especially in mainstream programming languages, inheritance is NOT compositional. A compositional system has operands and composition operators, the property of algebraic closure (every composite is suitable as an operand for further composition), and compositional properties."

The lack of composition operators is an issue but it does not mean the inheritance is not composition. Inheritance is composition even if the programming language has or not has composition operators. The problem is that it is implicit and not flexible. Check for instance the multiple inheritance in C++. It is implemented by composition. There is no other way. The result of inheritance is composition by definition. When we say "composition" we do not refer to the act of inheriting (that is a flow process by the way :) but to the fact of possessing something.
The act of inheritance can be done by duplication (replication) or by sharing. In C++ when a class inherits from another is by sharing. This creates brittle software. Sometimes you need to share, sometimes you don't. Unfortunately, sharing runtime objects is treated differently from sharing information by inheriting it in classes. Anyhow I do not make a big difference between classes and instances as I do not make a big difference between data and functions. They are different aspects of the same thing. In other words a "thing" can have both aspects: model (class) and instance or data and function.
So, in the main stream programming languages inheritance is not compositional (i.e. lack of composition operators) but it is implemented by composition (that you can not control) and it is composition.

Regards,
Dan

[David Barbour]

> When we say "composition" we [...] refer to [...] the fact of possessing something.

That's a very different use of "composition" than was meant by its use for FBP or when people assert composition is desirable.

Are you sure you wish to engage in such equivocation? It only costs you your credibility.

[Dan]

There are two different terms: Aggregation and Composition that are not quite synonyms but in some contexts they refer to the same thing.
When we say "function composition" most of the people refer to how functions are chained together i.e. function composition is the application of one function to the results of another. When we say aggregation one might refer to how several elements are grouped together. This builds the relation "has a". A parent component is defined by how the subcomponents are interconnected and this automatically means that we define what subcomponents the parent component has (what it aggregates). Both aggregation and composition are aspects of the same thing i.e. component (functional or not).

dmbarbour: "That's a very different use of "composition" than was meant by its use for FBP or when people assert composition is desirable."
So what's different and who defined what was meant for FBP? Are we refering to the plain english word "composition" or to some other definition?

dmbarbour: "Are you sure you wish to engage in such equivocation? It only costs you your credibility."
Can you explain?

Regards,
Dan

[David Barbour]

Aggregation and composition do not have the same meaning, but may coincide if the mechanism for aggregation is compositional. Many operations on collections are compositional, even if the elements are not. (E.g. we can compose "collections of matchbox cars" without composing matchbox cars.) 

Both aggregation and composition are aspects of the same thing i.e. component (functional or not).

No, they aren't. Composition is something might do with components, but is not an aspect of components.  Aggregation is something you might do with collections of components, but is not an aspect of components.

dmbarbour: "Are you sure you wish to engage in such equivocation? It only costs you your credibility."
Can you explain?

Equivocation is a well known fallacy that results from using the same word with different meanings or senses to connect points in an argument. (http://en.wikipedia.org/wiki/Equivocation). Even if a word has more than one plain English meaning, it's equivocation if you use two different meanings. This argument first mentioned "composition" in Paul Tarvydas's comment (FBP uses "composition" to create networks of components). There, composition is used in the same sense as functional composition - chaining FBP networks together into larger networks. You then claimed "Inheritance is composition", and you've elaborated that you're using a different meaning and definition for the word composition. That's a clear example of equivocation. 

 

[Ron Lewis]

Using pseudo code, I think inheritance (aggregation) is:

// define parent class "shape"

class shape

{

    // definition in terms of methods (functions) and variables (state)

}

// define child class box inherits from parent class shape

class box : shape

{

    // definition in terms of methods (functions) and variables (state)

}

// define another child class circle, also inherits from parent class shape

class circle : shape

{

    // definition in terms of methods (functions) and variables (state)

}

Say that shape has no methods and no variables. It is just an empty definition. I can define a method somewhere called draw that will take either box or circle but nothing else because only box and circle inherit from shape:

// function definition. (void means nothing is returned)

void draw( shape variable_shape)

{

     // operations that define the method (function) 

}

==============

Now, I think that composition would not allow me to define the function draw as I did. Maybe I want to express that box has four sides. So, I define a class side:

class side

{

    // methods and variables defining side.

}

Then, class box would be:

class box : shape

{

    side top

    side left

    side right

    side bottom

    // the rest of the methods and variables defining box

}

Now, I don't think that I can write the same kind of draw function that takes both side and box.

H O W E V E R, HOWEVER, HowEver, However, however ... [...fading echos...]

I also see aggregation, composition, and subroutines as being very much alike in that all of them give me the same effect of writing some code once and then referring  to that code in several other places in the program. So, I can refer to code instead of repeating the code.

In my example classes, I think we see some flexibility in that if I want to add a shape such as a cloud, I can simply add it without effecting anything else. But we also see some rigidity in that if I want to change the parent class shape, then I need to be very careful that I do not break box, circle, cloud, or any other shape. I think OOP is all about variable state.

Am I missing anything?

[David Barbour]

People have developed compositional approaches to drawing. E.g. where you compose two diagrams into a larger diagram, potentially with rotation, translation, color, etc.. A box is a diagram composed, according to some function, from four other diagrams. In your case, you could have a generic mechanism of creating a shape from multiple shapes, or by rotating another shape.  

effect of writing some code once and then referring  to that code in several other places

You speak of abstraction, which is an orthogonal issue to composition or aggregation. Even if we lack abstraction, we can still create components and compose them (even if doing so involves much replication or copy-and-paste). Even in a compositional system, we can create abstractions that aren't compositional (missing essential parameters or dependencies). 

(That said, a compositional system does provide some natural boundaries for abstraction.)

Procedures are one way to abstract behaviors along a composition boundary (sequential). It isn't a very rich form of composition, but you can generically glue two procedures together into a larger procedure by saying that one executes after the other.

[Sam Watkins]

inheritance of implementation is an abominable hack,
a weak short-hand for a decent system of composing objects

inheritance of interface is a misnomer for a weak excuse of a type system

the only inheritance you should be thinking about
is what to leave behind for your kids and lady-friends :)

burn your C++ and Java tomes, and use something better

now, I don't want to hear any more about inheritance!

[David Barbour]

You've made up your mind, eh? Inheritance isn't my preferred mechanism, either. But inheritance can be (and has been) done better than in Java and C++. I even find the richer variations (traits, mixins) quite intriguing.

[Sam Watkins]

On Tue, Nov 27, 2012 at 04:31:38PM -0800, David Barbour wrote:
> You've made up your mind, eh? Inheritance isn't my preferred mechanism,
> either. But inheritance can be (and has been) done better than in Java and
> C++. I even find the richer variations (traits, mixins) quite intriguing.

Sure, the trouble is that using "just inheritance" is a distraction
from better ways to create objects

Sam

[Sam Watkins]

people fiddle about with "the perils of multiple inheritance" when they
could be composing objects and forwarding requests more flexibly

[David Barbour]

The real peril of inheritance is sinking time into those inheritance hierarchies. That doesn't even require multiple inheritance. :)

[Paul Morrison]

At the risk of causing a resurgence of religious wars, I wonder if any of the participants in this discussion have read my chapter on FBP vs. OOP - http://www.jpaulmorrison.com/fbp/oops.shtml ?  I didn't change it much for the 2nd edition.  The only thing I might change is the last paragraph, upgrading 5,000,000 to, say, a billion (US billion, that is) to take advantage of the power of modern-day machines.  The cited paragraph referred to a banking system running, at that time, on a 2 Mips machine.

Regards to all,

Paul

[Ged Byrne]

Hi Paul,

In the end Inheritance is a metaphor, a rhetorical device to explain the way in which the implementation is deferred until later.  In your book you talk about "black box" composite components with subnets being packaged as separate load modules:

This approach naturally coordinated the hierarchy of processes with the stream hierarchy. In addition, since the monitoring process's other job is to stitch the composite into the main network, we could now have "black box" composite components. This facility would allow subnets to be packaged as separate load modules for distribution, which could later be linked with other components by a developer to form the full application network. This seems very attractive, as a software manufacturer will be able to sell a composite component without having to reveal its internals (as would be required by DFDM)! 

http://www.jpaulmorrison.com/fbp/compos.shtml

You could explain this using the inheritance metaphor but it isn't necessary.  The component metaphor is good enough and much easier to grasp.  The board is wired up but some slots are left empty for you to plug in the required behaviour.

It seems to me that the core metaphor within FBP is routing rather than inheritance, which is far more accessible. Consider this from the OOP chapter:

Generally, as you move down the class hierarchy, you add more attributes - so start off with the set of attributes common to all vehicles. When you discover that a file record represents a Pontiac, you now know how to read the remaining attributes. This concept could in fact be added quite naturally to the descriptor mechanism of FBP.

This approach is fundamental to the Internet.  The metaphor of packets with a "header" and "payload" seems so much easier to grasp than inheritance:

http://en.wikipedia.org/wiki/Internet_Protocol

Inheritance is such a tempting metaphor but it obscures things rather than making them clearer.

Regards,

Ged

[Paul Morrison]

Hi Ged, I agree absolutely - you're so right about inheritance, e.g.  banking account types or professions (p. 253 in 2nd edition)...

One of the things we noticed about FBP components is that they often come in matched pairs, e.g. read and write, pack and unpack, etc. so your diagram can be implemented rather neatly as 3 components to send (wrapping the payload) and 3 components to receive (unwrapping).  Easy to understand and easy to reconfigure!

Regards,

Paul