is "simpleinput => arrayport[2] a()" possible?

[stewart mackenzie]

Is this:

"simpleinput => arrayport[2] a()"

possible in classical flow based programming?

If not, why not?

kr/sjm

[Paul Morrison]

Hi Stewart,

I am guessing that "simpleinput" is an IIP, and "a()" is meant to be the process name...  if so, then I see no problem with it.  Is it giving you problems in some implementation?   Although I would guess arrayport[0] and arrayport[1] are also connected to something...?

Regards,

Paul

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-programming+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[stewart mackenzie]

I just don't recall any mention of this anywhere. Our implementation doesn't support this scenario, so we'll fix that asap, as we have a need for this type of behaviour.

Basically we'll have a named data network router implemented in fractalide and it exposes one input array port and one output array port. The idea is that a bunch of business logic components that need to get data from the network can just use names and not bother about routing, ip address, socket connection, retries and all that. These names are sent via a simple port into one element of the array input port of the ndn router. That way we have a very simple reusable interface that can accept as many incoming information packets from many different components (as long as they speak the same cap'n proto contract). The use of ndn in a microservice setup makes for much easier networking, as the networking is entirely folded away and only names are exposed.

[stewart mackenzie]

"simpleinput" is not an IIP it is a simple input port which is a virtual port.

It was a contrived example to illustrate the concept of a simple port feeding an element of an array port.

Our IIPs are more complex!

'${maths_boolean}:(boolean=true)' => a nand(${maths_boolean_nand})
'${maths_boolean}:(boolean=true)' => b nand()

Is a taste of our contract IIPs.

[Paul Morrison]

I can sort of see virtual ports - two of my implementations (JavaFBP and C#FBP) support subnet external names - this sounds pretty close!   Did you want to expand on the idea of virtual ports?

Your IIPs look interesting - but I get worried whenever I see processes like NAND as it seems the wrong level of granularity.  IMO FBP should be concerned with structured objects, rather than scalars - the latter being more the concern of procedural code.  However your next post sounds more like the level of granularity I am comfortable with, and actually sounds very promising!  So I assume the NAND example was just to show the syntax!

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-programming+unsubscri...@googlegroups.com.

[stewart mackenzie]

On Monday, October 31, 2016 at 7:28:35 AM UTC+8, Paul Morrison wrote:

I can sort of see virtual ports - two of my implementations (JavaFBP and C#FBP) support subnet external names - this sounds pretty close!   Did you want to expand on the idea of virtual ports?

Our virtual ports / subnet external names are the same as yours. 

It's absolutely critical for abstraction purposes, and I have no idea how one can effectively use FBP without this concept!

 

Your IIPs look interesting - but I get worried whenever I see processes like NAND as it seems the wrong level of granularity.  IMO FBP should be concerned with structured objects, rather than scalars - the latter being more the concern of procedural code.  However your next post sounds more like the level of granularity I am comfortable with, and actually sounds very promising!  So I assume the NAND example was just to show the syntax!

Yes the NAND example is contrived and doesn't reflect reality.

This is a more real world example of a contract IIP

This contract: https://github.com/fractalide/fractalide/blob/master/contracts/js/create/default.nix#L8-L29

Is reflected in this subnet using this structure : https://github.com/fractalide/fractalide/blob/master/components/app/growtest/default.nix#L18

Notice the [...] allows us to stick lists in the contract.

This is an example of processing such a contract: https://github.com/fractalide/frac_ui_js/blob/master/components/ui/js/tag/src/lib.rs#L92

This is why I prefer a more simple contrived NAND example to convey the point :-)

But back on topic!

So the concept of feeding a simple output port feeding into an element of an array input port is viable? It seems totally legit to me.

kr/sjm

 

[Ged Byrne]

Hi Stewart,

I'm getting a bit confused by this conversation.  Do you mind if I just cover some basic concepts to help me understand?

In FBP we have three fundamentals concepts: Components, Ports and Connectors.

The wikipedia article explains these concepts as follows:

A, B and C are processes executing code components. O1, O2, and the two INs are ports connecting the connections M and N to their respective processes. ...  M and N are what are often referred to as "bounded buffers", and have a fixed capacity in terms of the number of IPs that they can hold at any point in time.

So we have:

• Three component processes: A, B and C
  
• Four Ports:
• Two Output: A->O1 and A->O2
• Two Input: IN->B and IN->C
• Two connectors:
• M: A-01 => IN->B
• N: A-02 => IN->C

In your question you have most of these elements, except you don't have an output port for SimpleInput.  It should look like this:

• simpleinput->OUT => arrayport[2]->a()
  

Within classical FBP you can connect a simple output port to an array-type input port. It is shown on the Wikipedia page:

Collate has an array-type input port called N.  The two Read components both have a simple output port called IN.

So this gives you two connections:

• ReadMasters->OUT => N[0] -> COLLATE
• ReadDetails->OUT => N[1] -> COLLATE

The following shows the creation of the first of these connections in three different implementions:

• JsFBP
• https://github.com/jpaulm/jsfbp/blob/master/test/components/collate.js
  
• network.connect(sender0, 'OUT', collate, 'IN[0]', 5);
  
• JavaFBP
• https://github.com/jpaulm/javafbp/blob/master/src/main/java/com/jpaulmorrison/fbp/resourcekit/examples/networks/Update.java
  
• connect(component("Read Master"), port("OUT"), component("Collate"), port("IN[0]"));
  
• C#FBP
• https://github.com/jpaulm/csharpfbp/blob/master/TestNetworks/Networks/Update.cs
  
• Connect("@Read@ Master (1).OUT", "@@Collate (3).IN[0]");
  

As you can see, the concept remains consistent across the implementations.

So in JavaFBP your example creating the connector would look like this:

• connect(component("Simple Input"), port("OUT"), component("a"), port("ARRAY_PORT[2]"));

What does this line look like in your implementation?

If you were to build the collate graph in your implementation what would this line look like?

Regards, 

Ged

[Ged Byrne]

My apologies for posting an early draft by mistake.

---

From: Ged Byrne
Sent: ‎31/‎10/‎2016 12:54
To: Flow Based Programming
Subject: Re: is "simpleinput => arrayport[2] a()" possible?

Hi Stewart,

I'm trying to follow your examples, and I'd just like to confirm that you are supporting the concept of an explicit connector.

Consider this initial concepts diagram and text from the book and Wikipedia page:

"A, B and C are processes executing code components. O1, O2, and the two INs are ports connecting the connections M and N to their respective processes...  M and N are what are often referred to as 'bounded buffers', and have a fixed capacity in terms of the number of IPs that they can hold at any point in time."

https://en.wikipedia.org/wiki/Flow-based_programming#Concepts

In classical FBP with have processing components like A, B and C which have ports that allow them to be connected together.  In this example "01" and "02" are both output ports for A.  B and C both have input ports called "IN".

These ports are linked to each other with connectors.  In this example the M connects A->01 to IN->B while N connects A->02 to IN->C. 

That gives us two possible paths through the graph:

A->01=>M=>IN->B

A->02=>N=>IN->B

Comparing this you describe the following: 

• "simpleinput => arrayport[2] a()" 
  

Paul's initial assumption is that simpleinput is an IP waiting within a connection buffer.  I hope this makes sense given the full model described above.

You seem to say that "simpleinput" is actually a processing component that will obtain some input.  Is that correct?  Does simple input have an output port?  You represent the connection as =>, in your implementation is that connection an explicit, first class entity?

i.e. Can you recast the above like this:

• "SimpleInput -> <OutputPort> => <Connection> => arraport[2] a()"
  

That said, the connector is usually anonymous so a single => is enough.  It is still, however, an essential first class citizen because the receiving component cannot know anything at all about the sending component.  Instead they know about the connection that is bound to their input port.

Within classical FBP you can connect a simple output port to an array-type input port.  The Wikipedia page includes an example using the Collate component:

https://en.wikipedia.org/wiki/Flow-based_programming#Batch_update

In this example we have the following two paths linking the two reading component's simple output ports to the array-type input port of Collate:

• ReadMasters->OUT => N[0]->Collate
  
• ReadDetails->OUT => N[1]->Collate

The reason for having these ports and connectors is to make the processors entirely independent black boxes.  Collate has no knowledge of either ReadMasters or ReadDetails.  These two readers have no knowledge of Collate.

Here is a test case for collage in the different implementations.  They all show a connector between a simple and array-type ports.  I've shown the line that creates the connection "ReadMasters->OUT => N[0]->Collate" :

• JSFBP

• https://github.com/jpaulm/jsfbp/blob/master/test/components/collate.js
• network.connect(sender0, 'OUT', collate, 'IN[0]', 5);
  

• Note: ("Sender0" is a mock of "ReadMaster")
  

• JavaFBP
• https://github.com/jpaulm/javafbp/blob/master/src/main/java/com/jpaulmorrison/fbp/resourcekit/examples/networks/Update.java
  
• connect(component("Read Master"), port("OUT"), component("Collate"), port("IN[0]"));
  
• C#FBP
• https://github.com/jpaulm/csharpfbp/blob/master/TestNetworks/Networks/Update.cs
  
• Connect("@Read@ Master (1).OUT", "@@Collate (3).IN[0]");

As you can see, the concepts remains fairly constant across the languages.

If you were to create the same graph in your implementation, what would this line look like?

Regards, 

Ged

On Sunday, 30 October 2016 05:05:19 UTC, stewart mackenzie wrote:

--
You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-progra...@googlegroups.com.

[stewart mackenzie]

The subnet implementation:
simple_virtual_input_port => array[1] comp()
is not allowed in Fractalide.

which is exactly the same as:
simple_virtual_input_port => input do_nothing(${pass_through}) output -> array[1] comp()
is allowed in Fractalide.

Does this make it clear?

[stewart mackenzie]

Wait, maybe the confusion is arising from the syntax?

we use the => to indicate a virtual/subnet input or output "terminal".

see the not gate utilizing the nand gate: https://github.com/fractalide/fractalide/blob/master/components/maths/boolean/not/default.nix#L6-L8

Then notice our implementation of xor https://github.com/fractalide/fractalide/blob/master/components/maths/boolean/xor/default.nix#L6-L11

We use the -> to indicate message passing cap'n proto contracts between components.

The naming structure xxx_yyy_zzz directly translates to the filesystem hierarchy i.e. 

component name "maths_boolean_and" is in folder fractalide/components/maths/boolean/and 

component name "maths_boolean_or" is in folder fractalide/components/maths/boolean/or etc

(to be clear this maths_boolean_* stuff is simple contrived stuff to help us test subnets, don't take these as good examples of using fbp at all please) 

Lastly here are examples of the implementation language connecting components together:

https://github.com/fractalide/fractalide/blob/master/support/vm/src/lib.rs#L57-L89

This is one of the few components that actually does this, as it bootstraps all the components needed to parse flowscript.

In other words, our flowscript language doesn't support "simple_virtual_input_port => array[1] comp()" I'm still seeing if we actually need it or not.

[Paul Morrison]

I share Ged's confusion!  From my point of view, the two lines are exactly the same, except for the addition of the PassThru process.  My question remains: by "simple_virtual_input_port" do you mean the external port of the subnet containing the network you show?  If not, it seems to represent a concept I am not (yet) familiar with!  Also, why would the first line not be allowed in Fractalide, while the second line is?

In some of my implementations I do use => to represent the external-port-to-internal-port connection...  As external ports in JavaFBP and C#FBP are handled by means of special components SubIn and SubOut, which take the external port name as an IIP, there is special notation in DrawFBP to simplify this.  Diagram attached.

[Paul Morrison]

Sorry, Stewart, I didn't see this note - but my question in my last note still stands!

Regards,

Paul

--

You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-programming+unsub...@googlegroups.com.

[Ged Byrne]

I see, so "Simple Input" is not a component but a input port to a subnet?

So this seems closer to the promotion of services from components to composites found in SCA, as shown in the following diagram:

Does that look right to you?

Subnet behavious in FBP is handled with the concept of substream sensitivity.  Here's a test for JavaFBP:

 

https://github.com/jpaulm/javafbp/blob/master/src/main/java/com/jpaulmorrison/fbp/resourcekit/examples/networks/SubnetX.java

This test is similar to the following diagram from the book: 

Regarding composites I'm not so sure about their exact behaviour.  Hopefully Paul can clarify.  

@Paul, what would the collate example look like if Collate was in a Subnet?

Regards,

Ged

[stewart mackenzie]

On Monday, October 31, 2016 at 10:34:45 PM UTC+8, Ged Byrne wrote:

I see, so "Simple Input" is not a component but a input port to a subnet?

This is correct, forgive me I thought my words "The subnet implementation:" conveyed my meaning accurately.

Whereas the subnet interface would look like this:

'${some_random_contract}:(some_random_field="random_data")' -> simple_virtual_input_port random_subnet(${some_random_subnet})

So, it would seems like we might have an issue, but it hasn't bitten us yet, so I'll just make an issue w.r.t. this.

 

[stewart mackenzie]

Ah okay, I've just had a quick word with Denis, my partner. He says the backend is supports the concept: https://gitlab.com/fractalide/fractalide/blob/master/support/rustfbp/src/scheduler.rs#L285

but our flowscript parser has a bug!

Thanks folks!

[Paul Morrison]

Hi Stewart,

In the signature

connect_to_array(&self, comp_out: String, port_out: String, comp_in: String, port_in: String, selection_in: String)

What is the semantics of "selection_in"?  If you are selecting on the contents of the packets being sent, what happens to rejected packets?  This also seems to assume that the contents of the packets are always strings - I can see this as useful between systems, or independent applications, but it seems unnecessarily restrictive within an application.

Do I deduce that you have different "connect" statements - if so, I am not sure why, esp. since there doesn't seem to be a separate (input or output) port element parameter - in "classical" FBP, you just connect, e.g.

COMP1 OUT[1] -> IN[0] COMP2...

or in JavaFBP

connect(component("Read Details"), port("OUT"), component("Collate"), port("IN[1]"));

We get further simplification by allowing [0] to be omitted, as the component knows whether the port is an array port or not..

Regards,

Paul M.

[stewart mackenzie]

On Tuesday, November 1, 2016 at 10:06:20 PM UTC+8, Paul Morrison wrote:

Hi Stewart,

In the signature

connect_to_array(&self, comp_out: String, port_out: String, comp_in: String, port_in: String, selection_in: String)

Yes we have a number of different connects, have a look at all the methods.

 

What is the semantics of "selection_in"?  If you are selecting on the contents of the packets being sent, what happens to rejected packets?  This also seems to assume that the contents of the packets are always strings - I can see this as useful between systems, or independent applications, but it seems unnecessarily restrictive within an application.

selection_in is the array element, we chose it to be a string because then you can do neat things like this: https://github.com/fractalide/frac_ui_js/blob/master/components/ui/js/edit/default.nix#L21-L32 

"If you are selecting on the contents of the packets being sent, what happens to rejected packets?" - We are not selecting on the contents of packets being sent at this level.

"This also seems to assume that the contents of the packets are always strings" - it really doesn't assume that at all, if you read the other function signatures you will see a very clear pattern. You also know that we are using Cap'n Proto contracts as our IPs which is essentially a type system for distributed systems., so definitely the packets are not always strings! 

selection_in is probably misleading and badly named. maybe "element_in" would be better?

 

Do I deduce that you have different "connect" statements - if so, I am not sure why, esp. since there doesn't seem to be a separate (input or output) port element parameter - in "classical" FBP, you just connect, e.g.

COMP1 OUT[1] -> IN[0] COMP2...

or in JavaFBP

connect(component("Read Details"), port("OUT"), component("Collate"), port("IN[1]"));

We lift out of the implementation language graph connections as fast as possible. Indeed we lift out of implementation language component connections in the first function executed (https://github.com/fractalide/fractalide/blob/master/support/vm/src/lib.rs#L22). Our sole interaction with with components is at a flowscript level, which is expressive enough.

In classical FBP we also just "connect" like this COMP1 OUT[1] -> IN[0] COMP2... 

By lifting out of implementation level component connections and into classical fbp / flowscript you don't have to all the wetware needed to differentiate between the different connections method signatures you just use ->

In other words, many fbp implementations enjoy staying at implementation language level when connecting components, those implementations need a more simple way to connect components, so a simple "connect" is desirable. For us this is not the case. 

 

We get further simplification by allowing [0] to be omitted, as the component knows whether the port is an array port or not..

yes we can also omit the [x] as the component also knows if its a simple port or array port.

[Paul Morrison]

Hi Stewart,

I read selection_in as indicating some kind of selection on the contents of the IPs, so you can ignore my remarks that followed from that!

Rather than having a separate parameter, I would suggest a notation like:  port_out: [String, String] where the first string is the port name, and the second is the index, or even just use some notation like portname[index], which just means square brackets cannot be part of the port name.  This means you don't have to have different flavours of 'connect'.

I'm afraid I have no idea what the section starting 'We lift out...'  - can you rephrase that?

TIA

Paul M. 

--

You received this message because you are subscribed to the Google Groups "Flow Based Programming" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flow-based-programming+unsub...@googlegroups.com.

[stewart mackenzie]

On Thursday, November 3, 2016 at 10:11:19 PM UTC+8, Paul Morrison wrote:

Rather than having a separate parameter, I would suggest a notation like:  port_out: [String, String] where the first string is the port name, and the second is the index, or even just use some notation like portname[index], which just means square brackets cannot be part of the port name.  This means you don't have to have different flavours of 'connect'.

The latter case is what we have.

 

I'm afraid I have no idea what the section starting 'We lift out...'  - can you rephrase that?

For this discussion's purpose, let's make a few definitions!

 Lets define this language which has this syntax: 

"in => a nand() out => out" 

as "mid-level fbp": 

sched.connect("open".into(), "output".into(), "lex".into(), "input".into()).expect("cannot connect");

as "low level fbp" 

High level would be an app that draws nice nodes and edges, something like noflo ui or drawfbp would be this level.

So we lift out of low level fbp as fast as possible using this neat wizardry: https://github.com/fractalide/fractalide/blob/master/support/vm/src/lib.rs#L24-L36

notice each of the "<name>.so"? Well those names are substituted in place by corresponding default.nix file: https://github.com/fractalide/fractalide/blob/master/support/vm/default.nix#L10-L27 just before compile time https://nixos.org/nix builds and replaces the specified name with the build arefact path.

This means when this top level rust program is ready to be compiled it has all the correct paths to components inserted into the source code.

These particular components are the minimal set of components needed to parse, execute mid-level fbp (and load other components). This bit of code is the nucleus of fractalide which can be found in this folder: https://github.com/fractalide/fractalide/tree/master/components/nucleus

the particularly interesting folder is the "flow" folder where you'll find all the juicy goodies.

The build product of the above process is basically the executable. 

But how do we inject a new business logic specific subnet each time? This is a statically compiled language after all! Well we call "$ nix-build" command with the argument "--argstr subnet test_sjm" which is this component: https://github.com/fractalide/fractalide/blob/master/components/test/sjm/default.nix or any other top level subnet you wish to call.

now nix substitutes in place one top level business logic subnet here: https://github.com/fractalide/fractalide/blob/master/support/vm/src/main.rs#L6 notice the "nix-replace-me".  Here's the bit of code that'll do the substitution: https://github.com/fractalide/fractalide/blob/master/support/vm/default.nix#L27

now as nix is a lazy language, all those components composed into the hierarchy of the chosen business logic subnet will be compiled and all the ${...} will be replaced with the build artefact path. That is the reason why we have the ${...} in our subnet code: https://github.com/fractalide/fractalide/blob/master/components/test/sjm/default.nix#L11 that ${...} is actually telling nix to evaluate that name and replace it with the absolute path to the build artefact.

Okay all this is to say, that we lift out of low level fbp into mid level fbp as fast as possible. As mid level fbp is more compatible with nix substitutions which allows us to evaluate entire deep hierarchies of subnets lazily fashion!

kr/sjm

[stewart mackenzie]

Oh there's a neat hack, I just had to share: Denis just put regular expressions in the port element bit between the [xxx]

i.e. https://github.com/fractalide/fractalide/blob/master/components/workbench/default.nix#L34-L36

This is the processing: https://github.com/fractalide/frac_net_http/blob/master/components/net/http/src/lib.rs#L148-L158

neat huh!? :-D