Re: Prismatic Plumbing and Graph Open-Source Release

[László Török]

Graph was THE library I've been waiting for to be open sourced!

Yay, thanks!

Las

Sent from my phone

On Jan 29, 2013 7:57 PM, "Aria Haghighi" <m...@aria42.com> wrote:

Hey all,

 Prismatic has open-sourced our Plumbing and Graph library on github. Jason Wolfe gave a talk about how we use graph for systems composition at Strange loop last year. Please give the library

a whirl and let us know if you're using it and if you find any issues or feature requests. We use this library very heavily throughout our code and hope others find it useful as well.

 Best, Aria

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tim Visher]

On Tue, Jan 29, 2013 at 2:16 PM, László Török <ltor...@gmail.com> wrote:
> Graph was THE library I've been waiting for to be open sourced!
>
> Yay, thanks!

+1 to that! Thanks so much!

--

In Christ,

Timmy V.

http://blog.twonegatives.com/
http://five.sentenc.es/ -- Spend less time on mail

[ronen]

Amazing work Prismatic team, is there a plan to release Flop?

http://lanyrd.com/2012/clojurewest/spdfg/

Thanks!

Ronen

[Aria Haghighi]

Yes, we'll be releasing flop soon. 

[ronen]

Great! 

Thanks again 

Ronen

[Alex Miller]

The video of the talk on Graph from Strange Loop just came out: http://www.infoq.com/presentations/Graph-Clojure-Prismatic

[AtKaaZ]

if you put that on youtube, let me know, currently I cannot see the slides or they are simply stuck on the first slide and never change (the video works though)

--

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Please correct me if I'm wrong or incomplete,
even if you think I'll subconsciously hate it.

[Baishampayan Ghose]

Try Firefox. ~BG

Baishampayan Ghose
b.ghose at gmail.com

[AtKaaZ]

Thanks, I am already using it v 19.0, at one time I was unable to see any videos on infoq but then I realized that's because I perma-denied flash*.exe outgoing in firewall but as soon as I upgraded flash the newly named exe trying outgoing made the firewall prompt me and realized that's why some flash-based things weren't working before (like sites that allow free uploading images). In truth if I ran firefox without any addons it would probably work on infoq ... even though I did my best to allow all needed in noscript+requestpolicy+refcontrol (addons)...

[László Török]

Hi,

Good stuff! I was wondering how do you guys deal with "decision nodes" or does is make sense to model such a thing at the nodel level.

Imagine having a diamond topology e.g.

            N1

           /    \

      N2a    N2b

           \    /

            N3

based on some input to N1, either N2a or N2b gets computed on which N3 depends.

Does this make sense or should this "decision" step be wrapped in a node "N2".

Thanks,

Las

2013/1/29 Aria Haghighi <m...@aria42.com>

Hey all,

 Prismatic has open-sourced our Plumbing and Graph library on github. Jason Wolfe gave a talk about how we use graph for systems composition at Strange loop last year. Please give the library

a whirl and let us know if you're using it and if you find any issues or feature requests. We use this library very heavily throughout our code and hope others find it useful as well.

 Best, Aria

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

László Török

[Jason Wolfe]

Hi Las,

Currently the graph topology is fixed at compile time, and cannot be changed by the values flowing through it. 

In such situations, we either: 

 - Have two copies of the graph with different node functions for N2 (i.e., when generating different types of newsfeeds, from our original blog example) and call different ones depending on input type

 - Put the conditional into a single node N2

It's hard to say more without knowing about your specific use case.  If you want to share details, I'd like to listen, either here or off-list.

Cheers,

Jason

[Edmund]

Hey Aria / Jason,

  Thanks for OSing this library, its really useful.  One question: how do you deal with nesting on the output side of graph declaration ?  I understand it for fnk, but but how would I achieve:

{:a {:b 1}} -> {:a {:b 1, :c 2}}

with a single declaration:

(graph/eager-compile
 {??? (fnk [[:a b]] (inc b))}

where ??? is the binding form I'm after to describe the nesting ? 

I'm currently doing this by breaking up my map and using update-in, but it seems you must have a sneaky trick to do this declaratively ?

I figure [:a :c] would be an possible syntax ?

Thanks again,

 Edmund

[Valentin Golev]

Graph definitions really remind me of `do` syntax in Haskell, where you can bind values and then use them in later steps of the computation.

[László Török]

+1, was also wondering....

2013/2/7 Valentin Golev <m...@valyagolev.net>

--

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

László Török

[Jason Wolfe]

Hi Edmund,

Thanks for your interest.  There's actually no way to fill in ??? in your example, because it's a requirement of Graph that node names must be unique, and distinct from top-level input keys.  This ensures that the Graph has a unique topological order, and it's always clear where the input to a node comes from.  So we can't both treat :a as an input and an output.  (Of course, we could always wrap the graph function to merge the inputs in, but it still won't quite fit your example)

Here are three ways to do something similar:

;; As a single fnk that returns an explicit map

user> ((eager-compile {:a2 (fnk [[:a1 b]] {:b b :c (inc b)})}) 

       {:a1 {:b 1}})

{:a2 {:b 1, :c 2}}

;; As a single fnk that modifies the input map

user> ((eager-compile {:a2 (fnk [[:a1 b :as a1]] (assoc a1 :c (inc b)))}) 

       {:a1 {:b 1}})

{:a2 {:c 2, :b 1}}

;; As a hierarchical graph

user> ((eager-compile {:a2 {:b (fnk [[:a1 b]] b) 

                            :c (fnk [[:a1 b]] (inc b))}}) 

       {:a1 {:b 1}})

{:a2 {:b 1, :c 2}}

Note that these all take input key :a1 and output :a2, to avoid a key clash, and explicitly copy :b to the output.  If you give me some more details about your use case I may be able to provide better advice.

Hope this helps -- let me know if you have any questions.

Cheers,

Jason

[Jason Wolfe]

Interesting.  Our model came from large 'let' statements in Clojure, which I think is similar (I'm not too familiar with Haskell).  The advantage over the let statement is that now you can manipulate the composition, run only part of it, wrap the value functions to monitor them, and so on.  I think the latter use-case is actually very closely related to the Haskell example, since it lets us do the same kinds of things you would do with a monad (see graph_examples_test for an example with 'resources') .  But here the wrapping is a higher-order function, rather than a monad that has to be specified up front.

[Jason Wolfe]

We've just posted a blog post with more high-level context on what we're trying to accomplish with Graph (plus more examples!)

http://blog.getprismatic.com/blog/2013/2/1/graph-abstractions-for-structured-computation

We're also answering questions and reading comments in the Hacker News thread, if that's your thing:

http://news.ycombinator.com/item?id=5183236

Cheers, Jason

[AtKaaZ]

Hello.
https://github.com/Prismatic/plumbing/blob/master/test/plumbing/graph_examples_test.clj#L148

Why do they return in a map instead of maybe a set ? do we ever get {:key false} ?

Thanks.

--

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

[Jason Wolfe]

On Thu, Feb 7, 2013 at 10:54 AM, AtKaaZ <atk...@gmail.com> wrote:
> Hello.
> https://github.com/Prismatic/plumbing/blob/master/test/plumbing/graph_examples_test.clj#L148
> Why do they return in a map instead of maybe a set ? do we ever get {:key
> false} ?
> Thanks.

The leaf value for output schemata is always 'true'. It's a bit odd,
but to support specifications of functions/graphs that return nested
maps, the outer layers need to be maps. It's true that the innermost
layers could be represented as sets, but we chose to keep things
uniform instead.

[Edmund]

Hey Jason,

    Thanks for the suggestions.  Going slightly OT here to describe my motivations, but you did ask
for more details ! :)

I wrote a little blog post + lib last month using Stuart Sierra's Flow (much like
Graph) to implement the steps in doing a very simple discrete inference task.  Its
demonstrating the same thing you talk about in your post about
declarative computation - here the point is that you can efficiently update your
inference based new data on account of having the
graph structure to work over.  Here's the post http://boss-level.com/?p=160 and a
literate version of the code http://ejackson.github.com/inflow/
This is my original motivation.

What I'm doing there is 'building-up' the computation over time,
so I start with a graph with just the input data, and end up with a
graph with the input and output.  But the 'building up' is not the principal hangup I
have: I'm happy to have an in- and out- map.  The problem is, when

[Edmund]

Hey Jason,

   Going slightly OT here to describe my motivations, but you did ask
for more details ! :)

I wrote a little blog post + lib last month using Stuart Sierra's Flow (much like
Graph) to implement the steps in doing a very simple discrete inference task.  Its
demonstrating the same thing you talk about in your post about
declarative computation - here the point is that you can efficiently update your
inference based new data on account of having the
graph structure to work over.  Here's the post http://boss-level.com/?p=160 and a
literate version of the code http://ejackson.github.com/inflow/
This is my original motivation.

What I'm doing there is 'building-up' the computation over time,
so I start with a graph with just the input data, and end up with a
graph with the input and output.  But the 'building up' is not the principal hangup I
have: I'm happy to have an in- and out- map.  The problem is, when

dealing with a very nested graph, getting a fine grained syntax for
describing where to put the outputs.  Ie how do I tell Graph to put just
the output x into
{:a {:b 1 :c x}} of the graph, without replacing the entire map of the :a key ?
That's obviously no good if I need to have lots of things changing
beneath :a in a single graph flow due to uniqueness.[

With Flow what I did is flatten the nested graph such that the keys are
vectors of the path to each node, run the flow on that graph, and then
re-nest the the graph afterwards (the flow-> and ->flow functions).
So the map {:a {:b 1 :c x}} becomes {[:a :b] 1, [:a :c] x}, and I can refer to
nested nodes this way.

In Graph such a scheme would be possible too, but that Graph enforces
keywords as the keys for the graph.  Having vectors for the keys still
allows you to
have uniqueness, topological ordering etc.  So the example I gave in my
last email:

{:a {:b 1}} -> {:a {:b 1, :c 2}}

could, hypothetically, be written for just the outputs as

((graph/eager-compile {[:a :c] (fnk [[:a b]] (inc b))})
  {:a {:b 1}})

>> {:a {:c 2}}

So my question is: how do I refer to nested nodes in the output
selector using Graph?  Conditionally, on that being impossible, would
there be any merit in the vector scheme that I have suggested ?

Sorry for the long post, I hope I haven't obscured my question. I also
get the feeling that I must be missing something very obvious in your
code for how to do this !

 Edmund

[Jason Wolfe]

On Friday, February 8, 2013 1:56:27 AM UTC-8, Edmund wrote:

Hey Jason,

   Going slightly OT here to describe my motivations, but you did ask
for more details ! :)

I wrote a little blog post + lib last month using Stuart Sierra's Flow (much like
Graph) to implement the steps in doing a very simple discrete inference task.  Its
demonstrating the same thing you talk about in your post about
declarative computation - here the point is that you can efficiently update your
inference based new data on account of having the
graph structure to work over.  Here's the post http://boss-level.com/?p=160 and a
literate version of the code http://ejackson.github.com/inflow/
This is my original motivation.

I think I understand what you're trying to do from the example, thanks!

 

What I'm doing there is 'building-up' the computation over time,
so I start with a graph with just the input data, and end up with a
graph with the input and output.  But the 'building up' is not the principal hangup I
have: I'm happy to have an in- and out- map.  The problem is, when
dealing with a very nested graph, getting a fine grained syntax for
describing where to put the outputs.  Ie how do I tell Graph to put just
the output x into
{:a {:b 1 :c x}} of the graph, without replacing the entire map of the :a key ?

Like I said earlier, it's a requirement of Graph that node names must be unique, and distinct from top-level input keys. 

Thus, there cannot be a way to 'put x into the map under :a at key :c' , because there is no way to in-place modify the inputs or other node values of the graph.  The output is just a map of all the node results, nothing more and nothing less.

That said, you can use a hierarchical graph to specify different node functions for the different sub-keys:

;; As a hierarchical graph

user> ((eager-compile {:a2 {:b (fnk [[:a1 b]] b) 

                            :c (fnk [[:a1 b]] (inc b))}}) 

       {:a1 {:b 1}})

{:a2 {:b 1, :c 2}}

So for an example like the one you gave in the above link, you can represent your nested stats directly as a nested graph without resorting to key flattening.  And if you want to compute just a subset of the graph for updated downstream values and then merge this into your previous map, you can do this in two steps:

1.  Pick out the subset of leaf functions that may have changed, using the same strategy you currently use with Flow, and construct a Graph with only those sub-keys.  For example:

user> ((eager-compile {:a2 {:b (fnk [[:a1 b]] b)}}) 

       {:a1 {:b 2}})

{:a2 {:b 2}}

, then 

2. deep-merge these results with the original result map.  

i..e, (deep-merge-with (fn [x y] y) original-results updated-results)

where deep-merge-with is http://clojuredocs.org/clojure_contrib/clojure.contrib.map-utils/deep-merge-with

This logic could be wrapped up into a single higher-order function that takes a graph and knows how to incrementally update the results given incremental changes in input.  Does that make sense / seem like it will satisfy your need?

Cheers, 

Jason

[Edmund]

Hi Jason,

  That's roughly what I figured, thanks for the helpful reply.

Thanks for taking the time release this lib; its really great

Edmund

[Takahiro Hozumi]

Thanks for amazing work!

I want to know how typical CRUD application is implemented.
Do you use single gigantic graph with lazy-compile or separated graph
for each operation?
How do you handle validation and error?
Will almost all x->y function disappear?

On Jan 30, 3:46 am, Aria Haghighi <m...@aria42.com> wrote:
> Hey all,
>

>  Prismatic has open-sourced our Plumbing and Graph library on github<https://github.com/prismatic/plumbing>.
> Jason Wolfe gave a talk<http://blog.getprismatic.com/blog/2012/10/1/prismatics-graph-at-stran...>about how we use graph for systems composition at Strange loop last year.

[Jason Wolfe]

Thanks!  

We use Graph as one a tool among many for writing clean, compositional code.  So Graph builds our services and newsfeeds and processes our documents, but 95% of the code we write does not use Graph explicitly.  We haven't tried using it for something like a CRUD application -- I'm not sure if it's a great fit, but if you give it a go and come up with something cool we'd love to see it.  

Cheers,

Jason

[kovas boguta]

To continue OT,

There is pretty interesting work being done on incremental datalog
computation, that some people might be interested in being aware of.

http://research.microsoft.com/en-us/projects/naiad/

http://channel9.msdn.com/posts/Frank-McSherry-Introduction-to-Naiad-and-Differential-Dataflow

"Naiad is an investigation of data-parallel dataflow computation in
the spirit of Dryad and DryadLINQ, but with a focus on incremental
computation. Naiad introduces a new computational model, differential
dataflow, operating over collections of differences rather than
collections of records, and resulting in very efficient
implementations of programming patterns that are expensive in existing
systems."

[Brent Millare]

Hi,

Very nice work. I'm interested in using graph but just curious in terms of your priorities for future development.

I noticed that you listed as a todo, you might want to save the body of a fnk which I see as a potential for inlining the bodies and thus eliminating the fn call when creating the final composite fnk. Is this correct?

Also how would you advice situations when you have mutual recursion (with conditions to bound execution) but still want the compositional power of graph? (like being able to substitute only certain components of the graphs in different situations). I understand that my usage of recursion implies there will be cycles and graph only emits code in topological order so its currently not supported. Is there future support for this or is this outside the scope of graph? Any tips for reclaiming compositional power?

-Brent

[Jason Wolfe]

Hi,

On Thu, Mar 7, 2013 at 7:53 PM, Brent Millare <brent....@gmail.com> wrote:
> Hi,
>
> Very nice work. I'm interested in using graph but just curious in terms of
> your priorities for future development.

Thanks!

> I noticed that you listed as a todo, you might want to save the body of a
> fnk which I see as a potential for inlining the bodies and thus eliminating
> the fn call when creating the final composite fnk. Is this correct?

That's right. Thus far the overhead has not been an issue for us,
since we usually do a significant amount of work in most Graph nodes,
to where the overhead for the input map construction and function call
is negligible. So this isn't a very high priority for us currently --
but it shouldn't be much work either. If it's something you're
interested in helping with, we'd be happy to work with you on that.

>
> Also how would you advice situations when you have mutual recursion (with
> conditions to bound execution) but still want the compositional power of
> graph? (like being able to substitute only certain components of the graphs
> in different situations). I understand that my usage of recursion implies
> there will be cycles and graph only emits code in topological order so its
> currently not supported. Is there future support for this or is this outside
> the scope of graph? Any tips for reclaiming compositional power?

How would mutually recursive nodes work, exactly? My guess is that
this is probably probably beyond the scope of Graph. That said, you
can always do the mutual recursion *inside* a single graph node, where
the functions come in as parameters or other nodes (and are thus still
easily substitutable), or *outside* the graph (by running (portions
of) it multiple times).

-Jason

>
> -Brent

>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---

> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/UYXcoK0bTNc/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to

[Vishi]

Nice!

Any more info on flip open source release? Possible timeframe?

thanks

[Jason Wolfe]

It's out now (under a different name) since it's not just for floating
point numbers anymore!

https://github.com/Prismatic/hiphip

Release announcement:

https://groups.google.com/forum/#!topic/clojure/F0jg2-uBW6U

> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit

> https://groups.google.com/d/topic/clojure/UYXcoK0bTNc/unsubscribe.