Graph Viz?

[Cody Hall]

Hello drake-workflow, 

I've sufficiently impressed my boss by building a workflow to handle our crazy data processing pipeline. Thanks to everyone who has contributed to make this a very pleasant tool. My question is, after reading as much as I could find on the subject, is there a correct{easy} way to generate a visualization of the dependency graph? 

I've found many references to visualization of the dependency graph:

Here's one in the spec page 49 : https://docs.google.com/document/d/1bF-OKNLIG10v_lMes_m4yyaJtAaJKtdK0Jizvi_MNsg/edit#

- I'm assuming that because it is in RED, it's not implemented.

Here's an email thread from this group, but the comment from year ago seems like the feature did not make it into a pull request?  https://groups.google.com/forum/#!searchin/drake-workflow/graph/drake-workflow/e8G8RN9jrcU/SU4oZfaqregJ 

And in Issue #37 Mr. Tellman has some code that's not in factual/drake that appears to make a graph. I'm so new to clojure... so I'm not even sure how I would use this. https://github.com/Factual/drake/issues/37

I've also played with the --print option, but I am not strong enough to parse this output and make a .dot file out of it. 

Does anyone have a quick tutorial about how to generate a visual dependency graph from the Drakefile or the --print option?

-Cody 

[Alan Malloy]

This sounds like a pretty cool feature that I don't think would be that hard; I'll give it a shot when I have some time. In the mean time, try looking at https://github.com/Factual/drake/blob/c3b28ac1ca3ebfb37d07c80a3df4ac459d4df395/src/drake/core.clj#L590 - this is the code that prints out the steps in --print. The idea is to add another command-line option like --graph, which does the same thing but uses rhizome (https://github.com/ztellman/rhizome) to produce a graph instead of printing to stdout.

[Cody Hall]

Hi Alan, thank your for pointing me to the correct area in drake source for further hacking. 

A friend helped me put this program together to take the output of drake --print and turn it into a .dot file. This is achieved by running 

$> drake -- print >> drake-graph.txt 

, and then using that as an input to the data processing in clojure.

I was able to generate a pdf of the workflow by running 

$> dot Tpdf dotf.dot -o graph1.pdf

, on the output file. 

Anyway I thought I would share back what I ended up using to prove this out, I'm going to make some enhancements over the weekend like accepting a file input name when calling the -jar file. https://github.com/codyhall/make-dot/blob/master/src/make_dot/core.clj

-Cody

[Alan Malloy]

I've got a first draft of this feature in the "viz" branch on github. The short version is that for the "human resources" demo, the command `drake --graph +people.skills first_gt_last.txt` produces this graph: 

The output of --help explains what the color-coding means, but basically green files are those that will be built, and black outlines were forced; white ones exist in the workflow, but don't need to be rebuilt. I'd love some feedback on what kind of information is actually useful to see in the graph: maybe we don't want to include the white nodes at all, for example?

[Cody Hall]

Hi Alan, 

I had a chance to play with the --graph option over the weekend. All I can say is wow, what an incredibly efficient way to generate the Viz! Thank you for taking the time to implement this feature. 

The defaults you've applied with white, green and the black outline are perfect for visually explaining the execution plan that drake creates. I think you picked winners.

I'll do some more testing and write back. 

-Cody

[Matan Safriel]

+1. Awesome and very useful in my view.

[ko...@tresata.com]

On Friday, November 14, 2014 at 8:27:51 PM UTC-5, Alan Malloy wrote:
> I've got a first draft of this feature in the "viz" branch on github. The short version is that for the "human resources" demo, the command `drake --graph +people.skills first_gt_last.txt` produces this graph: 
>
>
>

i tried this new feature today, but didn't realize it actually requires X11 :(
not going to happen on the server on which we run drake.

Caused by: java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.

i was hoping it would create a dot file that i can then visualize myself.

[Alan Malloy]

I think that feature shouldn't be too hard to add, since the first thing drake does is create a dot internally. I'm not sure how to expose such a feature, though: I don't like adding an optional argument to --graph, and I don't want to add a bunch of graph-related commands which are all distinct either. Any clever ideas?

[Koert Kuipers]

not sure. i think it is reasonable to expect drake to be run on headless servers. we dont have any servers or desktops with x11 that can be connected to the relevant hdfs. even if technically possible it would not be allowed.  so i cannot generate the graph on another box either. i believe this situation is fairly typical.

i like --dot but that is indeed adding yet another graph related command. or i would argue for --graph creating the dot file and not trying to render it within drake.

[Alan Malloy]

I just committed https://github.com/Factual/drake/commit/6ec6913886107a263ce11240a6e40f66bf89747e to the viz branch (the merge into develop is still pending some rhizome changes from Zach), which adds "--graph=dot" to just output the .dot file for your use, which works headless.

[Koert Kuipers]

great! we will give it a spin

--
You received this message because you are subscribed to a topic in the Google Groups "drake-workflow" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/drake-workflow/V0PypFEQqn8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to drake-workflo...@googlegroups.com.
Visit this group at http://groups.google.com/group/drake-workflow.
For more options, visit https://groups.google.com/d/optout.

[Alan Malloy]

I've merged all the graphing stuff, dot-only, into drake develop, so you don't have to keep using the viz branch.