[Pipes/Gremlin] Massive Refactoring -- Everything is in Pipes!

[Marko Rodriguez]

Hi,

In Gremlin, there are many "Gremlin specific pipes." A better term is "Groovy specific pipes" because these pipes make use of Closures. For example,

loop{it.loops < 4}
filter{it.weight > 0.5}
sideEffect{println it}
transform{it.outE.count()}
groupCount(m){key}{value}
...
..
.

In order to generalize this functionality (i.e. these pipes) and make it available to anyone wanting to use these pipes in native Java or any Gremlinesque-clone in another JVM language, I've pushed all these closure-based pipes to Pipes. You might ask,
"Well don't they still depend on Groovy as they require a Groovy Closures?"
The answer is
"No, they now require a PipeClosure which is an interface with a single method:
Object PipeClosure.compute(Object... parameters).
Similar to a Groovy Closure which has:
Object Closure.call(Object... parameters).

Next, now in Gremlin, when a Groovy closure is used in Gremlin, it is wrapped in a ClosurePipeClosure (a class that maps a Groovy closure to a native PipeClosure -- need to find a better name :) ). In this way, there are no longer any Pipes in Gremlin. Pipes has all the pipes with those that require a closure now requiring a PipeClosure. Gremlin is simply a bunch of metaClass loaders. Gremlin is as small as it can possibly be .. its tiny... nothing to it.

I have not committed or deployed this model yet. I'm still going through refactoring and migrating all the test cases for these Groovy/Gremlin-based pipes over to Pipes and writing documentation, etc. I wanted to explain what I'm doing and to get any comments/feedback on the idea before I fully commit.

Finally, note that I'm going to do a package name refactoring of Pipes so that the pipes in Pipes are better organized. Thus, if you use native Pipes in your project, you might have to rename some imports to make your project happy.

Thoughts?,
Marko.

http://markorodriguez.com

[Peter Neubauer]

Marko,
love to see Pipes being clean and doing what they should. Good call to
weed out Groovy stuff down to Pipes.

Cheers,

/peter neubauer

GTalk:      neubauer.peter
Skype       peter.neubauer
Phone       +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter      http://twitter.com/peterneubauer

http://www.neo4j.org               - Your high performance graph database.
http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.

[Pierre De Wilde]

Good to see all pipes in one place.

Ambitious refactoring, Thanks,

Pierre

2011/7/24 Peter Neubauer <peter.n...@neotechnology.com>

[James Thornton]

Wow -- you've cleared two big hurdles in two days (https://groups.google.com/forum/#!topic/gremlin-users/CofSlY9Cyds). You're on a roll! 

[Marko Rodriguez]

Hi,

All the changes have been pushed and deployed.

Here are the big changes in Pipes:

1. PipeClosure<T,P extends Pipe>: A Pipe can take a PipeClosure which is a computation that will occur somewhere internal to the Pipe. In short, its a way to give the Pipe "runtime" behavior.

- T is type of object that PipeClosure.compute() returns.

- P is the type of Pipe that serves as the host/delegate/owning Pipe.

2. All the Gremlin Pipes that made use of Groovy Closures are now in Pipes where the Closure required is a PipeClosure.

Here are the big changes in Gremlin:

1. A GroovyPipeClosure is a class that implements PipeClosure. In short, it takes a Groovy Closure and makes it a PipeClosure.

2. All the Pipes (besides GremlinPipeline and TablePipe -- for reasons explained later) are now in Pipes.

I've done my best with JavaDoc in Pipes with all the new Pipes. I'll keep going over it all again and again. Also, I will add more and more TestCases for all the migrated Pipes. Finally, I will need to do some major work in the Pipes Wiki documentation to articulate these concepts. This will all happen this week before the TinkerPop release the following week.

In short:

1. Whatever you can do in Gremlin, you can do in Pipes.

2. Any JVM language (Java, JRuby, Clojure, Jython, etc.) can take advantage of these Pipes to effect a graph DSL.

3. Gremlin codebase is tiny (near nothing--80% of the code is test cases). All the behavior is in Pipes.

Thoughts are more than welcome,

Marko.

http://markorodriguez.com

[Pierre De Wilde]

Hi Marko,

Incredible fast refactoring. 

Hopefully we have a few days before release to test it.

Thanks for your effort,

Pierre

2011/7/24 Marko Rodriguez <okram...@gmail.com>