Implementing own Gremlin Step

[David Kul]

Hi,

I'd like to perform specific queries on my Titan/ES setup, and that would mean defining my own methods in the groovy console.

My question is thus : how would I go about implementing my own steps ?

Do I need to fork everything, or maybe extend class GraphTraversal ? I'm not sure where to start.

Any advice ?

[Stephen Mallette]

If you are doing everything in the groovy console, then i suppose you could just do some groovy metaprogramming to add methods to GraphTraversal. That would be pretty fast and easy under that use case. I'm still not 100% clear on how DSLs are best implemented. Marko, might have more to say on the matter...

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/d2b31f3d-bf9f-4475-9fd6-573b2b81350c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Kul]

I used the term "groovy console" to mention the steps commonly used in said console.

But I would need these custom steps to be fully implemented and not just available through the console.
I'm using gremlin-server and performing remote requests, for example.

I've had a look at the code and it appears I could extend some of the dsl.graph classes, but it seems there are some private methods and attributes that will make the job quite difficult...

As I said, I'm not sure what's the best way to do this. I figured you might have an idea...

Thanks for any suggestions.

[Marko Rodriguez]

Hello,

It is NOT easy to simply extend GraphTraversal in Java. The way in which DSLs are to be written is by extending Traversal and adding steps you want.

However, in Groovy, given meta-programming, it is easy to “extend” GraphTraversal.

DefaultGraphTraversal.metaClass.myStep = {

  a,b,c,d -> return delegate.addStep(new MyStep(delegate, a, b, c, d));

}

EXAMPLE:

gremlin> g = TinkerFactory.createModern().traversal()

==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]

gremlin> import org.apache.tinkerpop.gremlin.process.traversal.step.map.SelectStep

gremlin> DefaultGraphTraversal.metaClass.myStep = { a,b ->

gremlin>   return delegate.addStep(new SelectStep(delegate,Pop.all,a,b))

gremlin> }

==>groovysh_evaluate$_run_closure1@2472c7d8

gremlin>

gremlin>

gremlin> g.V().myStep("a","b").toString()

==>[GraphStep(vertex,[]), SelectStep(all,[a, b])]

gremlin>

***NOTE*** Adding steps is for experts only. If you add custom steps (as opposed to having a myStep() that is simply the composite of existing TinkerPop steps), then the compiler may not be able to optimize accordingly and/or you will need to write your own optimizers.

HTH,

Marko.

http://markorodriguez.com

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/0dfede41-c64c-4283-a8e7-dfa46e4f1d7d%40googlegroups.com.

[ern...@rocketjourney.com]

Hi Marko, just one question. What class from what package do I have to extend to implement a step? are there any examples? tinkerpop3 I stopped following the project at version 2, I really liked Gremlin.defineStep... Would be cool if you point us on the right direction, I spent my weekend doing some TDD with tinkerpop3. 

[Marko Rodriguez]

Hi Ernesto,

The easiest way to do what you want is to simply use flatMap(), map(), filter(), or sideEffect() as such:

gremlin> g = TinkerFactory.createModern().traversal()

==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]

gremlin> myStep1 = {it.get().value('name').length() > 4}

==>groovysh_evaluate$_run_closure1@18ca3c62

gremlin> myStep2 = {(25..it.get().value('age')).iterator()}

==>groovysh_evaluate$_run_closure1@5c09d180

gremlin>

gremlin>

gremlin> g.V().hasLabel('person').filter(myStep1).values('name')

==>marko

==>vadas

==>peter

gremlin> g.V().hasLabel('person').flatMap(myStep2)

==>25

==>26

==>27

==>28

==>29

==>25

==>26

==>27

==>25

==>26

==>27

==>28

==>29

==>30

==>31

==>32

==>25

==>26

==>27

==>28

==>29

==>30

==>31

==>32

==>33

==>34

==>35

gremlin>

This way you don’t have to extend a Step and you can name your closure/lambda with a variable name and thus, its easy “to see” the “step” being used in your traversal.

HTH,

Marko.

http://markorodriguez.com

On Jun 27, 2016, at 3:10 PM, ern...@rocketjourney.com wrote:

Hi Marko, just one question. What class from what package do I have to extend to implement a step? are there any examples? tinkerpop3 I stopped following the project at version 2, I really liked Gremlin.defineStep... Would be cool if you point us on the right direction, I spent my weekend doing some TDD with tinkerpop3. 

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/88cc994d-4351-4a13-8607-5bc006f1a4f9%40googlegroups.com.

[ern...@rocketjourney.com]

What I wanted to avoid, is make my partners learn the QL, and instead, build one around our data model. For example, with tinkerpop2 I was able to do some queries like

g.findOrCreateWithFacebook(id, params).user.teams

And anyone on my team was able to query the graph.

[Marko Rodriguez]

Hi,

You can do stuff like this:

gremlin> GraphTraversalSource.metaClass.findOrCreate = { id, key, value ->

gremlin>   return delegate.V(id).hasNext() ? delegate.V(id) : delegate.addV('person').property(T.id,id).property(key,value)

gremlin> }

==>groovysh_evaluate$_run_closure1@4a8a60bc

gremlin>

gremlin> DefaultGraphTraversal.metaClass.teams = {

gremlin>   return delegate.out('created').values('name')

gremlin> }

==>groovysh_evaluate$_run_closure1@7bb3a9fe

gremlin>

gremlin> g = TinkerFactory.createModern().traversal()

==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]

gremlin>

gremlin> g.findOrCreate(1,'gender','male').teams()

==>lop

gremlin> g.findOrCreate(45,'gender','male').teams()

gremlin> g.V(45)

==>v[45]

gremlin> g.V(45).valueMap()

==>[gender:[male]]

gremlin>

*** NOTE: This only works for Gremlin-Groovy as it uses metaprogramming. What you really want to do if you want more than just a “quick and dirty” is create your own TraversalSource and Traversal. Then you would do this:

g = graph.traversal(FacebookTraversalSource.class)

g.findOrCrete(…).teams()

If you do it like that, then any Gremlin language variant — Gremlin-Java, Gremlin-Python, Gremlin-Groovy, etc. can use your DSL.

HTH,

Marko.

http://markorodriguez.com

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/5723ca3b-0afb-4d94-bc87-d7eeb7c1f9bb%40googlegroups.com.

[ern...@rocketjourney.com]

Actually, I did try to go that way during the weekend, I successfully extended TraversalSource, but my gremlin-server was having trouble loading the extension (using titan), I'll try using a plain TinkerPop graph and JUnit.  Can you point us on some file that successfully extends TraversalSource?    

I intend to make some tutorials in Spanish, maybe some video courses, it might be helpful to see some successful example of a DSL implementation. I believe graph dbs will be huge market in the following decade.

[David Kul]

Thanks for the help and the examples.

I'll definitely give meta-programing a go, at least as a temporary solution.

But I'm also interested in some examples of files implementing TraversalSource, to make my steps fully implemented.

[David Kul]

Also, I might not have been very clear: I don't really want to implement "TraversalSource", I'd much rather extend "GraphTraversalSource".

I think that's what the other person meant as well.

That way, I wouldn't have to copy all the code and update my class everytime a new version of gremlin is released.

The problem is that the constructor and other important methods are "private"... Any chance you might change these to "protected" in the next version ?

[Marko Rodriguez]

Hi,

Also, I might not have been very clear: I don't really want to implement "TraversalSource", I'd much rather extend "GraphTraversalSource”.

So you can not just extend GraphTraversalSource/GraphTraversal because of the way in which generics work in Java. What you can do is implement GraphTraversal (its an interface) and then override each method to return your Traversal’s type.

If you want to make your own DSL, you do as GraphTraversalSource/GraphTraversal. It is very important to use TinkerPop steps so that the compiler can optimize the traversal else you will have to write your own optimizers — and providers optimizers (e.g. index lookups with g.V().has(‘name’,’blah’)) expect TinkerPop steps. The model I have used in the past is simply:

public class MyTraversal<S,E> implements Traversal.Admin<S,E> {

private GraphTraversal.Admin<S,E > rawTraversal;

public MyTraversal<S,Vertex> find(Map<String,Object> properties) {

rawTraversal.V();

for(Entry entry : properties) {

rawTraversal.has(entry.key(), entry.value())

}

return this;

}

public MyTraversal<S,Vertex> friends(float minStars) {

rawTraversal.outE(“knows”).has(“stars”,gt(minStars));

return this;

}

public E next() {

return this.rawTraversal.next();

}

...

}

Why do it like this? Because GraphTraversal has all the steps and exposes a cleaner API than dealing with “new VertexStep(…)” and you are guaranteed to be using TinkerPop3 steps which is important for compilation.

HTH,

Marko.

http://markorodriguez.com

[ern...@rocketjourney.com]

It looks like a pretty big interface, do we have to implement all of this methods just to make a simple DSL? My compiler complains because of the huge amount of methods I haven't implemented. There is nothing like a default implementation?   I'm not sure what some of the methods are supposed to do...

static interface Admin <S, E> extends org.apache.tinkerpop.gremlin.process.traversal.Traversal<S,E> {
    default void addStarts(java.util.Iterator<org.apache.tinkerpop.gremlin.process.traversal.Traverser.Admin<S>> starts) { /* compiled code */ }

    default void addStart(org.apache.tinkerpop.gremlin.process.traversal.Traverser.Admin<S> start) { /* compiled code */ }

    java.util.List<org.apache.tinkerpop.gremlin.process.traversal.Step> getSteps();

    default <E2> org.apache.tinkerpop.gremlin.process.traversal.Traversal.Admin<S,E2> addStep(org.apache.tinkerpop.gremlin.process.traversal.Step<?,E2> step) throws java.lang.IllegalStateException { /* compiled code */ }

    <S2, E2> org.apache.tinkerpop.gremlin.process.traversal.Traversal.Admin<S2,E2> addStep(int i, org.apache.tinkerpop.gremlin.process.traversal.Step<?,?> step) throws java.lang.IllegalStateException;

    default <S2, E2> org.apache.tinkerpop.gremlin.process.traversal.Traversal.Admin<S2,E2> removeStep(org.apache.tinkerpop.gremlin.process.traversal.Step<?,?> step) throws java.lang.IllegalStateException { /* compiled code */ }

    <S2, E2> org.apache.tinkerpop.gremlin.process.traversal.Traversal.Admin<S2,E2> removeStep(int i) throws java.lang.IllegalStateException;

    default org.apache.tinkerpop.gremlin.process.traversal.Step<S,?> getStartStep() { /* compiled code */ }

    default org.apache.tinkerpop.gremlin.process.traversal.Step<?,E> getEndStep() { /* compiled code */ }

    void applyStrategies() throws java.lang.IllegalStateException;

    org.apache.tinkerpop.gremlin.process.traversal.TraverserGenerator getTraverserGenerator();

    java.util.Set<org.apache.tinkerpop.gremlin.process.traversal.traverser.TraverserRequirement> getTraverserRequirements();

    default void reset() { /* compiled code */ }

    void setSideEffects(org.apache.tinkerpop.gremlin.process.traversal.TraversalSideEffects traversalSideEffects);

    org.apache.tinkerpop.gremlin.process.traversal.TraversalSideEffects getSideEffects();

    void setStrategies(org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategies traversalStrategies);

    org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategies getStrategies();

    void setParent(org.apache.tinkerpop.gremlin.process.traversal.step.TraversalParent traversalParent);

    org.apache.tinkerpop.gremlin.process.traversal.step.TraversalParent getParent();

    org.apache.tinkerpop.gremlin.process.traversal.Traversal.Admin<S,E> clone();

    boolean isLocked();

    java.util.Optional<org.apache.tinkerpop.gremlin.structure.Graph> getGraph();

    void setGraph(org.apache.tinkerpop.gremlin.structure.Graph graph);

    default boolean equals(org.apache.tinkerpop.gremlin.process.traversal.Traversal.Admin<S,E> other) { /* compiled code */ }
}

[Marko Rodriguez]

Hello,

Please see DefaultTraversal. Also, please look at the GraphTraversal package — dsl.graph. Emulate the pattern there.

Marko.

http://markorodriguez.com

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/89272f26-3b1b-4248-882f-0f3d40222aae%40googlegroups.com.

[David Kul]

Alright. It looks a bit "heavy" though.

I feel like it would be easier if some of the classes could be extended. I only need to route a step or two for specific index queries.

I don't really want to reimplement the whole dsl. I'd rather rely on the original clean Tinkerpop3 classes and methods, as much as possible.

I'll try to make it work with your example.

[Marko Rodriguez]

Hi,

Yes, it is “heavy.” We haven’t officially published how to create your own DSL (or extend existing DSLs). Thus, if you have a good model, please propose.

Thanks,
Marko.

http://markorodriguez.com

> --
> You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/b338a33c-8bba-49e1-9ec5-d846cf99fd9a%40googlegroups.com.

[Geoff Reedy]

I've implemented a code generation based solution for combining and extending tinkerpop DSLs where you write an interface specifying just the additional methods you are adding and an annotation processor generates a new interface with all the forwarding methods with down-casts to keep the traversals in your custom DSL. It also generates a __ class for anonymous traversals in the DSL. For example the social DSL implementation looks something like this:

@TraversalDSL
public interface SocialDSL<S,E> extends GraphTraversal<S,E> {

  /* Use this if you need an anonymous instance of this traversal DSL; can't just use __ because it doesn't exist yet */
  public default <A> SocialDSL<A, A> __();

  public default GraphTraversal<S, Vertex> know(final String personName) {
    return out("knows").hasLabel("person").has("name", personName);
  }

  /* ... */
}

which generates something like

public interface SocialTraversal<S, E> extends SocialDSL<S,E> {

  @Deprecated
  public default <A> SocialDSL<A, A> __() { return __.start(); }

  public SocialTraversal<S, Vertex> know(final String personName) {
    return (SocialTraversal<S, Vertex>)SocialTraversal.super.know(personName);
  }

  /* ... */

  public SocialTraversal<S, Vertex> out(final String... edgeLabels) {
    return (SocialTraversal<S, Vertex>)GraphTraversal.super.out(edgeLabels);
  }

  /* ... */
}

along with an implementation of __, DefaultSocialTraversal. Right now the SocialTraversalSource has to be written by hand, there were some issues with the implementation of GraphTraversalSource that made it difficult to extend that just to instantiate a new traversal type.

Unfortunately it's unlikely I'd be able to share the code for this, but I'd be keen on seeing something like this be part of tinkerpop proper.

[kaniska Mandal]

quick questions :

org.apache.tinkerpop.gremlin.structure.Graph  -> provides a method ...  
   traversal(final TraversalSource.Builder<C> sourceBuilder) 

Can you provide any example how to create traversal source builder and register with gremlin server , so that I can directly send the 'custom query' to graph server (do we need to build Titan 1.0.0 source code with Tinkerpop 3.1.0 ? )

My goal is to send the expressions like ---  users().withFriends().connectedWithNetwork() to Gremlin .

Much appreciate any help.

Thanks

Kaniska

[Stephen Mallette]

As i mentioned at the start of this thread, if you are using Gremlin Server I don't see a reason to not just do some groovy metaprogramming to get this to work. It's pretty simple and can be implemented in a variety of ways. Easiest way? Just write your metaprogramming code directly into a Gremlin Server init script. Using the standard Gremlin Server zip distribution I edited the generate-modern.groovy file to include this line at the start:

GraphTraversal.metaClass.knows = { delegate.out('knows') }

then i start the server with:

bin/gremlin-server.sh conf/gremlin-server-rest-modern.yaml

Then a simple curl:

$ curl "http://localhost:8182?gremlin=g.V(1).knows().values('name')"

{"requestId":"66c2c929-9cc2-491d-acb0-ae4d7eef6b00","status":{"message":"","code":200,"attributes":{}},"result":{"data":["vadas","josh"],"meta":{}}}

If you have a complex DSL you might not want all that logic trapped in a groovy script. Easy enough - just build a standard groovy project, construct a jar, include some form of static initializer to call your metaprogramming code and place it on Gremlin Server's path. then your init script just needs to call that static initializer and your DSL is loaded.

Until we really nail down how we want DSLs to work across GLVs I think that this is the easiest way to do what you're trying to do.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/1fd6295d-8088-4b9e-a657-ad7cc17fcc5d%40googlegroups.com.

[Vignesh Ganapathy]

There seems to be 3 options for providing custom functionality

- Use groovy metaprogramming to update the gremlin server

- Extend GraphTraversal to implement it as a custom step (much more heavy duty)

- Put all the custom functionality in a plugin and use it? How does this compare to the other 2 options?

[NAGARAJA UPADYAYA]

Hi,

I am new to Gremlin and Orient and I have requirement for implementing something similar.  Please share the details or references wrt how to register a DSL or user defined step in a Gremlin server so that it is can be used in Gremlin query executed in Studio, gremlin console or any java application send Gremlin query as a string to Orient DB server.

Thank you in advance :)

Best Regards,
Nagaraj