[TinkerPop3] Remote Scripts in Gremlin Console

[Stephen Mallette]

In TinkerPop2 we had Rexster Console to issue scripts to be executed in a remote Rexster Server.  It was basically a window into a Gremlin ScriptEngine session on the server.  In TinkerPop3, we've been trying to get away from having lots of consoles (Gremlin Console, Rexster Console, if you use Titan, there's a flavor for that, etc.) as it leads to much confusion.  To unify the consoles we needed a robust plugin system which existed to some degree in TP2 and a way submit scripts remotely in Gremlin Server instead of having them just evaluate locally (which is pretty much what happens when you hit enter at the end of the console line).

To do this we introduced two custom commands to the console: :remote and :submit (or :>).  you use :remote to configure the console to connect to a particular Gremlin Server and you use :> to send scripts to it.  So by default, the console is configured to connect to a Gremlin Server on "localhost":

gremlin> :remote

==>remote - results as text [localhost/127.0.0.1:8182]

You could alter that with:

gremlin> :remote connect 192.168.1.1

==>connected - 192.168.1.1/192.168.1.1:8182

Alternatively, I could have specified a properties file containing my connection information.  With that connection in place and Gremlin Server running I can do:

gremlin> :> 1+1

==>Item{resultItem=2 class=java.lang.String}

gremlin> :> g

==>Item{resultItem=tinkergraph[vertices:0 edges:0] class=java.lang.String}

The Item object is just a wrapper for a result that comes back from Gremlin Server.  Note that in both results class is a String.  By default, the console requests that Gremlin Server toString all results, for viewing purposes.  You can change that with:

gremlin> :remote as objects

==>results as objects

with the caveat being that Gremlin Server knows how to serialize the results you are requesting (there are configurations for such things).  TinkerGraph is not one of those things it can serialize, but an integer should work just fine:

gremlin> :> g

==>Server could not serialize the result requested. Server error - Error during serialization: Class is not registered: com.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph

gremlin> :> 1+1

==>Item{resultItem=2 class=java.lang.Integer}

Graph elements serialize to standalone objects (note that a CachedVertex is disconnected from its source Graph instance - still thinking on that):

gremlin> :> g.addVertex(Element.ID, 100, "name", "stephen")

==>Item{resultItem=v[100] class=com.tinkerpop.gremlin.structure.util.cached.CachedVertex}

The nice thing about ":remote as objects" is that you can work with the data in a real way in the console.  To get that vertex you do:

gremlin> v = _l[0].getVertex()

==>v[100]

gremlin> v.getValue('name')

==>stephen

The reserved variable "_l" always represents the last result returned from a :submit.  Item has a number of methods on it for getting values out by type.  You can also specify the class, if a helper method like getVertex is not available for what the item contains.

gremlin> :> [name:"daniel"]

==>Item{resultItem=name=daniel class=java.util.AbstractMap.SimpleEntry}

gremlin> _l.get(0).get(Map.Entry.class).value

==>daniel

gremlin> _l.get(0).get(Map.Entry.class).key

==>name

Stephen

[Todd Leo]

Hi Stephen,

How do you run a gremlin script file on a remote gremlin server? Could you provide an example?

Todd Leo

[Stephen Mallette]

This is the best i have at the moment:

gremlin> myScript = new File('my-script.groovy').getText('UTF-8')

==>x = 1

y = 2

addItUp(x,y)

gremlin> :remote connect tinkerpop.server conf/remote-objects.yaml

==>Connected - localhost/127.0.0.1:8182

gremlin> :> @myScript

==>3

not sure if we could make that nicer. maybe:

gremlin> :> ~my-script.groovy

Created this to track it:

https://issues.apache.org/jira/browse/TINKERPOP-1056

Please yell if you have other suggestions...

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/1554b6c6-6a3a-4ab3-8800-231a52d24b14%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[SLiZn Liu]

Hi Stephen,

Thanks for the response and creating the issue on jira.

I tried to simplify your process by adding all codes, including :remote ... to the script, then execute from gremlin shell by bin/gremlin.sh -e ~/tmp/my-script.groovy:

// File: my-script.groovy
:remote connect tinkerpop.server ~/titan/conf/remote.yaml 
:> x = 1                                                                        
:> y = 2                                                                        
:> x+y

which yields error:

org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script1.groovy: 1: unexpected token: : @ line 1, column 1.
   :remote connect tinkerpop.server /home/users/liulx/confs/titan/conf/remote.yaml
   ^
1 error

I guess the :remote and :> commands is not valid for groovy compiler, thus script execution can’t be done without input in the gremlin shell.

---

BR,
Todd Leo

​

You received this message because you are subscribed to a topic in the Google Groups "Gremlin-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gremlin-users/sETDv5AEUl8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H438VcUmQX09e8ud8KOaVYgkT8MTU8L%3DvTLy%3DnO89awuf0A%40mail.gmail.com.

[Stephen Mallette]

Yeah - there is a bit of asymmetry there.  The -e option executes the script in the gremlin-groovy ScriptExecutor which doesn't support the notion of plugins or "console commands".  You have to do everything more manually if you want to automate something like remoting for example.  Instead of using :remote and :> you would create a Cluster/Client instance manually in your script and send the remote scripts with client.submit(). Not sure if we could change that somehow.  If you would like to see that feature, please consider creating a ticket in JIRA.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAD_PBkS6mS0S%3DnVuQn%3DTg-ru_kZzk5_zzUxpkSP7JMbaxoqy%2Bg%40mail.gmail.com.

[SLiZn Liu]

Could you also provide a minimal example of how to:

…create a Cluster/Client instance manually in your script and send the remote scripts with client.submit().

or, is there any guides that I can refer to? I’d like to test if running gremlin script remotely works and fits our data import work flow. Then let’s talk about how to improve this feature. : )

​

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H43_7E09Xtn%3DJsi_6M_NN%2BkfeNT7AsybEp-OkSKjf2qcQAA%40mail.gmail.com.

[Stephen Mallette]

http://tinkerpop.apache.org/docs/3.1.0-incubating/#_connecting_via_java

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAD_PBkTyk%2B05ypJUC6Da77%3DSOQGsMFi%3DvKqKKVapFHT4qWh%3DYg%40mail.gmail.com.

[SLiZn Liu]

I’m going to load massive dataset to Titan. The dataset is located on HDFS. After a bit of searching, I found hdfs.ls() and other provided commands can be used to load and parse the dataset. However, the hdfs object cannot be found on remote server. Do I need to import something or initiate before using? Or, is there a better way to parse the dataset on(or parsed locally and submit to) the remote server?

​

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H439UezujLLU%2BdzdvrPO9M_zZetN9S6a%3DXbYip_rZNebj8g%40mail.gmail.com.

[Stephen Mallette]

That bug was noticed recently and fixed:

https://issues.apache.org/jira/browse/TINKERPOP-860

Will be part of 3.1.1-incubating. I think you can work around it by doing what the plugin does manually (i.e. add imports, eval HadoopLoader and create the hdfs object):

https://github.com/apache/incubator-tinkerpop/blob/2f16c774ea44f9681afb58cb313d02b32376821e/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java

I'm mostly sure that will work as the bug had nothing to do with operations of those steps (and more to do with the bindings lifecycle). 

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAD_PBkTS57EagZgV%2BV0nrpfOGS6v7HfSHdJSw2oBcR1p37DQjA%40mail.gmail.com.

[SLiZn Liu]

Sad to hear that, since I'm using Gremlin that comes with Titan 1.0.0. Need to think about the necessity to fall back to load and creating graph on client side, rather than on remote server. Really appreciate for your help, Stephen.  

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H43_%2BPTOwchdZkV3Kezx-9YhSa-hEVYZ098xBEPzNYJSGJg%40mail.gmail.com.

[Mark Henderson]

Great post. I was going to ask this in a few days: what functions allow for bound parameters? I notice that aliases .as() .back() do not allow it. 

Where should I look in the source to determine this? I'm a "read the source" type of guy, and I've tried looking through it a few times with no success. 

I'm sure that I'll have a lot more questions, can we just keep this thread going?

[Stephen Mallette]

>  what functions allow for bound parameters? I notice that aliases .as() .back() do not allow it. 

not sure what version of Gremlin you are looking at but 3.x doesn't have back() anymore.  wrt to as(), why do you say you can't parameterize that?

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/0534bf60-5eb8-4d4d-9785-c19460d6fdaf%40googlegroups.com.

[Mark Henderson]

I'm not in front of my code right now, but I feel like I was getting errors when using as() and select() with bound params. If they can accept bindings, that would make things a lot easier for me. (I am using TP3, got back() mixed up with select())

[Stephen Mallette]

Please provide an example of what you were trying to do and where it failed - we can discuss from there.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/942f5b9e-1aac-4e7f-9d93-49b1a7cfcb2e%40googlegroups.com.

[SLiZn Liu]

Hi Stephan,

In TP 2, one can use the following code to determine if two vertices are connected:

gremlin> g.v(1).both.retain([g.v(3)]).hasNext()
==>true

while in Gremlin comes with Titan 1.0 (which corresponds to TP3, I presume), it doesn’t work:

groovy.lang.MissingMethodException: No signature of method: org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.DefaultGraphTraversal.retain() is applicable for argument types: (com.thi
nkaurelius.titan.graphdb.vertices.StandardVertex) values: [v[4184]]
Possible solutions: tail(), wait(), reset(), min(), getAt(int), getAt(java.lang.String)

What is the current method to achieve this?

---

BR,
Todd Leo

​

You received this message because you are subscribed to a topic in the Google Groups "Gremlin-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gremlin-users/sETDv5AEUl8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H438vfxhknEXz5gp53DoZ6jWxcrcOROh%3DU2H3AnFKOfGCWw%40mail.gmail.com.

[Daniel Kuppitz]

Either:

g.V(1).both().hasId(3).hasNext()

Or:

other = g.V(3).next()

g.V(1).both().is(other).hasNext()

Cheers,

Daniel

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAD_PBkSjV1tFd50OpOZmiHRS7ac3FneJTUCXXgbkzpN2WN77dA%40mail.gmail.com.

[SLiZn Liu]

Hi Daniel, 

I tested your code, and it works well. Thanks!

---

BR,

Todd Leo

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CA%2Bf9seUKmWCNhjr05b3FbmHEme_g94RGaDb%3DzYhLTp4TigypLw%40mail.gmail.com.

[Mark Henderson]

I cannot seem to get this script working over the wire. However, it works fine in the console with the params filled in. 

1. Is it possible to bind params within the console?
2. I hope this is okay, but can I reuse variables in a script (as seen below with VERTEX)?

gizmo_testing.V(V_OUT_ID).bothE(LABEL).as(VERTEX).inV().hasId(V_IN_ID).select(VERTEX)

{'VERTEX': 'vertex', 'LABEL': 'mapper_test_edge', 'V_OUT_ID': 282, 'V_IN_ID': 288} 

[Mark Henderson]

Okay, so what works is not binding the alias and select:

x = gizmo_testing.V(V_OUT_ID).bothE("mapper_test_edge").as("vertex").inV().hasId(V_IN_ID).select("vertex"); x

{"V_OUT_ID": 531, "V_IN_ID": 536}

So it looks like these functions do not accept bound params:

• All of the functions listed as predicates in the documentation
  
• alias "as" and "select"
• bothE, I'm sure, but haven't tested: outE, outV, out, in, inV

At this point I need to see a list of what function can and cannot accept bindings as arguments. Do they share a common class/interface in the Java code?

Thanks

[Stephen Mallette]

> Is it possible to bind params within the console?

The console is meant for interactive exploration of a graph and less for submitting the same query over and over again, so, no, you can't paramterize from a :remote 

> I hope this is okay, but can I reuse variables in a script (as seen below with VERTEX)?

that's fine

> it looks like these functions do not accept bound params

Your code doesn't show much and you don't provide an error, so all i can say is that there is no problem using parameters for virtually any parameter to a traversal.  See my example below:

gremlin> client.submit("g.V().hasId(x).bothE(z).as(y).inV().select(y)",[x:1,y:"anything",z:"knows"]).all().get()

==>result{object=e[7][1-knows->2] class=org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge}

==>result{object=e[8][1-knows->4] class=org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge}

The main restriction with parameters is that they must be serializable given the message serializer you are using (gryo, graphson, etc).  So you couldn't make a variable that held a lambda expression for example and try to pass that as a binding.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/a91bfddb-b979-4884-ad09-cab304ff095a%40googlegroups.com.

[Mark Henderson]

You are right, everything does work with bound parameters. My issue is the name of the parameters that I was passing (LABEL, VERTEX). I didn't even think of that being the case until I read through your answer.

By the way, there are no errors in the console when misusing the names for bound params. 

Thanks again,

Mark

[Stephen Mallette]

VERTEX seems ok:

gremlin> client.submit("g.V().hasId(VERTEX).bothE(z).as(y).inV().select(y)",[VERTEX:1,y:"anything",z:"knows"]).all().get()

==>result{object=e[7][1-knows->2] class=org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge}

==>result{object=e[8][1-knows->4] class=org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge}

LABEL, as a variable name, really shouldn't be a reserved word:

gremlin> client.submit("LABEL.toString()").all().get()

==>result{object=~label class=java.lang.String}

but apparently it is.  technically, the real reserved variable should be "~label".  The problem is that groovy ignores the private scope on T.LABEL and imports it to be accessible.  :/

seems like you could do this:

gremlin> client.submit("g.V().hasId(VERTEX).bothE(Label).as(y).inV().select(y)",[VERTEX:1,y:"anything",Label:"knows"]).all().get()

==>result{object=e[7][1-knows->2] class=org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge}

==>result{object=e[8][1-knows->4] class=org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge}

anyway, probably best to avoid "graph terms" in your variable names as a general practice.  i've added to the list of "invalid bindings keys" so that we should get an error back from the server for these unintended imports.  It will be part of 3.1.1-incubating:

https://git-wip-us.apache.org/repos/asf?p=incubator-tinkerpop.git;a=commit;h=ef40dce314c1f9ee48661227d992beef8cad64d0

--

You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/ea5db3b1-fcd9-4612-9d81-9cdc695f1847%40googlegroups.com.

[Kevin I]

Hi Stephen,

I was about to ask about how to execute gremlin scripts from inside the groovy shell, when I found this thread.

I am trying to write a gremlin script for testing the remote database. Queries like `g.V()` work, but using a print or println statement doesn't work as expected. For one, they don't print anything (though they work when typed in the console itself). When the print statement is used after `g.V()` doesn't even return all the vertices but instead returns null, which seems weird.

I am not a Java/Groovy guy so forgive if its obvious. Any way I can print something from a script?

Thanks.

[Stephen Mallette]

I don't follow what you're trying to do - do you mean you want to do:

gremlin> client.submit("g.V();println('test')").all().get()

?

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/8f1f0e3c-1907-4c31-80c3-864040f4db4d%40googlegroups.com.

[Kevin I]

Hi Stephen,

I'm sorry, I just realize that I am trying to print in the remote server instead of the console itself.

I am trying to execute this:

gremlin> script = new File('gremlin.groovy').getText('UTF-8')

gremlin> :> @script

`gremlin.groovy` file contains the queries in addition to a few print statements to make it verbose (like 'Retrieving all vertices now...', etc). Is there any way (or hack) I can execute those particular statements alone in the console itself? So for example, a few statements to be executed in the remote and a few to be executed in the console.

[Stephen Mallette]

No - you can't send "feedback" to the console from the server. The script goes into evaluation mode on the server and while in that mode isn't aware of the client in any way.  All you can do is write to the server output at this point. I guess you could also breakup your script into local and remote parts.  Like I wrote this test.groovy script:

:remote connect tinkerpop.server conf/remote.yaml

:> 1+1

println "finished adding"

:> 1-1

println "finished subtracting"

:remote close

:x

then issued it with: 

bin/gremlin.sh test.groovy

it errors on :x but i think that's a bug that doesn't hurt anything:

https://issues.apache.org/jira/browse/TINKERPOP-1092

Not a great solution to your problem depending on what you're doing with transactions and such, but that's the best workaround that there is right now.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/17a1aec7-d7e5-469a-9e2d-f5d9a2ecbbcd%40googlegroups.com.

[Kevin I]

Hey Stephen,

Looks like breaking up is the closest to what I want. But, everytime loading the gremlin shell loads all the plugins too which consumes quite some time.

What do you mean by writing to the server output? Is it like configuring the server to write to a file instead of trying to print in the console or something like that?

[Stephen Mallette]

I just mean that you println in your script which should write to the server console output.  Might be nicer if you had a "log" instance available so that you could write to standard logging.  If you think that would be useful, please create an issue in JIRA.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/3f15430e-d2a0-4679-802e-caa1f888eddd%40googlegroups.com.