Tinkerpop3 WebSocket Server

[MERAL Guillaume]

Hi everyone,

I'm a trainee in a visualisation research team in the Labri (Computer Science Research Lab of Bordeaux) and i've been asked to try some of the new Tinkerpop3 features.
I'm trying to interact with the websocket based gremlin server using the "Simple Web Socket Client" plugin for chrome.

I connected successfully to the server but i can't figure out what is the expected request format.

Any help would be greatly apreciated.

Thanks.

G.M.

[Stephen Mallette]

Keeping in mind that Gremlin Server is under major development right now........the basic format of an "eval" messages (meaning execute a script) is:

application/json|-{"requestId":"655BD810-B41E-429D-B78F-3CC5F3B8E9BA","processor":"","op":"eval","args":{"gremlin":"1+1"}}

it's a text based protocol where the request consists of the serialization format of the message (in this case "application/json"), followed by a delimiter of "|-", then the message.  This will return a text based result consisting of two separate responses:

655BD810-B41E-429D-B78F-3CC5F3B8E9BA>>2

655BD810-B41E-429D-B78F-3CC5F3B8E9BA>>655BD810-B41E-429D-B78F-3CC5F3B8E9BA

where the first response is the result of the script (i.e. 2) and the second is the response terminator.  So from this you know two things:

1. that each response belongs to a particular request

2. the the server is done iteration results back to the client

Text isn't so great so you could issue something more advanced with:

application/json|-{"requestId":"655BD810-B41E-429D-B78F-3CC5F3B8E9BA","processor":"","op":"eval","args":{"gremlin":"[1,2,3]","accept":"application/json"}}

You'll see that the it returns 4 results formatted as JSON (given the "accept" argument):

{"result":1, "code":200, "requestId":"655BD810-B41E-429D-B78F-3CC5F3B8E9BA","version":"1.0.0"}

{"result":2, "code":200, "requestId":"655BD810-B41E-429D-B78F-3CC5F3B8E9BA","version":"1.0.0"}

{"result":3, "code":200, "requestId":"655BD810-B41E-429D-B78F-3CC5F3B8E9BA","version":"1.0.0"}

{"result":"655BD810-B41E-429D-B78F-3CC5F3B8E9BA", "code":200, "requestId":"655BD810-B41E-429D-B78F-3CC5F3B8E9BA","version":"1.0.0"}

I tried issuing these requests in the chrome plugin you are using against Gremlin Server built from this branch:

https://github.com/tinkerpop/tinkerpop3/tree/gremlin-server-refactor

and it seemed to work just fine, though that plugin is a little clunky...couldn't even copy/paste the results of requests from it :/

Hope that helps you get on your way in your experiments with TinkerPop3.

Stephen

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[MERAL Guillaume]

Thank you so much for your quick answer, this is exactly what i was looking for.

GM.

[Stephen Mallette]

Just a quick note that I've merged that branch back to master, so you can just work from there going forward.

Stephen

On Wed, Apr 2, 2014 at 11:07 AM, MERAL Guillaume <guillau...@etu.u-bordeaux1.fr> wrote:

Thank you so much for your quick answer, this is exactly what i was looking for.

GM.

--

[Henry Tao]

Will Tinkerpop3 WebSocket Server support streaming.

For example, if I do g.V.has("name",CONTAINS,"xyz") and it returns 5K vertices, it would be great to immediately add a node to the browser for rendering a viz instead of waiting for the entire 5K response:

socket.on("data", function(err, d) {

  canvas.addNode(d);

});

[Stephen Mallette]

That's the idea.  The entire result set does not need to be realized on the server before items are sent back.  They are sent back as the result of the evaluated Gremlin statement is iterated.  You can also control how often Gremlin Server flushes results.  You could set a "batch size" of 10 which would bundle results in items of 10 back to the client.  Right now that is just a server configuration but I expected that value to be overriden by individual requests. 

[Henry Tao]

Thanks Stephen.  That's awesome.

[MERAL Guillaume]

Hi,

The next step of my work is to create a gremlin console for Tulip thin client (emscripten generated Javascript).

The purpose of this client is only to render a TulipGraph in a web browser (for now).

Thanks to this console, i could manipulate my graph with gremlin and then refresh the visualization to see the modifications.

I would like to be able to send gremlin commands to a gremlin server through a WebSocket connection.

How could i reproduce in Javascript the command wrapping used by the :submit command of the gremlin console ?

Regards,

Guillaume

[Stephen Mallette]

I'm not completely clear on what you are asking.  It sounds like you want to issue commands from the Gremlin Console to render a TulipGraph to your Tulip thin-client.  Maybe something similar to what I described here using gephi:

https://groups.google.com/forum/#!topic/gremlin-users/pT87rPvty0g

Is that right?  Or something else? 

--

[MERAL Guillaume]

It's very similar but unfortunately we don't have any streaming api for tulip so i will have to export my graph as a file and reload it in

the thin-client.

I need to issue gremlin requests from the thin-client.

To achieve this i want to write a REPL to communicate with a gremlin server. 

(A sort of gremlin console written in Javascript that could only process gremlin command remotely).

In short i want to have the gremlin console functionalities inside the thin-client.

[Stephen Mallette]

As far as I know, at this time there is no javascript "driver" for Gremlin Server.  I only have my java one and it's still incomplete.  I've not documented the subprotocol yet as I don't feel 100% confident that it won't change before official release.  So, if you really want to dig into it, I can only advise you to look at the code to figure out how to submit gremlin to it.  

Jean-Baptiste Musso (developer on Mogwai and Grex) has been following the TP3 docs pretty closely.  Perhaps he has some insights as to the future of JS with TP3.  You might wish to reach out to him.

Stephen

--

[jbm...@gmail.com]

Guillaume,

I believe a REPL should be fairly easy to write in JavaScript using WebSockets. If you feel like using a library rather than the native WebSocket API, I suggest using the excellent and battle tested socket.io-client library (https://github.com/Automattic/socket.io-client) which works both in Node.js and the browser.

If HTTP is fine with you, you could use gRex in the browser with browserify. It could also work as a standalone library included in a <script> tag with a couple minor tweaks.

gRex currently sends a fat, parameterized string of Gremlin-Groovy over HTTP and waits for the results. I'm unsure how I'd implement that with WebSockets, but performance-wise my guess is that I would make it possible to optionally batch-send several lines/instructions rather than send a message for each of them. WebSocket support on gRex is on the todo list, but it's a low priority for now. It shouldn't be too hard, however, to add support for that (basically, subclass RexsterClient https://github.com/gulthor/grex/blob/72f0d79853494416b3ecb73531203633cd5e5222/src/rexsterclient.js ).

Feel free to contact me, I'd be happy to discuss this topic with you.

Jean-Baptiste

[MERAL Guillaume]

Hi Jean-Baptiste,

  So, this where i am now :

•  I handled the UI and common REPL concerns with this library  : http://terminal.jcubic.pl/
  
•  I can connect with a specific command my terminal to a gremlin-server through a websocket.

 I quickly took a look at socket.io-client, grex and browserify.

socket.io-client looks very usefull for multiple connections handling. I may need something like this in a near future.

gRex seems to be exactly what i need but is it compatible with Tinkerpop3 gremlin-server ?

 The web development is something fairly new to me so feel free to correct me if i got anything wrong.

Thank you very much for your help,

Guillaume

[jbm...@gmail.com]

gRex is currently HTTP only. It could be compatible right now with TP3 Gremlin Server if HTTP is still supported out of the box. I believe however that WebSocket replaces HTTP in TP3 Gremlin Server, but I may be wrong (I haven't looked at the details much). Stephen should be able to give insights on that point.

If TP3 Gremlin Server supports handling fat strings of Gremlin-Groovy over HTTP either by default or via an extension (just like Rexster did with the Gremlin extension), gRex should work with TP3. The lib may require a patch, though, for the URL to which strings of Gremlin should be POST'ed to may be a bit different in Gremlin Server (canonical URL in TP 2.x Rexster Server with Gremlin Extension enabled is http://localhost:8182/graphs/tinkergraph/tp/gremlin). Only host, port and graph name are currently configurable in gRex.

Feel free to drop by on Freenode IRC #tinkerpop, I often hang there as Gulthor. If you're new to JavaScript and web development, I'd be happy to help.

Jean-Baptiste

[MERAL Guillaume]

As much as i know, TP3 server only handles Websocket and the text protocol looks like a JSON representation of a request with several arguments appended to a MIME type header.

[Stephen Mallette]

Gremlin Server is highly focused on websockets, so right now there is no classic HTTP/REST endpoint.  Various factors are making me rethink the websocket-only approach, in favor of a configurable protocol pipeline.  At this point I've proven the approach works as I now have a standard Java NIO pipeline tested out in addition to the websocket one (still needs more work to get it actually configurable).  

A REST endpoint pipeline is a little tricky because the core of Gremlin Server relies on messages to be streamed back to the client as a set of "frames".  For REST, I guess I'd need to aggregate the frames on the server and send them back as one response.  That highlights why I didn't really want to support REST anyway, but the nice thing about REST is that it's easy for folks to get started with and get immediate results.  It would also provide a slightly easier way for tools like gRex and other libs that depend on Rexster's Gremlin Extension to upgrade. Still thinking on the matter.

On Thu, Jul 3, 2014 at 7:53 AM, MERAL Guillaume <guillau...@etu.u-bordeaux1.fr> wrote:

As much as i know, TP3 server only handles Websocket and the text protocol looks like a JSON representation of a request with several arguments appended to a MIME type header.

--

[Stephen Mallette]

Wow - I must have the right level of abstraction in Gremlin Server (netty definitely does for its part).  That wasn't hard at all:

$ curl -v http://localhost:8182?gremlin=100

* Rebuilt URL to: http://localhost:8182/?gremlin=100

* Hostname was NOT found in DNS cache

*   Trying 127.0.0.1...

* Connected to localhost (127.0.0.1) port 8182 (#0)

> GET /?gremlin=100 HTTP/1.1

> User-Agent: curl/7.35.0

> Host: localhost:8182

> Accept: */*

> 

< HTTP/1.1 200 OK

< Content-Type: application/json

< Content-Length: 85

< Connection: keep-alive

< 

* Connection #0 to host localhost left intact

{"result":100,"code":200,"requestId":"ea5d09de-3665-4f6d-b853-15590286b89a","type":1}

My implementation is full of holes and there's no way to configure this for usage in Gremlin Server without just editing and recompiling the code, but the concept seems to hold water.

[MERAL Guillaume]

Great, this Http channel can definitely be a convenient feature for web developpers.

Maybe passing the gremlin command in a POST body  would be more appropriate?

I tested your implementation a bit : 

[gmeral@gmeral-labri tinkerpop3]$ curl -v http://localhost:8182?gremlin="1-1"                                                                                                                                                 

...                                                                                                                                                                                                                         

...                                                                                                                                                                                                                        

{"result":0,"code":200,"requestId":"71a1d1f8-832d-418f-9f55-09ec65650089","type":1}[gmeral@gmeral-labri tinkerpop3]$ 

when i receive a response from the server, my prompt is appended to it, maybe a \n is missing somewhere.

[gmeral@gmeral-labri tinkerpop3]$ curl -v http://localhost:8182?gremlin="1+1"                                                                                                                                                 

...                                                                                                                                                                                                                         

...                                                                                                                                                                                                                        

{curl: (52) Empty reply from server"                                                                                                                                                             

1-1 triggers the right answer, but 1+1 an empty reply. Why is that so ?

[Stephen Mallette]

A POST would probably be better and more inline with the standard Gremlin Server presents (i.e. POST a serialized version of a RequestMessage).  Like I said, what I have so far is just a proof of concept.  If anyone can help build that out properly, the help would be appreciated, because REST isn't super high on my priority at the moment.

As for your problem with curl, I think you just have a problem with encoding your query.  Try as:

curl "http://localhost:8182?gremlin=1%2B1"

--

[MERAL Guillaume]

I wrote a quickfix to solve cross domain issues :

In HttpGremlinEndpointHandler.channelRead

String origin = req.headers().get("Origin");

if (origin != null)

response.headers().set(ACCESS_CONTROL_ALLOW_ORIGIN, origin);

[Stephen Mallette]

Could you please offer a pull request for such things?  

--

[MERAL Guillaume]

Sure, now it's done.

[gandh...@gmail.com]

HI GM we are also struggling with websocket connection - The Titan version we are using is 3.0.1. Can you confirm which version of Titan you were able to connect with? 

Thx

Gandhi Sivakumar