Gremlin Server clients for Python programmers

[David Brown]

Hello everyone,

Over the past months I have published 3 different Python clients for the Gremlin Server: aiogremlin [1], gremlinrestclient [2], and gremlinclient [3]. This may seem like a lot of work to accomplish basically the same thing, so today I would like to explain the reasoning behind each of these libraries. I first became interested in the Gremlin Server because my dissertation will use a graph database for data storage and analysis. I had used Neo4j quite a bit at work, but I was interested in something a bit more flexible for analysis. When I saw the new TInkerpop3 stack I thought it would be a perfect fit, so I set out to develop a driver using my language of choice: Python.When I saw that the Gremlin Server used websockets for communication I decided to take the opportunity to learn the new Python Asyncio [4] module that was introduced in the standard library with the Python 3.4 release. As this was more of a personal project, I didn't really worry about the fact that this would only work with Python 3, which unfortunately has still not been fully adopted by the Python community (I understand why, and this isn't a criticism). I didn't want to implement the websocket protocol from scratch, so I decided to use the aiohttp [5] implementation, as this one of the "sexiest" new Python libraries for asynchronous IO. This worked out great, and it gave me the opportunity to contribute to the websocket client implementation of this library. Hence, aiogremlin was born. To my delight, some other Python developers started experimenting with the library.

Later, we decided at my lab that we wanted to add the Gremlin Server as an optional backend for SylvaDB [6], my lab's graph database management system. SylvaDB previously supported Neo4j and Rexter, but we were intrigued by the possibility of plugging in any of the Tinkerpop vendor implementations into the system. Due to the fact that SylvaDB is a Python 2.7 Django based application, aiogremlin would not be compatible. So, over a period of a couple days I implemented a simple client for Gremlin Server using the REST architecture. This new client, gremlinrestclient, was based on Requests [7], the de facto standard library for HTTP with Python. This gave Python 2 developers an option for using the Gremlin Server, but it limited them to HTTP.

Later, in a conversation with Cody Lee, who developed the Mogwai ORM [8] for Rexster, we decided that the Python community needed an option that was compatible with Python 2 and Python 3 and allowed the user to choose between websockets and HTTP. Furthermore, we wanted a library that was compatible with the Trollius [9] style coroutine syntax, the Tornado [10] asynchronous web development framework, the current Asyncio coroutines, and the future Python asynchronous coroutines introduced in PEP 0492 [11]. This client could then serve as the basis for a new version of Mogwai that was flexible depending on user needs. To do this, we decided to use the Tornado implementation of the websocket protocol to create a client that uses pure callbacks internally (avoiding coroutines) and returns futures. This allows the user to choose the best coroutine style based on the tool kit that they are already using. Thus, gremlinclient, which is still pre-alpha, was born. In the following weeks you can expect to see new releases of gremlinclient and, hopefully soon, a new version of Mogwai. 

Phew, that was a really long explanation, so in summary:

aiogremlin is a Python 3 library based on asyncio and aiohttp that uses websockets to communicate with the Gremlin Server.

gremlinrestclient is a Python 2/3 library that uses HTTP to communicate with the Gremlin Server.

gremlinclient is an asynchronous Python 2/3 client for the Gremlin Server that allows for flexible coroutine syntax--Trollius, Tornado, Asyncio--and (will) support both websockets and HTTP.

If you have any more questions, please do not hesitate to ask.

Dave

1. aiogremlin - http://aiogremlin.readthedocs.org/en/latest/

2. gremlinrestclient - http://gremlinrestclient.readthedocs.org/en/latest/

3. gremlinclient - https://github.com/davebshow/gremlinclient

4. Asyncio - https://docs.python.org/3/library/asyncio.html

5. Aiohttp - http://aiohttp.readthedocs.org/en/stable/

6. SylvaDB - http://sylvadb.com/

7. Requests - http://docs.python-requests.org/en/latest/

8. Mogwai - https://github.com/platinummonkey/mogwai

9. Trollius - http://trollius.readthedocs.org/en/latest/using.html

10. Tornado - http://www.tornadoweb.org/en/stable/

11. PEP 0492 - https://www.python.org/dev/peps/pep-0492/

[Cody Lee]

Yeah! Rock on!

Cody

[Stephen Mallette]

Thanks for the update, Dave - this was a nice post. I've updated the TinkerPop home page to reflect the revised descriptions of your work and to include "gremlinclient".  Also, I'm looking forward to hearing about a Mogwai release for TP3 when that gets stamped out.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/a9ae28f6-c480-4ec8-8c94-12f4b601d5d4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jason Plurad]

+1 nice writeup Dave. Sounds like you're on the way to becoming the supernode of Python graph development. It will be a fun ride.

[Kevin Adda]

Hi everyone,

Very nice right up David!

I myself worked on Titan graphs 2.5 years ago. Lacking of a good python driver and object mapper, we began working on one we called graphalchemy, which we put aside quickly to wait for a 1.x version of titan (I believe titan 0.4 was released at that time). We finally waited a bit more, and are looking back at graphs now.

We are thus looking for a python 2 tinkerpop3 ready python driver and object mapper. I can see the running issues and branches on the repositories you linked above, but it is not clear to me what is done and what is not.

I would love to know how far you have got with the integration of gremlinclient into mogwai, what is left to do: if the timing is right, we would be happy to help with it.

[David Brown]

Hi Kevin,

To answer your question, gremlinclient is still in alpha, but it's moving along. It is now working Python 2.7-3.5 using either Tornado, Trollius, or Asyncio (including PEP 492). I actually released the 0.0.2 version today, which features some simple classes and actual examples in the README. While the client isn't yet feature complete--no ssl/sasl or connection pooling--the API is stable and I think the basics of the internals are solid enough to begin integration with Mogwai; however, as I far as I know, nothing has been done yet. Dropping gremlinclient in as a replacement for rexpro should be pretty straightforward, but I expect the rest of the library will need considerable refactoring to deal with the futures returned by gremlinclient. I definitely would like to get this project going, but we are all busy, so any help would be greatly appreciated. I am always happy to receive and review pull requests, and I expect Cody Lee (@platinummonkey) would say the same. I will definitely be working on the client here a bit more over the next week, but any feedback there is helpful too--feel free to open issues or submit PRs. Definitely let us know if we can help with anything or if you have any questions.

Dave

[Stephen Mallette]

kevin, if you want to get into more of the development discussion around tinkerpop and the drivers/gremlin server i'd encourage you to subscribe to the dev list:

http://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/70b4aaf7-4d14-43cd-ba40-f2c9404d723a%40googlegroups.com.

[Kevin Adda]

David, thanks for the clarifications, I'll dive into the code between today and next monday, I will get back at you if required or allow myself to make a pull request and/or create issues.

Stephen, thanks, I just did!

[nikhil....@innoplexus.com]

Hi David,
Do these libraries support neo4j database

[Stephen Mallette]

They support TinkerPop which is an abstraction over different graph databases. Neo4j is one such graph database that is TinkerPop-enabled and therefore will work with those libraries. Note that there is effort currently underway within TinkerPop to directly include a Gremlin Language Variant for python. You can read more about Gremlin Language Variants here:

http://tinkerpop.apache.org/docs/3.2.1-SNAPSHOT/tutorials/gremlin-language-variants/

--

You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/d7139c31-278e-4d7f-9d7b-502177127bb7%40googlegroups.com.

[ypanc...@relpro.com]

Hi David, I am new to graph databases and I am trying to connect to Gremlin server using gremlin-python. I am using this code:  

from gremlin_python import statics
from gremlin_python.structure.graph import Graph
from gremlin_python.process.graph_traversal import __
from gremlin_python.process.strategies import *
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
graph = Graph()
g = graph.traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/','g'))

But, I am getting this error:

C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\python3.exe C:/Users/Yash/IdeaProjects/new/gremlin.py
Traceback (most recent call last):
  File "C:/Users/Yash/IdeaProjects/new/gremlin.py", line 10, in <module>
    g = graph.traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/','g'))
  File "C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\lib\site-packages\gremlin_python\driver\driver_remote_connection.py", line 45, in __init__
    password=password)
  File "C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\lib\site-packages\gremlin_python\driver\client.py", line 76, in __init__
    self._fill_pool()
  File "C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\lib\site-packages\gremlin_python\driver\client.py", line 88, in _fill_pool
    conn = self._get_connection()
  File "C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\lib\site-packages\gremlin_python\driver\client.py", line 101, in _get_connection
    self._transport_factory, self._executor, self._pool)
  File "C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\lib\site-packages\gremlin_python\driver\connection.py", line 40, in __init__
    self.connect()
  File "C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\lib\site-packages\gremlin_python\driver\connection.py", line 46, in connect
    self._transport.connect(self._url)
  File "C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\lib\site-packages\gremlin_python\driver\tornado\transport.py", line 33, in connect
    lambda: websocket.websocket_connect(url))
  File "C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\lib\site-packages\tornado\ioloop.py", line 458, in run_sync
    return future_cell[0].result()
  File "C:\Users\Yash\AppData\Local\Programs\Python\Python36-32\lib\site-packages\tornado\concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
ConnectionRefusedError: [Errno 10061] Unknown error

Can you help me out ?

[Stephen Mallette]

This question is on SO:

https://stackoverflow.com/q/50864225/1831717

As i questioned there, are you certain that Gremlin Server is actually running? can you connect from Gremlin Console?

--

You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/0b9d366b-0dce-468d-9a07-68aa831b392c%40googlegroups.com.

[ivaylo frankov]

Hi All,

I have a similar problem, but I receive connection the problem is that after the connection I am not able to get and set Vertexes

I downloaded prebulded 0.3.0 RC from 

https://groups.google.com/forum/#!topic/janusgraph-dev/0ng5lHio4TQ

and I tried to make a connection with python 

I installed gremiln-python 3.3.2 (previously I used 3.2.6 with 0.2.0 Release )

I loaded GraphOfTheGodsFactory.load(graph)

and I tried in gremiln console :  

gremlin> saturn = g.V().has('name', 'saturn').next()

==>v[4096]

After that I tried the same in python:

print("Test")

from gremlin_python import statics
from gremlin_python.structure.graph import Graph
from gremlin_python.process.graph_traversal import __
from gremlin_python.process.strategies import *
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection

import time


from gremlin_python.process.traversal import T
from gremlin_python.process.traversal import Order
from gremlin_python.process.traversal import Cardinality
from gremlin_python.process.traversal import Column
from gremlin_python.process.traversal import Direction
from gremlin_python.process.traversal import Operator
from gremlin_python.process.traversal import P
from gremlin_python.process.traversal import Pop
from gremlin_python.process.traversal import Scope
from gremlin_python.process.traversal import Barrier

graph = Graph()
g = graph.traversal().withRemote(DriverRemoteConnection('ws://localhost:8182/gremlin','g'))
print("start")
saturn = g.V().has('name', 'saturn').next()
print("end")

and I received 

/home/ivo/anaconda3/envs/tensorflow/bin/python /home/ivo/PycharmProjects/test2/testp.py
Test
start
Traceback (most recent call last):
  File "/home/ivo/PycharmProjects/test2/testp.py", line 24, in <module>
    saturn = g.V().has('name', 'saturn').next()
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/site-packages/gremlin_python/process/traversal.py", line 70, in next
    return self.__next__()
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/site-packages/gremlin_python/process/traversal.py", line 43, in __next__
    self.traversal_strategies.apply_strategies(self)
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/site-packages/gremlin_python/process/traversal.py", line 346, in apply_strategies
    traversal_strategy.apply(traversal)
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/site-packages/gremlin_python/driver/remote_connection.py", line 143, in apply
    remote_traversal = self.remote_connection.submit(traversal.bytecode)
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/site-packages/gremlin_python/driver/driver_remote_connection.py", line 54, in submit
    results = result_set.all().result()
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/site-packages/gremlin_python/driver/resultset.py", line 81, in cb
    f.result()
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/site-packages/gremlin_python/driver/connection.py", line 77, in _receive
    self._protocol.data_received(data, self._results)
  File "/home/ivo/anaconda3/envs/tensorflow/lib/python3.6/site-packages/gremlin_python/driver/protocol.py", line 71, in data_received
    result_set = results_dict[request_id]
KeyError: None

I checked it is the right Version of Gremlin-Python 3.3.2

Would you mind to tell me how to make it works?

Thank you very much in advance!

Cheers,

Ivo

[Florian Hockmann]

You already asked exactly the same question in the JanusGraph developers group and I tried to help you there: https://groups.google.com/forum/#!forum/janusgraph-dev

Did you try what I suggested there regarding configuration of the GraphSON serializers?