Can Gremlin-Python work with thread locals?

[Dave vU]

I am trying to use Tinkerpop via the gremlin python module, in a flask application, but from what I can see, creating and using graph traversal objects in each request, leads to a crash as soon as two concurrent requests are handled:

[Tue Dec 06 01:11:34.062338 2016] [wsgi:error] [pid 15537] [remote ::1:38744]     remote_traversal = self.remote_connection.submit(traversal.bytecode)

[Tue Dec 06 01:11:34.062373 2016] [wsgi:error] [pid 15537] [remote ::1:38744]   File "/lib/python3.5/site-packages/gremlin_python/driver/driver_remote_connection.py", line 53, in submit

[Tue Dec 06 01:11:34.062402 2016] [wsgi:error] [pid 15537] [remote ::1:38744]     traversers = self._loop.run_sync(lambda: self.submit_traversal_bytecode(request_id, bytecode))

[Tue Dec 06 01:11:34.062435 2016] [wsgi:error] [pid 15537] [remote ::1:38744]   File "/lib/python3.5/site-packages/tornado/ioloop.py", line 456, in run_sync

[Tue Dec 06 01:11:34.062461 2016] [wsgi:error] [pid 15537] [remote ::1:38744]     raise TimeoutError('Operation timed out after %s seconds' % timeout)

[Tue Dec 06 01:11:34.062530 2016] [wsgi:error] [pid 15537] [remote ::1:38744] tornado.ioloop.TimeoutError: Operation timed out after None seconds

Is this a known limitation of the module?

Is there any way to use gremlin python in a WSGI-type server? 

Thanks in advance…

-- 

Dave

[Dave vU]

Answering my own question:

gremlin-python should work fine with threads, but there is a bug in the current code that makes it ignore the 'loop' parameter when it's there (and puts all calls on the same loop when it isn't, leading to crashes when running many threads):

At line 43 in driver_remote_connection.py (function `DriverRemoteConnection`):

        if loop is None:
            self._loop = ioloop.IOLoop.current()

should be:

        if loop is None:
            self._loop = ioloop.IOLoop.current()
        else:
            self._loop = loop

I would gladly submit a bug report, but do not see a public issue tracker (is there one somewhere?)…

-- 

Dave

[Stephen Mallette]

TinkerPop uses JIRA: 

https://issues.apache.org/jira/browse/TINKERPOP/ 

If you'd like to submit a pull request with your fix that would be cool too. You can review our contributing guide here:

http://tinkerpop.apache.org/docs/current/dev/developer/#_contributing

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/da68a55c-4b93-4dc8-9075-4266eaacd055%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Marko Rodriguez]

Hi,

If you could also provide a threaded test case in test_driver_remote_connection.py, that breaks without you change and works with it, that would be slick.

Note that I made a JIRA ticket for you so all you have to do is make a PR to it:

https://issues.apache.org/jira/browse/TINKERPOP-1581

You can see the pattern for how we do PRs at TinkerPop by looking at few examples:

https://github.com/apache/tinkerpop/pull/511

https://github.com/apache/tinkerpop/pull/506

https://github.com/apache/tinkerpop/pull/512

NOTES:

1. Ticket number and title is the name of the PR.

2. Ticket URL is the first thing in the description of the PR.

Thanks,

Marko.

http://markorodriguez.com

To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H4393rohUvF7%3DOak7i_bKLUacd_LRhCNcXhAfiiqNAypkVA%40mail.gmail.com.

[Dave vU]

Hi Marco,

I will gladly submit a PR as soon as I have 5 mins (I initially thought the git repo was a read-only mirror and wasn't sure how PR could be submitted).

As for a threaded test case, I can look into it, but it will be a bit more involved: I managed to produce a reduced test-case for my own purpose here, but it still required using mod_wsgi-express to run a mini-server… Not sure how that can be done easily in a unit test (if you have any suggestions). I guess the solution is to manually create two threads and have each submit some graph commands… But I am not very familiar with that use of Python threading lib (I'll have a look).

Cheers

-- 

Dave

[Stephen Mallette]

the mirror sorta confuses people unfortunately. it is readonly but you can still submit a pull request as you normally would. we just can't merge it through the github ui. an apache commiter has to ultimately just merge/push it manually to apache git (which then mirrors back to github).

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/fd96dbe9-cd7f-4cad-b2af-e479c89b433d%40googlegroups.com.

[David Brown]

I've been using gremlin-python with threads and multiprocessing lately, so I went ahead and opened a PR with some multi-threaded tests: https://github.com/apache/tinkerpop/pull/515. Thanks for bringing this up Dave. 

To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.