is it possible to get details from gremlin-server about queries as a provider plugin?

[Thomas Kolanko]

We are trying to investigate how to get things like which specific queries succeeded / failed and what was the reason for the failure and do it all by graph. We looked into the metrics reporting but it doesn't include traversal names (as far as we can see).

We can see that the GremlinExecutor builder has afterSucces/afterFailure etc consumers that look promising but as far I can tell there is no way to hook into them since the ServerGrmelinExecutor builds it's GremlinExecutor https://github.com/apache/tinkerpop/blob/master/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/util/ServerGremlinExecutor.java#L119 when it gets called from GremlinServer

Our current initialization is done as a custom WebSocketChannelizer. In it's init method we start our graph manager and pass it the executor. I was wondering if there is anyway to update or append to those afterSuccess/afterFailure hooks so we could do our own per query by graph tracking

[Stephen Mallette]

i dont think there is an easy way to do that. you would have to build your own OpProcessor or something similar I would think and might add a fair bit of copy/paste code.

> We are trying to investigate how to get things like which specific queries succeeded / failed and what was the reason for the failure and do it all by graph. We looked into the metrics reporting but it doesn't include traversal names (as far as we can see).

In general, this seems like a fairly useful feature. It might be easier if you considered contributing that change to Gremlin Server and then just consuming it for its next release.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/406128fe-c42f-4bda-8091-de927f3b5f46n%40googlegroups.com.

[Thomas Kolanko]

I'm taking a look at implementing this but wanted to run the approaches I could think of by you:

The first thing I tried was a listener approach. I added the ability to set listeners on GremlinExecutor that match the life cycle methods like beforeEval and withResult. 

In TraversalOpProcessor I get the gremlin executor from the context, iterate through any added listeners and call the matching lifecycle method. In AbstractEvalOpProcessor I added similar calls when the LifeCycle gets built.

In our provider we then create a class that implements the listener and call gremlinExecutor.addListener(...). This seems to work but there's no 1:1 matching arguments between TraversalOpProcessor and AbstractEvalOpProcessor that could call the listeners

The other approach I tried is to create a new setting called queryNotifications (for now) that works similar to graphManager where I can provide a class I want to use to deal with notifications. In Context I conditionally create the instance if the setting is present and provide a getter for it. Then I follow a similar pattern in AbstractEvalOpProcessor and TraveraslOpProcessor but I get the notification class from context and add a null check before calling it.

I'm not sure if you have a preferred approach or if my description of both approaches make senses or if there is a different way to approach this

You received this message because you are subscribed to a topic in the Google Groups "Gremlin-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gremlin-users/UbsEsYSaWPM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H439p8f_b2VYiky4uz-n1L-tLa%3DEq20DRLc_5WmPfxHEdhg%40mail.gmail.com.

[Thomas Kolanko]

just had another idea; I added the following to the GraphManager interface:

default void beforeQueryStart(String requestID) {}
default void onQueryError(String requestID, String error) {}
default void onQuerySuccess(String requestID) {}
default void onQueryTimeout(String requestID) {}

In AbstractEvalOpProcessor and TraversalOpProcessor I get the graph manager from context then call the appropriate methods as needed.

[Stephen Mallette]

I think your GraphManager approach is nice in the sense that it's already operating within the context of the query execution given the pluggable transaction semantics it has. 

You originally wrote that you wanted to know "which specific queries succeeded / failed and what was the reason for the failure and do it all by graph". Could you say some more about the use case you have in mind? What do you expect to do with your implementation of GraphManager when each of those four methods are called?

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CABFoXOr5PDP_7owdvV1zC1P%2BdUDTWzdHz-Sp4Euws2HFWN-ifQ%40mail.gmail.com.

[Thomas Kolanko]

> I think your GraphManager approach is nice in the sense that it's already operating within the context of the query execution given the pluggable transaction semantics it has. 

Ok great, that was the easiest one to implement

> Could you say some more about the use case you have in mind? What do you expect to do with your implementation of GraphManager when each of those four methods are called?

I work on IBM Db2 Graph which is a 3.6.0 implementation of Apache Tinkerpop. I'm investigating the ability to show a dashboard on our user interface where you can monitor the health of your server. For example, some charts showing query executions with success / failure rates. Right now we know when a query starts because our strategies take the gremlin steps and convert them to SQL so they can be run against Db2, however, unless there is a failure in the SQL processing we don't know if a query succeeded or failed. 

We tried implementing an end of query step but it was problematic because the step receives an iterator from the previous step and we don't know how many items are in that iterator. The previous step may send the first item of the iterator because the previous step has finished completely, this puts us in a scenario where the previous step may still need to process items in SQL. 

If this change is accepted we are planning to use the methods as part of a metrics tracking system to build the dataset needed for a health dashboard.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H43_BN7pv7T2L_YsMSqhDw6dYtwgvHqj7%2BJgBtqRruXfb6g%40mail.gmail.com.

[Stephen Mallette]

Interesting - thanks for the information. I suppose this approach will work to satisfy what you want to do. I guess what I'm still thinking about is whether or not there are other use cases we might satisfy and if these interface additions you suggested are the right signatures to do the widest range of things you might want to try to do. 

I see you created a ticket. Would you mind writing a DISCUSS thread for the dev list to draw some attention there to what you propose to do (and reference the JIRA)? If I have any other thoughts I can respond there along with what other might have to add.  

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CABFoXOp%3D%3DuobAqxRR4OXDMxTy4vc_AqndhW0QXdgQur%3D6UtfRw%40mail.gmail.com.