(Rexpro) Getting different output from rexster-console

[Dmill]

Hi guys.

I'm running a simple g.v(1).properties command on the neo4jsample graph through rexpro from a php client. However this does not return any propertyKeys results where as doing the same through the console gives me a correct resultset. (same with .map() or .addVertex([name:'john']) who respectively don't return anything or create a vertex without properties on the client but work fine in the console)

I used tcp dump to compare messages sent and received and parsed them for readability (I can provide raw dumps if need be) :

Php client with no results:

Array
(
    [0] => 0
    [1] => 3
    [2] => 82
    [3] => Array
        (
            [0] => 49149f97-4304-4b9f-a54e-b5f4eb94a952
            [1] => 70d6f78d-6440-48cc-b84d-00d9b1b5a9e4
            [2] => Array
                (
                    [inSession] => 1
                    [channel] => 1
                )

            [3] => groovy
            [4] => g.v(1).properties
            [5] => stdClass Object
                (
                )

        )

)

To compare with "correct" console parsed packet:

Array
(
    [0] => 0
    [1] => 3
    [2] => 104
    [3] => Array
        (
            [0] => bc5488db-c5ac-4a07-88ee-c566819685d9
            [1] => eea2e5c7-bc27-4010-aa9b-677d8608f211
            [2] => Array
                (
                    [transaction] =>
                    [inSession] => 1
                    [channel] => 1
                    [isolate] =>
                )

            [3] => groovy
            [4] => g.v(1).properties
            [5] => stdClass Object
                (
                )

        )

)

The results are as follows:

Php client (has no propertyKeys):

Array
(
    [0] => 0
    [1] => 4
    [2] => 291
    [3] => Array
        (
            [0] => 49149f97-4304-4b9f-a54e-b5f4eb94a952
            [1] => 70d6f78d-6440-48cc-b84d-00d9b1b5a9e4
            [2] => stdClass Object
                (
                )

            [3] => Array
                (
                    [0] => rawVertex=Node[1]
                    [1] => class=class com.tinkerpop.blueprints.impls.neo4j.Neo4jVertex
                    [2] => rawElement=Node[1]
                    [3] => id=1
                    [4] => propertyKeys=[]
                )

            [4] => Array
                (
                    [g] => neo4jgraph[EmbeddedGraphDatabase [/tmp/neo4j-graph]]
                    [rexster] => com.tinkerpop.rexster.protocol.RexsterApplicationHolder@5aa6343d
                )

        )

)

Correct result set from console response:

Array
(
    [0] => 0
    [1] => 4
    [2] => 461
    [3] => Array
        (
            [0] => bc5488db-c5ac-4a07-88ee-c566819685d9
            [1] => eea2e5c7-bc27-4010-aa9b-677d8608f211
            [2] => stdClass Object
                (
                )

            [3] => Array
                (
                    [0] => rawVertex=Node[1]
                    [1] => class=class com.tinkerpop.blueprints.impls.neo4j.Neo4jVertex
                    [2] => rawElement=Node[1]
                    [3] => id=1
                    [4] => propertyKeys=[name]
                )

            [4] => Array
                (
                    [g] => neo4jgraph[EmbeddedGraphDatabase [/tmp/neo4j-graph]]
                    [context] => javax.script.SimpleScriptContext@257f6796
                    [rexster] => com.tinkerpop.rexster.protocol.RexsterApplicationHolder@4e19b97c
                    [out] => java.io.PrintWriter@7ae0a3f2
                )

        )

)

Does anyone have an idea where the issue could come from? Does it have to do with how the sessions are set up? I've tried loading graphs the same way via g = rexster.graphName('neo4jsample') in both the console and on the client but no difference. Also worth noting that I don't have any trouble getting properties from TinkerGraphs.
Could it be an issue with the configuration file? I'm using the default file with the neo4jsample graph enabled (that's the only change I made)

Again thanks in advance!

I also have a couple of quick simple questions:

1) in these, the response type value is 4 and I don't see anything about that in the documentation? What is it if I may ask

2) if I add vertex or edges then shut rexster down g.V will not display any vertex as if the graph were reset but the node ID increment is still in place. am I missing something?

Sorry if I'm being really slow minded here. Thanks for putting up with it.

I'm using rexter-server 2.3.0 and rexster-console 2.3.0

[Dmill]

Just realized from re reading that in my little bonus question 2) I meant that I restart rexster. Not shut it down. I can't seem to get any kind of persistent data as if I were meant to save the graph before restarting rexster

[Stephen Mallette]

I'm not really sure what's going on for you.  As a general warning, you really can't directly compare the Rexster Console serialization channel with the MsgPack serialization channel.  Rexster Console uses a serialization channel that simply does a toString of everything that it finds.  Clients using MsgPack serialization have limits to what will cleanly serialize back.  In this case, however, I'm not seeing an issue.  I added a test to the Rexster Integration test suite and it passes:

https://github.com/tinkerpop/rexster/commit/f219f1e6463dbb0155a0d06878132f42ec1162ee

in regards to:

> 1) in these, the response type value is 4 and I don't see anything about that in the documentation? What is it if I may ask

As i said, you can't really examine Rexster Console messages as they are in their own serialization channel and really only apply to console operations (unless you want toString to occur for all results).  Anyway, 4 is the CONSOLE_SCRIPT_RESPONSE message type.

> 2) if I add vertex or edges then shut rexster down g.V will not display any vertex as if the graph were reset but the node ID increment is still in place. am I missing something?

Not sure how you are issuing rexpro commands.  If you are not using sessionless or autocommit in a session you are responsible for commiting your own transactions.  Is it possible you are operating in this mode and failing to commit() before shutdown?  If you could provide a series of commands you are issuing with the mode of communication that recreates this problem that would help us to better address the issue.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dmill]

Hi Stephen.

Thanks a lot for taking the time to reply. I have to say you guys have always been top notch and fast with your replies.

This helped me a lot. Here's what's been up:
My script requests contain transaction : true and inSession : true  so I assumed that whatever writes my scripts made were auto commit (as per the documentation). It turns out this isn't the case. I however seem to have to commit changes to be able to get Properties back.
Basically:

neo4jsample:
mssg 1 : g.addVertex([name:'john']);g.commit()
mssg 2 : g.v().map();

This works, I can find the vertex and get properties. If I don't commit() I can find the vertex but not the map() (it returns empty). In TinkerGraphs I don't have to commit to get the map(). I still have to commit to save changes to graph though.

For full disclosure here's how the session and script are set up:

Open session with:
- protocol version 0
- Message type 1
- randomly generated session UUID
- randomly generated request UUID
- killSession set to false, graphName set to 'neo4jsample'
- Channel : 2
- username : null
- password null

Script:
- protocol version 0
- Message type 3
- Session UUID returned by the SESSION_REPLY
- randomly generated request UUID
- channel set to 2, transaction set to true, inSession set to true
- language : 'groovy'
- script : <script>
- bindings : null

How am I meant to get auto-commit to work? And am I doing anything wrong that would result in getting empty .map() even if I don't commit.

Thanks in advance if you have any more information that could help.

[Stephen Mallette]

I can recreate your issue and it is tied to Neo4j.  Perhaps Peter can comment on whether or not it is expected behavior.

I've added a test and a failing "untest" (an untest is one that is currently failing and maybe someday will pass so it's prefixed with "un") to Blueprints to demonstrate the behavior, 

test (https://github.com/tinkerpop/blueprints/blob/master/blueprints-test/src/main/java/com/tinkerpop/blueprints/TransactionalGraphTestSuite.java#L412)

untest (https://github.com/tinkerpop/blueprints/blob/master/blueprints-test/src/main/java/com/tinkerpop/blueprints/TransactionalGraphTestSuite.java#L649)

The tests show that with Neo4j, a vertex created within one thread is only partially available to other threads prior to commit.  By "partially", I mean that the vertex reference will have the element id, but the properties will return an empty set.  Once commit is called the properties become available to other threads accessing it.  OrientDB seems to handle the untest ok and I think Titan would as well.

Note that we are tracking Blueprints Transaction Consistency issues here (consistency meaning "consistency in operations across blueprints implementations"):

https://github.com/tinkerpop/blueprints/issues/363

It would be great if we could clear this issue for Neo4j as all blueprints implementations would have similar behavior.  

Blake, just a RexPro thought here...we could get around some of this by pushing serialization down into individual Rexster sessions so that they get serialized by the same thread that mutated the graph.  I suppose that is a workaround to consider, but as i mention in the following issue I'd prefer to handle this at a blueprints level:

https://github.com/tinkerpop/rexster/issues/305

dylan, i think you will need to issue your neo4j related gremlin as:

v = g.addVertex([name:'john']);g.commit();v;

to get back the properties.  I know...not so good if you want to issue a series of commands in a session prior to commit.  unfortunately, i'm not sure i can think of another workaround at the moment.  

Stephen

[Blake Eggleston]

Hi Stephen,

Serialization occurs immediately after the script is executed, do you mean serialization should take place after the transaction is committed? This is possible of course, but would break RexPro for Titan. I guess we could decide when to serialize base on the graph implementation we're dealing with, but that seems kinda messy.

[Stephen Mallette]

No....i meant to push serialization down into this class (which is where the thread bound to the session lives):

https://github.com/tinkerpop/rexster/blob/master/rexster-protocol/src/main/java/com/tinkerpop/rexster/protocol/RexProSession.java

If the serialization occurs within the same thread as the script engine evaluation that mutated the graph in the non-committed transaction it should have access to the properties.  In that sense, we work around the Blueprints inconsistency and it should work for all graphs.  Just a thought...again, I'd prefer to see consistency in Blueprints itself though.

--

You received this message because you are subscribed to the Google Groups "Gremlin-users" group.

[Dylan Millikin]

Thanks again Stephen for being incredibly useful.
I can live with this being a pending issue it's not a probleme at all for the moment as I'm still in the process of setting things up. I can very well test functionality with another graph database and then revert back to neo4j when I'm done with development.

There is still something that is unclear to me. If my script request meta contains transaction:true and inSession:true should this not auto-commit? The documentation states:  "Executes script within a transaction, committing if successful, rolling back if not. Has no effect with non transactional graph. Defaults to true. Optional."

Am I missunderstanding this statement? If I am how should I proceed to auto-commit my script requests in rexpro?

Thanks again.

2013/5/18 Stephen Mallette <spmal...@gmail.com>

--
You received this message because you are subscribed to a topic in the Google Groups "Gremlin-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gremlin-users/vAWfdnSwzaM/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to gremlin-user...@googlegroups.com.

[Blake Eggleston]

Hi Dmill,

I just pushed up a fix for this. You are correct about the expected behavior of transaction and inSession flags. Internally, Rexster will execute the script for an in session request in the same thread the session was opened in, which may or may not be the thread handling your request. The problem was that Rexster was opening and closing the transaction in one thread (the one handling your request), and executing the script in another thread (the one that owned the session), leading to the problems you've been having.

On Thursday, May 16, 2013 1:43:34 PM UTC-7, Dmill wrote:

[Blake Eggleston]

Also, unrelated, but I didn't know that a php client existed for rexpro. Is there a repo you can point me to so I can check it out?

Thanks,

Blake

On Thursday, May 16, 2013 1:43:34 PM UTC-7, Dmill wrote:

[Dylan Millikin]

Hi Blake,

Thanks for the update. I'll make sure I pull to have a play around with it.

The php client is one I made. To be honest it started as something I mustered up quickly for integration into a framework I use. The idea was that I would later make a c client and wrap it into a php extention instead. But with that said, a php client has it's use and it's currently fully functional so I've started cleaning and documenting the code.
Once I'm done I'll be sure to put it up and link you to the repo. Ditto with the c client.

Take care, and thanks again.

2013/5/18 Blake Eggleston <bdegg...@gmail.com>

--

[Dylan Millikin]

Just a quick note to say that the fix is working well on my end. thanks!

2013/5/18 Blake Eggleston <bdegg...@gmail.com>

--