Rexster Rest API binding for .Net

[RajBha]

Hello,
Is there a client which can be used to talk to Rexster using .Net

Is the performance of Rest api ok as comparable to native performance
regards

[Stephen Mallette]

TinkerPop does not maintain a .NET client for Rexster and I've yet to
see any third-party support for that appear. As with almost any REST
API, you would find it to be slower than native performance as it
brings the overhead of serialization and network transfer. That said,
I think you will find that Rexster is quite fast for many applications
and is a smart way to expose graph traversals as REST endpoints via
the extension model or the Gremlin Extension:

https://github.com/tinkerpop/rexster/wiki/Extensions
https://github.com/tinkerpop/rexster/wiki/Gremlin-Extension

Aside from REST, we've been experimenting heavily with a binary
protocol for Rexster. It's proven to be much faster than REST in our
tests, however it's not yet a complete and polished feature at this
time.

Stephen

> --
>
>

[RajBha]

Hello Stephen,

Will a .net client be able to use the binary protocol (as you are still in development mode). Is there a test class availble , so we can use it.

regards

[Stephen Mallette]

One of the ideas behind RexPro (still have never been quite happy with
the name), is to encourage the development of different language
connectors to Rexster. I haven't built any yet outside of Java, but
in principle it should be possible to do one in .NET. My intention is
to test the "principle" with C#, but haven't quite gotten around to
doing that as the implementation of RexPro has been shifting for some
time now.

Stephen

> --
>
>

[RajBha]

Hello Stephen,
Can we use RexPro directly to talk to Titan from .Net client. I would be very much interested in porting the java code to .net , if you can point me to it.
regards

[Stephen Mallette]

Yes, you can definitely use RexPro to send Gremlin to Titan. You are
certainly free to develop a connector, just know that RexPro is still
subject to change and there is no real documentation yet. I expect
those things to change in our next release, where I believe that
RexPro will be more official. I'll do my best to give you an overview
of how it works here so you can get a feeling for what it is doing.

RexPro utilizes MsgPack for serialization.

http://msgpack.org/

You should find a .NET library there that can help you. These are the
available messages that can be sent back and forth between Rexster and
the client:

https://github.com/tinkerpop/rexster/tree/master/rexster-protocol/src/main/java/com/tinkerpop/rexster/protocol/msg

These messages allow you to open sessions, send gremlin, etc. You will
find that RexPro sessions are constrained by "channels". A channel
controls the type of serialization that is returned from Rexster.
There currently are channels for MsgPack, GraphSON and Console (which
is just converts all results to a string list). All channels encase
their results within one of the MsgPack serialized messages that I
referenced above.

A working example for RexPro sessions is the Rexster Console. It uses
RexPro for it's communication with Rexster:

https://github.com/tinkerpop/rexster/blob/master/rexster-console/src/main/java/com/tinkerpop/rexster/console/RexsterConsole.java

Here's an example of use "sessionless" RexPro which is the lightest,
fastest way you can go (it uses the MsgPack channel only):

https://github.com/tinkerpop/rexster/blob/master/rexster-protocol/src/main/java/com/tinkerpop/rexster/protocol/TryRexProSessionless.java

I would suggest you start with the model exposed by RexsterClient
(which is used by TryRexProSessionless):

https://github.com/tinkerpop/rexster/blob/master/rexster-protocol/src/main/java/com/tinkerpop/rexster/client/RexsterClient.java

as that is what we've been working with most in our testing with Titan.

Here's some of the things I'm still considering with RexPro which is
why it's not quite "ready":

+ Messages formats have always seemed a bit long to me...would be nice
to trim some bytes and complexity out of there.
+ Messages basically act as envelopes for results from a Gremlin
script. The envelopes are serialized by MsgPack. So...you basically
get a situation where the results get serialized to a byte array given
the channel requested and they get pushed into the envelope. That
design evolved a bit out of my conversion to MsgPack for envelope
serialization as opposed to one I chose from the outset of building
RexPro. For some reason that's always bugged me...maybe not bad.
+ RexPro operates at a very low-level for purpose of communication
with Rexster. It's basically send Gremlin to Rexster and get back the
results. The trick with that is the results can be anything. It
might be a vertex or an edge or a string or a list of maps, etc.. The
point is that RexPro really enables the building of higher level
client libraries that can interact with Rexster in more specific ways.
For example, a JDBC library that can deal with data returned only as
tables or scalars or the Rexster Console which just treats everything
as strings or a revised RexsterGraph that works at the RexPro level.
So...if I were to start building a connector in C#, I would try to
think in those terms a bit.

I assume you will write this in C#? Will your work be open source?
If so, please point me to the repo when you have it up and running.

Best regards,

Stephen

> --
>
>

[sangau]

Hello Stephen,

I along with my team will also like to join in creation of .Net port. I have one question, have you thought of using websockets as the underlying communication mechanism? for then the communication become standardized

Please do tell me, what do you think.

regards

[Stephen Mallette]

Grizzly does support websockets, so in theory it would be possible,
but I can't say I've considered that in great detail as something
Rexster would support. I've created an issue to look into it as an
option:

https://github.com/tinkerpop/rexster/issues/258

Stephen

> --
>
>

[sangau]

Hello Stephen,

I am trying to port the msg as you mentioned above. There is a class  MsgPackScriptResponseMessag which has dependency on com.tinkerpop.rexster.gremlin.converter.MsgPackResultConverter which has further dependency on tinkerpop core like edge,vertex,direction etc.

Do you think, we should port them also. If yes, what are the various classes do you think would be needed.

regards

[Stephen Mallette]

I'm not sure that you want to try to do a direct port of classes file
for file. I think that you'll find yourself getting lost in
dependencies. My suggestion would be to look at the format of the
messages (and the java code) to understand what bytes need to be sent
back and forth to Rexster. Once you understand that, you can consider
how to be structure that for a C# client.

To specifically answer your question about
MsgPackScriptResponseMessage, that message class (and some of the
others) have some static methods on them that i've used to help work
with the message itself. The dependency you mention has nothing to do
with the structure of the message being sent back and forth (for that,
you just need to pay attention to the public fields...those are the
bits that are serialized by MsgPack). I've been considering moving
those static methods out to leave Messages as just containers for
data...just haven't done it yet.

stephen

> --
>
>

[Daniel Kuppitz]

Hi,

has anybody managed to send and receive messages from C# to Rexster? I've tried it for several hours now, but I wasn't able to figure out the correct format. The RexPro message documentation doesn't really help and existing clients (Java and Python) are too complex. Can anybody give me the MsgPack byte sequence for a simple script request message (something really simple like g.V.count(); leaving out the optional meta stuff) . I think the byte sequence together with the documentation will help me a lot more than reading through existing code that's not written in my language (C#).

Cheers,

Daniel

[Blake Eggleston]

Hi Daniel, can you post what you've written so far?

Thanks,

Blake

[James Thornton]

Hi Guys -

Last weekend I started to research options and things to consider when writing RexPro clients -- we're working on creating a RexPro client design doc that will explain how to do this in detail so that writing RexPro clients is easy for lib authors.

As Stephen said, the RexPro message format is still somewhat in flux. For example, right now it uses message length prefixes to signal message boundaries, but this may change to using out-of-band delimiters.

To understand the message length prefix vs out-of-band delimiter issue, see Meredith Patterson's CCC talk "The Science of Insecurity" (https://www.youtube.com/watch?v=3kEfedtQVOY). The issue is also discussed here (https://news.ycombinator.com/item?id=5086002). 

C# has a msgpack library so it's a non issue in this case, but not every language has a MsgPack library (http://msgpack.org/) for binary serialization; however, RexPro also supports JSON/GraphSON so you can always fall back to it.

Here a some things to consider when writing a client (more details coming later)....

Rexster uses Grizzly (http://grizzly.java.net/), a high performance server for Java NIO (http://en.wikipedia.org/wiki/New_I/O) non-blocking socket connections.

With Grizzly's non-blocking IO, a RexPro server can serve thousands of simultaneous connections (http://docs.oracle.com/cd/E19776-01/820-4496/ggrib/index.html) since it does not need to allocate one thread per request.

RexPro uses a session protocol. This allows you to make multiple requests over one connection, and each client can treat all connections as persistent until the client closes them. This means you don't have have the overhead of building up and tearing down connections each time.  See this Java sockets thread on persistent sockets and NIO: https://forums.oracle.com/forums/thread.jspa?threadID=1692077

To keep connections open, you can use a one-connection per process model, or if you need to make multiple simultaneous/async requests, you can use a connection pool. 

Managing connection pools can be tricky because there are a ton of corner cases to consider and so writing a proper pooling library is not easy. We don't want to each RexPro client author to have to do this so we're looking ways of approaching this to make things easy.

There are different approaches to connection pooling. Many have experience using connection pools with relational databases like PostgreSQL. Postgres doesn't support connection pooling directly (for long standing philosophical reasons), and each connection ties up resources (a thread) so you usually don't want to tie up the DB's resources with an overflow of persistent connections. 

Since Postgres doesn't have connection pooling built in, many use a pooling proxy like PgBouncer (http://wiki.postgresql.org/wiki/PgBouncer) for persistent connections. You connect to PgBouncer directly, which manages the connection pool and proxies the connections back to Postgres.

This is effectively what Grizzly is doing -- Grizzly manages the DB connection pool and acts as a proxy to the underlying database -- so implementing a RexPro client connection pool is not needed for DB performance reasons. 

Rather for RexPro, a client connection pool would be useful in cases where you need to make multiple simultaneous requests within a process. But keep in mind, Rexster has a session protocol. Furthermore  you can often get around the need to send multiple requests by encapsulating all the logic in a single Gremlin script so that it executes in a single request.

Here's an example of a "save_blog_entry" Gremlin script that encapsulates everything into request (https://github.com/espeed/lightbulb/blob/master/lightbulb/gremlin.groovy) --- more details on those types of considerations will be in the design doc.

However, if you decide you need client-side pooling, you should try to use a battle-tested pooling library rather than rolling your own because there are so many corner cases. It's easy to have pool full of hung connections.

You can find external pooling libraries, but most are found within ORM packages. In the case of Python, SQLAlchemy (http://www.sqlalchemy.org) and SQLObject (http://www.sqlobject.org) both contain connection pooling code, and with SQLAlchemy, you can hook into its pooling library even though you aren't using SQLAlchemy as a whole, i.e. NoSQL DBs can use it too.

There was a long discussion about connection pooling issues on HN last month (https://news.ycombinator.com/item?id=5345577), which began when Kenneth Reitz (http://kennethreitz.org/) published a Django-Postgres adapter that uses the SQLAlchemy pooling library. This fixed Django's long-standing performance issue of having to build up and tear down new connections for each request. 

If anyone is interested, I encourage you to read the HN post because the top comments cover a lot of the edge cases -- BTW zzzeek is Mike Bayer (http://techspot.zzzeek.org/), the author of SQLAlchemy.

Mike also goes over connection pooling here: http://docs.sqlalchemy.org/en/latest/core/pooling.html

Regardless of the language you're writing the RexPro client in, to hook into an existing connection pooling lib, you probably need to implement the client according the language's database API spec. 

In the case of Python, the DB-API 2 is defined in PEP 249 (http://www.python.org/dev/peps/pep-0249/), which is rather simple -- you just implement a connection interface and a cursor interface (similar in concept to implementing a Blueprints interface). Once that is done, you can hook into SQLAlchemy's battle-tested pooling lib. 

Another consideration is how to implement the async piece, but how you do that will vary from language to language. 

In Summary, determine whether you really want/need pooling because Grizzly is already handling the DB pool for you. And if you decide you want connection pooling, try to use a battle-tested pooling library rather than rolling your own. 

- James

[Daniel Kuppitz]

Hi Blake,

here's what I have so far: Simple Console Test Application

I've tried different variations to serialize dictionaries/maps, There might be some strange and/or useless code snippets (like in line 77); these were just experiments to somehow get a positive response from Rexster. Also note, that the use of MsgPack above line 54 does not effect the actual serialization, which comes later. As soon as line 80 is executed, Rexster logs an exception:

32526929 [Grizzly(4)] ERROR com.tinkerpop.rexster.protocol.filter.RexProMessageFilter  - Error during message deserialization of a message of type [3].

org.msgpack.MessageTypeException: Unexpected map value

... 

The exception is always the same, so I'm quite sure, that I just don't know how to serialize the maps correctly.

Cheers,

Daniel

[Daniel Kuppitz]

Hi James,

how does a request message in JSON/GraphSON look like? I've seen, that response messages can be in JSON format, but for requests it seems to be undocumented. Something like this?

{

  "protocolVersion":0,

  "messageType":3,

  "message":{

    "session":"AAAAAAAAAAAAAAAAAAAAAA==",

    "request":"pV62Z8NSvE2q5bBbxNfs0w==",

    "meta":{

      "channel":2,

      "graphName":"graph"

    },

    "languageName":"groovy",

    "script":"g.V.count()",

    "bindings":{}

  }

}

Cheers,

Daniel

[Blake Eggleston]

Hi Daniel,

I took a look at your code, I'm not a C# user, but I did notice something that would be causing you problems.

Here:

https://gist.github.com/dkuppitz/02fa0f1935c8b654d1b8#file-prgram-cs-L27

the `message` field needs to be an Array<Object>, not a dictionary, the order you have the fields declared in looks correct.

Again, not a C# guy, but it should look something like this:

https://gist.github.com/bdeggleston/5364119

Give that a try, and let us know how it goes.

Thanks,

Blake

On Tuesday, December 4, 2012 10:35:08 PM UTC-8, RajBha wrote:

[Daniel Kuppitz]

Awesome, it works! Here's the final code for my test: C# <-> RexPro
I've also measured the average execution time in 100 runs for this code versus the same query over the REST endpoint.

REST: 55 ms

RexPro: 31 ms

That's why it's worth all the work :). I'm still not sure about the output format; no idea what the lines 3, 4, 5 and 31 are, but as I don't need them right now, it's ok.

Thanks for pointing me to the right direction, Blake.

Cheers,

Daniel

[Stephen Mallette]

Daniel, nice!  I forked your Gist for future reference.  Thanks for posting that.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Daniel Kuppitz]

The response can be really confusing. As you can see in my gist, there are 6 numbers/bytes before the actual message body begins. What are these bytes? Protocol version, message type, and...?

My problem now is, that I just changed the query to g.V.next() and suddenly I only get 5 bytes at the beginning. I'm not quite sure how to deserialize this in a general way.

Cheers,

Daniel

[Stephen Mallette]

The first 6 bytes should be just as they are described here in the Basic Message Structure:

https://github.com/tinkerpop/rexster/wiki/RexPro-Messages#basic-message-structure

+ first byte is the version

+ second byte is the message type

+ next four bytes is an integer value that represents the length of the message in bytes to be deserialized by MsgPack

Not sure why you would only get 5 bytes when you call next().  Please let us know if that clarifies anything or if the problem persists....

--

[Daniel Kuppitz]

Hi Stephen,

I accidently skipped the first byte when I wrote my last question. It was just too much test code with all variants of serialization and deserialization :).

I started with a new project and voila: everything works as expected. Once it works, everything looks super easy. The only thing that's really strange, is that vertex ids are strings.

Here's the new code: https://gist.github.com/dkuppitz/5383947

In theory you can send any (valid) script with this code and receive a valid response. However, in practice it looks different. Changing the 5 in line 35 in Program.cs to 4, gives me an error in line 75:

Cannot convert 'System.Byte' type value from type 'Unknown'(0xD8).

Cheers,

Daniel

[Daniel Kuppitz]

Ok, there doesn't seem to be a single method in MessagePack-CLI that is able to read this 0xD8 byte. Luckily it's the only error I get right now, so it's easy to work around this issue. Now I simply skip the 4 message length bytes (I still don't know why they are there at all) and have minmal working RexPro client in C#. I'll update the previously posted gist in a minute.

Cheers,

Daniel

[Daniel Kuppitz]

And finally we have a Rexster client for .NET: https://github.com/dkuppitz/rexster-client

I've only implemented the message types ScriptRequest, ScriptResponse and ErrorResponse. The other message types will follow when I need them or when anybody else makes a pull request.

Cheers,

Daniel

[Stephen Mallette]

Daniel...cool...thanks for your contribution.  I will check it out more carefully later

what happens if you change this line: 

https://github.com/dkuppitz/rexster-client/blob/master/RexsterClient/RexsterClient.cs#L108

to stream.readInt(out someIntValue)

will it read those bytes properly?

[Daniel Kuppitz]

No, I've tested nearly every method provided by MsgPack-CLI, but they all threw an exception. However, IMO the message length is totally useless, so it's just fine to skip the 4 bytes.

Cheers,

Daniel

[Daniel Kuppitz]

FYI: I've changed the project name. New git url is https://github.com/dkuppitz/rexpro-client