gephi with remote

[Dylan Millikin]

Hey guys,

I was wondering if it was possible to use gephi with a remote server ie a graph in titan for example? 

Can I possibly duplicate a graph (or subgraph) "locally" so that I can then stream it to gephi by resetting the remote?

Thanks in advance.

[Stephen Mallette]

Hi Dylan, can you expand your question a little bit...i'm not sure I fully understand what you are asking.  

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAK-m_Ky27B-dL4KLrFaEedrQ3CFt0Phq%3D%2BrU_5O7zu9mEKSRjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Dmill]

Sure,

Essentially, we use :remote to connect to gephi and stream a local graph to it. 

In my case I'm using :remote to connect to titan and I would like to stream that titan graph to gephi. In reality I don't need all functionalities, I would like to do a simple visualization.

I was wondering what my options were.

Thanks in advance.

[Stephen Mallette]

The Gephi plugin only works with results in the console of Graph and Traversal.  When you get back results from Gremlin Server they return as neither of those types so right now that won't work. It took me a longer than I expected to come up with a way for to do this - we had about half-dozen ways it could have worked and i didn't realize the best way until i'd run through all the others.  Basically, you can "move" a subgraph over :remote as follows:

PREREQUISITE: Edit your Gremlin Server yaml file to have this custom import:

scriptEngines: {

  gremlin-groovy: {

    imports: [org.apache.tinkerpop.gremlin.structure.util.star.*],

Then in the console:

gremlin> :import org.apache.tinkerpop.gremlin.structure.util.star.*

==>groovy.grape.Grape, org.apache.commons.configuration.*, .... , org.apache.tinkerpop.gremlin.structure.util.star.*

gremlin> :remote connect tinkerpop.server conf/remote-objects.yaml

==>Connected - localhost/127.0.0.1:8182

gremlin> :> g.V().map{StarGraph.of(it.get())}  // assume this is the "subgraph" - the key is to convert Vertex to StarGraph

==>stargraph[starOf:v[1]]

==>stargraph[starOf:v[2]]

==>stargraph[starOf:v[3]]

==>stargraph[starOf:v[4]]

==>stargraph[starOf:v[5]]

==>stargraph[starOf:v[6]]

gremlin> graph = TinkerGraph.open()

==>tinkergraph[vertices:0 edges:0]

gremlin> g = graph.traversal()

==>graphtraversalsource[tinkergraph[vertices:0 edges:0], standard]

gremlin> result.each{it.get(StarGraph).getStarVertex().edges(OUT).each{e->e.attach(Attachable.Method.getOrCreate(graph))}}

==>result{object=stargraph[starOf:v[1]] class=org.apache.tinkerpop.gremlin.structure.util.star.StarGraph}

==>result{object=stargraph[starOf:v[2]] class=org.apache.tinkerpop.gremlin.structure.util.star.StarGraph}

==>result{object=stargraph[starOf:v[3]] class=org.apache.tinkerpop.gremlin.structure.util.star.StarGraph}

==>result{object=stargraph[starOf:v[4]] class=org.apache.tinkerpop.gremlin.structure.util.star.StarGraph}

==>result{object=stargraph[starOf:v[5]] class=org.apache.tinkerpop.gremlin.structure.util.star.StarGraph}

==>result{object=stargraph[starOf:v[6]] class=org.apache.tinkerpop.gremlin.structure.util.star.StarGraph}

gremlin> graph

==>tinkergraph[vertices:6 edges:6]

gremlin> g.E()

==>e[7][1-knows->2]

==>e[8][1-knows->4]

==>e[9][1-created->3]

==>e[10][4-created->5]

==>e[11][4-created->3]

==>e[12][6-created->3]

gremlin> g.V(1L).out('knows')

==>v[2]

==>v[4]

Now you can create a remote to Gephi and submit the Console side subgraph to as you please.  I kinda wish this was nicer:

gremlin> result.each{it.get(StarGraph).getStarVertex().edges(OUT).each{e->e.attach(Attachable.Method.getOrCreate(graph))}}

it's not hard to see what's going on once you stare at it for a bit, but i wonder if we couldn't have something more streamlined - added this for future thought:

https://issues.apache.org/jira/browse/TINKERPOP3-728

I may have also found a bug while doing this:

https://issues.apache.org/jira/browse/TINKERPOP3-729

while a bit less efficient, it seems like we really should be able to shorten this:

result.each{it.get(StarGraph).getStarVertex().edges(OUT).each{e->e.attach(Attachable.Method.getOrCreate(graph))}}

to this if we wanted to:

result.each{it.get(StarGraph).edges().each{e->e.attach(Attachable.Method.getOrCreate(graph))}}

but it doesn't seem to work - throws an exception stating the edge id already exists.....

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/9348c67a-4804-4e69-8b68-d2734357b937%40googlegroups.com.

[Dylan Millikin]

Thanks a lot for this Stephen. 

I only just now managed to take a closer look at this.

I seem to be getting an error while using conf/remote-objects.yaml when I build the StarGraph via g.V().map{StarGraph.of(it.get())} (Where the graph is a TitanGraph)

The only output I get from the operation is the following WARN:

WARN  org.apache.tinkerpop.gremlin.driver.MessageSerializer  - Response [PooledUnsafeDirectByteBuf(ridx: 2730, widx: 2730, cap: 2730)] could not be deserialized by org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.

After this things just hang (my graph is only 30 maybe 40 elements big at this point)

It however works fine on a remote TinkerGraph (just like you described)

--
You received this message because you are subscribed to a topic in the Google Groups "Gremlin-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gremlin-users/WEvVzPECnVo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H43976HwXX3SCk%2B9jTb2Fhxg0Rt_rFziDD9kx%3DH5aCgmRqA%40mail.gmail.com.

[Stephen Mallette]

I think you just have a serializer configuration problem in the console as our default remote-objects.yaml that ships with Titan distribution isn't so good.  It should look like this:

hosts: [localhost]

port: 8182

serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { custom: [com.thinkaurelius.titan.graphdb.relations.RelationIdentifier] }}

You need to register the serializers on the client side to match the server side.  This issue in jira should help make this easier:

https://issues.apache.org/jira/browse/TINKERPOP3-740

I'll make sure Titan gets updated to have this revised configuration.

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAK-m_KwQudg-vJeSgJAgYRO_USb-_9XXH_48hQZdrVGF5Hzd7A%40mail.gmail.com.

[Dylan Millikin]

Thanks for that but I'm still getting the following error after connecting with the new remote-objecst.yaml. The gremlin-server configuration uses the same serializers as the default one (included bellow the error for reference).

It's probably an easy config fix but I can't put my finger on it.

Thanks!

bin/gremlin.sh 

         \,,,/

         (o o)

-----oOOo-(3)-oOOo-----

plugin activated: tinkerpop.server

plugin activated: tinkerpop.utilities

13:38:22 INFO  org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph  - HADOOP_GREMLIN_LIBS is set to: /home/parallels/titan-0.9.0-SNAPSHOT-hadoop1/bin/../lib

plugin activated: tinkerpop.hadoop

plugin activated: tinkerpop.tinkergraph

plugin activated: aurelius.titan

gremlin> :remote connect tinkerpop.server conf/remote-objects.yaml

==>Connected - localhost/127.0.0.1:8182

gremlin> :> t.V()

13:38:55 WARN  org.apache.tinkerpop.gremlin.driver.MessageSerializer  - Response [PooledUnsafeDirectByteBuf(ridx: 3062, widx: 3062, cap: 3062)] could not be deserialized by org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.

13:38:55 ERROR org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler  - Could not process the response - correct the problem and restart the driver.

io.netty.handler.codec.DecoderException: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: java.lang.IndexOutOfBoundsException: Index: 11, Size: 0

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler.channelRead0(WebSocketClientHandler.java:90)

at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:182)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: java.lang.IndexOutOfBoundsException: Index: 11, Size: 0

at org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.deserializeResponse(GryoMessageSerializerV1d0.java:183)

at org.apache.tinkerpop.gremlin.driver.handler.WebSocketGremlinResponseDecoder.decode(WebSocketGremlinResponseDecoder.java:49)

at org.apache.tinkerpop.gremlin.driver.handler.WebSocketGremlinResponseDecoder.decode(WebSocketGremlinResponseDecoder.java:36)

at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)

... 20 more

Caused by: java.lang.IndexOutOfBoundsException: Index: 11, Size: 0

at java.util.ArrayList.rangeCheck(ArrayList.java:653)

at java.util.ArrayList.get(ArrayList.java:429)

at org.apache.tinkerpop.shaded.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60)

at org.apache.tinkerpop.shaded.kryo.Kryo.readReferenceOrNull(Kryo.java:831)

at org.apache.tinkerpop.shaded.kryo.Kryo.readObjectOrNull(Kryo.java:725)

at org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.deserializeResponse(GryoMessageSerializerV1d0.java:166)

... 23 more

Error while processing results from channel - check client and server logs for more information

Display stack trace? [yN] y

org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException: Error while processing results from channel - check client and server logs for more information

at org.apache.tinkerpop.gremlin.console.groovy.plugin.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:118)

at org.apache.tinkerpop.gremlin.console.commands.SubmitCommand.execute(SubmitCommand.groovy:41)

at org.codehaus.groovy.tools.shell.Shell.execute(Shell.groovy:101)

at org.codehaus.groovy.tools.shell.Groovysh.super$2$execute(Groovysh.groovy)

at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:130)

at org.codehaus.groovy.tools.shell.Groovysh.executeCommand(Groovysh.groovy:254)

at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:153)

at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:119)

at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:94)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:130)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:150)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:123)

at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:58)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)

at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:324)

at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1207)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:130)

at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:150)

at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:82)

at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:215)

at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:144)

at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:215)

at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:302)

Here's the configuration:

host: localhost

port: 8182

threadPoolWorker: 1

gremlinPool: 8

scriptEvaluationTimeout: 30000

serializedResponseTimeout: 30000

channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer

graphs: {

  graphT: conf/titan-cassandra-es.properties,

  graph: conf/gremlin-server/tinkergraph-empty.properties

}

plugins:

  - aurelius.titan

  - tinkerpop.tinkergraph

scriptEngines: {

  gremlin-groovy: {

    imports: [java.lang.Math, org.apache.tinkerpop.gremlin.structure.util.star.*],

    staticImports: [java.lang.Math.PI],

    scripts: [scripts/empty-php.groovy]},

  nashorn: {

      imports: [java.lang.Math],

      staticImports: [java.lang.Math.PI]}}

serializers:

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { useMapperFromGraph: graphT }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { useMapperFromGraph: graphT }}

  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { useMapperFromGraph: graphT }}

processors:

  - { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}

metrics: {

  consoleReporter: {enabled: true, interval: 180000},

  csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},

  jmxReporter: {enabled: true},

  slf4jReporter: {enabled: true, interval: 180000},

  gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},

  graphiteReporter: {enabled: false, interval: 180000}}

threadPoolBoss: 1

maxInitialLineLength: 4096

maxHeaderSize: 8192

maxChunkSize: 8192

maxContentLength: 65536

maxAccumulationBufferComponents: 1024

resultIterationBatchSize: 64

writeBufferHighWaterMark: 32768

writeBufferHighWaterMark: 65536

ssl: {

  enabled: false}

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H438SpppUGicUfQ4H-nAZ-UuyNDaDjXwsKnjdmRV8jtBB%2BQ%40mail.gmail.com.

[Stephen Mallette]

what's in your remote-objects.yaml?  it looks just like the one i pasted to you in my last post?

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAK-m_Kx%3DNopSqBz0COU4w5cimzSXf5asJWz-kBu9biMv%2BDNjww%40mail.gmail.com.

[Dylan Millikin]

yeah:

hosts: [localhost]

port: 8182

serializer: { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { custom: [com.thinkaurelius.titan.graphdb.relations.RelationIdentifier] }}

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H439Cv%2BrYgg3VKbodKAU3DV6PwwHJgyDpULqy_zQg1O43Pw%40mail.gmail.com.

[Stephen Mallette]

This seems to work with the Titan SNAPSHOT distribution i built up - i'm guessing you have some version issue.  

gremlin> :import org.apache.tinkerpop.gremlin.structure.util.star.*

==>groovy.grape.Grape, org.apache.commons.configuration.*, org.apache.tinkerpop.gremlin.process.traversal.strategy.verification.*, 

... 

com.thinkaurelius.titan.graphdb.tinkerpop.computer.bulkloader.*, org.apache.tinkerpop.gremlin.structure.util.star.*

gremlin> :remote connect tinkerpop.server conf/remote-objects.yaml

==>Connected - localhost/127.0.0.1:8182

gremlin>  :> g.V().map{StarGraph.of(it.get())} 

==>stargraph[starOf:v[4304]]

gremlin>  graph = TinkerGraph.open()

==>tinkergraph[vertices:0 edges:0]

gremlin> result.each{it.get(StarGraph).getStarVertex().attach(Attachable.Method.getOrCreate(graph))}

==>result{object=stargraph[starOf:v[4304]] class=org.apache.tinkerpop.gremlin.structure.util.star.StarGraph}

gremlin> graph

==>tinkergraph[vertices:1 edges:0]

gremlin> graph.vertices()

==>v[4304]

Note that my syntax for re-attaching the StarVertex to TinkerGraph is a little different because I have no edges in my graph, but the premise is the same.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAK-m_KwRi4sNt7dVyHrUd_%2BX7hCbZs-%2BOtgOWQrQQUFvPmjv9Q%40mail.gmail.com.