Gephi Streaming and Neo4j

[Caleb Jones]

I'm currently working on building a Java client for the Gephi streaming API and will be presenting at the Seattle Graph Meetup group. I'm aware of the Neo4j Gephi plugin (https://marketplace.gephi.org/plugin/neo4j-graph-database-support/) and plan on showing how that can be used too, but I'm curious if there are any specific Neo4j applications that would fit well with the streaming work I'm doing.

One thought is to have a mode in the streaming client I'm writing that tees the streaming to both Gephi and Neo4j. Of course, someone could just stream to Gephi then export to Neo4j as well.

I'm not drowning in free time to do this, so I'm looking for simple integrations to do.

Thoughts?

[Caleb Jones]

Is there a way to do a live view of a Neo4j graph similar to how Gephi streaming works?

[Peter Neubauer]

Caleb,

I have been trying to get in contact with the developers of the existing Neo4j Gephi plugin, but really, with Cypher, a streaming plugin that just executes Cypher would really be the cleanest solution. I think a prime target is e.g. data with a certain size, for instance the Cineast dataset?

Superstoked you are doing this, Michael and I would love to comment and contribute, as many users are asking for this!

/peter

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Kids in Malmö this summer?       - http://www.kidscraft.se
Neo4j questions? Please use SO - http://stackoverflow.com/search?q=neo4j

--
You received this message because you are subscribed to the Google Groups "Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Michael Hunger]

Thanks Caleb, this is much appreciated.

We tried to update the existing plugin to Neo4j 1.9 but the Netbeans build setup has been a bit of a pain. 

For the streaming plugin:

You can register a transaction event handler in Neo4j that gets called as transactions are committed, those results can then be streamed to Gephi.

Best would be to register a kernel extension for Neo4j so it could work both for Neo4j embedded or server. Otherwise go with an unmanaged server extension.

What kind of protocol does streaming gephi use?

It would be cool to have a cypher statement visualized in gephi and the results updated as new information comes into neo4j.

Another approach that is really interesting would be to have an additive version where the shown data is added to as users execute new queries against neo4j via the streaming plugin.

I also don't know if there is a good way of publishing the work-in-progress as an ongoing thing, besides building gephi on our own and checking it out.

Cheers

Michael

Cheers,

Michael

(you)-[:GET]->(graphconnect-ticket)<-[:DISCOUNT {value:25%}]-(NEO4GC)

[Caleb Jones]

Let me clarify a bit to make sure we're on the same page.

I'm not planning on working inside the Gephi streaming plugin code but instead writing a client that can talk to it. The Gephi streaming plugin sets up a an HTTP port that Gephi listens on and allows you to POST JSON to it to do basic node/edge insert/update/delete operations. I looked and couldn't find a Java client that abstracts the HTTP API, so I'm writing one. I've done this for a POC and am now just cleaning it up to be more extensible/usable. Note that I'm less interested in simply dumping the information into Gephi and instead more interested in actually watching how graphs evolve over time by tuning the speed at which nodes/edges are added. Some interesting insights come when you do this that aren't as evident when you only ever look at the final completed graph.

I am interested in expanding this effort a bit to incorporate Neo4j. Interesting use cases I'd like your feedback/help on:

1. Add a Neo4j streaming client so people could use the same Java API and stream to either Gephi, Neo4j, or both (via simple multiplexing).
2. Add a Neo4jCypher client that takes a cipher query, send it to Neo4j, and then streams the results into Gephi using the Gephi streaming API.
   

I have no experience doing Netbeans plugin development, so I don't think I will be very successful at creating an actual Gephi plugin.

What are your thoughts?

[Peter Neubauer]

Caleb,

I think 2) is most interesting, since it will be useful in a wider range of usecases. Do you have the code or a screencast somewhere already?

If one could specify the Cypher query and then ship it via the API to Neo4j, that would be supernice. Also, the current HTTP Cypher endpoint already supports streaming (http://docs.neo4j.org/chunked/milestone/rest-api-streaming.html), so the integration should be pretty straightforward? http://docs.neo4j.org/chunked/milestone/rest-api-cypher.html

/peter

Cheers,

/peter neubauer

G:  neubauer.peter
S:  peter.neubauer
P:  +46 704 106975
L:   http://www.linkedin.com/in/neubauer
T:   @peterneubauer

Kids in Malmö this summer?       - http://www.kidscraft.se
Neo4j questions? Please use SO - http://stackoverflow.com/search?q=neo4j

[Caleb Jones]

I'm going to try to get Cypher integration to work from Neo4j -> Gephi (not the other way around).

So a data flow would be:

1. Send Cypher query to Neo4j
2. Stream results back at a tunable rate
3. Use Gephi streaming plugin API to write results to Gephi

Parts I'll need to write (I'm working in Java land):
A. Gephi Streaming client API (I'm nearly done with this one)
B. Code that streams the results of a Cypher query into Gephi

Part "B" I have questions about mostly due to my unfamiliarity with what's included in the Neo4j Cypher streaming response. I DON'T just want to stream nodes into Gephi (that's not a graph). I need to stream both nodes AND any applicable edges between them involved in the Cypher query. Does the result coming back from the Cypher streaming API include both the node and edge information?

[Michael Hunger]

It depends on your query,

if you execute something like this:

start person=node:node_auto_index(name={name})

match path=person-[rels:FOLLOWS*0..2]->other

return path

you have in the path all the nodes and relationships in between

you can also chose to 

RETURN person, rels, other

where rels would be a collection of relationships.

HTH

Michael

[Marcelo Gagliano]

Hi, Caleb.

Did you developed that client? If so, could you share the source code?

I am trying to create a similar solution, but I am not having much success.

Thank you,

Marcelo Gagliano

[Michael Hunger]

+1 that would be awesome

I wanted to give it a try myself but haven't found the time.

Btw. my neo4j-shell-tools now export Neo4j to GraphML, so you can visualize your db in Gephi, would love some feedback:

https://github.com/jexp/neo4j-shell-tools/tree/20#graphml-export

Michael

[Patrick Durusau]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael,

I tried out the export to GraphML today.

I was using data from a Twitter feed.

The first issue on trying to load into GraphML was that the "&"
character was not written "&amp;"

When converting files to XML, escape "&" with "&amp;", "<" as &lt; and
">" as &gt;

The next several issues were control characters ^B, ^C, etc. embedded
before TM and R, etc.

Conversion to UTF-8 and stripping anything that doesn't convert would
be nice.

The parser in my Emacs must not match what is being used in Gephi
because it would choke even though Emacs said all was well.

Hope you are at the start of a great week!

Patrick

On 01/17/2014 07:49 PM, Michael Hunger wrote:
> +1 that would be awesome
>
> I wanted to give it a try myself but haven't found the time.
>
> Btw. my neo4j-shell-tools now export Neo4j to GraphML, so you can
> visualize your db in Gephi, would love some feedback:
>
> https://github.com/jexp/neo4j-shell-tools/tree/20#graphml-export
>
> Michael
>
> Am 18.01.2014 um 00:29 schrieb Marcelo Gagliano

> <marcelo....@gmail.com <mailto:marcelo....@gmail.com>>:

>
>> Hi, Caleb.
>>
>> Did you developed that client? If so, could you share the source
>> code? I am trying to create a similar solution, but I am not
>> having much success.
>>
>> Thank you, Marcelo Gagliano
>>
>>
>> On Friday, June 21, 2013 3:12:20 AM UTC-3, Caleb Jones wrote:
>>
>> I'm currently working on building a Java client for the Gephi
>> streaming API and will be presenting at the Seattle Graph Meetup
>> group. I'm aware of the Neo4j Gephi plugin
>> (https://marketplace.gephi.org/plugin/neo4j-graph-database-support/

>> <https://marketplace.gephi.org/plugin/neo4j-graph-database-support/>)

>>
>>
and plan on showing how that can be used too, but I'm curious if
>> there are any specific Neo4j applications that would fit well
>> with the streaming work I'm doing.
>>
>> One thought is to have a mode in the streaming client I'm
>> writing that tees the streaming to both Gephi and Neo4j. Of
>> course, someone could just stream to Gephi then export to Neo4j
>> as well.
>>
>> I'm not drowning in free time to do this, so I'm looking for
>> simple integrations to do.
>>
>> Thoughts?
>>
>>
>> -- You received this message because you are subscribed to the
>> Google Groups "Neo4j" group. To unsubscribe from this group and
>> stop receiving emails from it, send an email to
>> neo4j+un...@googlegroups.com

>> <mailto:neo4j+un...@googlegroups.com>. For more options,

>> visit https://groups.google.com/groups/opt_out.
>
> -- You received this message because you are subscribed to the
> Google Groups "Neo4j" group. To unsubscribe from this group and
> stop receiving emails from it, send an email to
> neo4j+un...@googlegroups.com. For more options, visit
> https://groups.google.com/groups/opt_out.

- --
Patrick Durusau
pat...@durusau.net
Technical Advisory Board, OASIS (TAB)
Co-Chair, OpenDocument Format TC (OASIS)
Editor, OpenDocument Format TC, Project Editor ISO/IEC 26300
Former Chair, V1 - US TAG to JTC 1/SC 34
Convener, JTC 1/SC 34/WG 3 (Topic Maps)
Co-Editor, ISO 13250-5 (Topic Maps)

Another Word For It (blog): http://tm.durusau.net
Homepage: http://www.durusau.net
Twitter: patrickDurusau
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJS3IL+AAoJEAudyeI2QFGoW6AQAJVbb64p40AE5YIgtJuJiWOK
ZVNkXeARDAqPNa5HUpnL4FMaN4NK856mdtQ/JhbBTz8/sQpUc7D7melq91hf6ksT
8mlbJlvqqUzq/025edaUAZgyy9TJc0lPcMA1VJxJlBsapxRIwclyj5bO/DJ4PT8+
48w5FhDNNjQMGxUHPuqUb3R66JFjvIUbY5e/zdZZ5eVcJPo1f7ASjZA4VpNZJX3S
ccNFo3nZdZx1FbJBSPe6PC1Bw/z3uQVRLaDBlq2UoUCUITcdJey0vpgFA0VYBEZv
LexEmnv+7+O5ibUJgp/5OXnveH4Q8Wv0WBg82PlsuEIR6ZXees0lzHs1vmERMRw+
Vok9e3r5szqkys9lqQVLw7jX/fAjaeTQ4j0pVf82+mYMMqtKvhPFihNyi9p4Y32d
wXoEyT50NusE6WoLlSVgKr3ej3Fs2deiT0AnIF9mLo0ZyOQuJ5dnIwmsPwDx8EbB
pfcUmjeVQksAVcP/DXDCHeiOaYwpbd5Opl8ITbfNRna1FCyAsNbNrrHANWntyR3W
emheb+I+mBxeUREwz4j4QVrKD5AdTG+oN7KkF5HYxDBPu+Y8waMVTd+EE8v/lI2r
9rqPMbZSl0mcWp2rT3OhJsj/Bf8e/3oZqmYcxdRz5btGIY1FjCUhLOKhRv22IXUb
RGdNXHFdZYLQoTRV68bV
=g6RA
-----END PGP SIGNATURE-----

[Caleb Jones]

Here's the client that I've written for the Gephi streaming API:

https://github.com/calebjones/gephi-streaming-client

[Michael Hunger]

Thanks for the feedback, will fix these issues.

Do you know where the control characters came from?

Michael

[Patrick Durusau]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael,

On 01/20/2014 02:48 AM, Michael Hunger wrote:
> Thanks for the feedback, will fix these issues.
>
> Do you know where the control characters came from?
>

Guessing I would say that Twitter accepts pasted content. Works ok as
long as you are in the lower ASCII set but for things like trademark
(tm) and the R with a circle? Sorry, I'm real tired.

I still have an uncorrected version of the data and will try to fish
out the lines in question. The full file is large in email terms and
most of it would not be helpful.

I'll get some sleep and look at it in the morning.

Will verify the problems still exist in Gephi as well with the much
smaller version of the file.

Hope you are having a great day!

Patrick

> Michael
>
> Am 20.01.2014 um 02:59 schrieb Patrick Durusau
> <pat...@durusau.net>:
>

iQIcBAEBAgAGBQJS3cB1AAoJEAudyeI2QFGoAosQAPbeUPm1UUaAUrWlRK2oIx9f
b2EmwNH8+vWrLCIHI4HcmWTFZ3hzzuXvulpzeTWIqJxdDR3zuP+SqchhYEY9RTmT
jcjyif9aMLiElZnO5sTOZzW5UtKoEx9Q+F/4dyU7Fsh2cJZI9grsyO+BTsqDJmQJ
k4w44lGWW1Q9ct5wx3T2fHfYFdEHlHOaTVjPEaK5C8Agk9oNWonu/f4RgbBnOCwU
rBntsp6DPJLrVLVqkyl1+whUqSKUWyUXuj26Vf8ub7G5Xg5wrIclgKX3xQtuc/1s
8D94PIaVHdTnvQBB+gDUKr2dGh01ItYKDCAUEfqPrhHjxEh0v23kPKF8IggkSfth
iCd5ULlqLIm/5IxNryfLLQi782y+1QxMkbBHYeDyEmmckhKPDrB4sf4O4vtOgIxq
2eule+WbE3L8dy3OowDxE2rNYNWntWUx4B7x1YbEMuW0/t2JqD1wIdWTk+NSlmfF
lY9cKl1fmtDuiJIWQ5COPuZyOX9PVxp/NpoMlxYPM6yjXzBv8pO+Mjfyo+MgMb/H
IGsRq3kHvUxBaaKeIP2ot2tL+MKmUUY+FkS+nqCb1d4vwPCld5XudGFvbe3/T+WR
EIwm0HEJ9IJg8Y8MVSpkyHu60iRgW8Uzap/4xKAUJDrtNoc9a0Z4Wolmh7CY4g9Q
x2YCM4Q1Mfdp/AC2c595
=jbwa
-----END PGP SIGNATURE-----

[Michael Hunger]

Patrick,

the xml encoding issues for <> & etc. should be addressed.

Not sure how do deal with the control characters though. The only thing I could think of is to write data as CDATA fields?

Or strip them somehow upfront.

Michael

[Patrick Durusau]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael,

I just downloaded the latest neo4j-shell-tools and an export got:

<node id="n1" labels="User,expertValidation,SeedNode" ><data
key="labels">User,expertValidation,SeedNode</data><data
key="id_str">269740110</data><data key="name">Andreyana
Ivanova</data><data key="screen_name">adiivanova</data><data
key="description">Passionate and inspiring Equality & Diversity
Practitioner</data><data key="followers_count">35</data><data
key="friends_count">52</data><data key="listed_count">2</data><data
key="statuses_count">11</data><data
key="favourites_count">0</data><data key="location">London</data><data
key="time_zone">London</data><data key="utc_offset">0</data><data
key="lang">en</data><data
key="profile_image_url">http://pbs.twimg.com/profile_images/1281406234/IMG_5547_normal.JPG</data><data
key="geo_enabled">false</data><data key="verified">false</data><data
key="notifications">false</data></node>

Between "Equality & Divesity" is an example of a & that should be
written as: &amp;"

Sorry for not checking my email earlier but I wanted to create a file
with several examples (I have inserted XML comments for each example)
of what markup errors I am picking up.

BTW, there is a edge case where < and > should not be converted but
that is in XML processing instructions and it is unlikely anyone will
be encountering those in an graph database. As soon as I hit "send" an
example case will hit the email list. ;-)

It has been a while since I have looked at conversion filter libraries
but I suspect there is something that would correct the character and
markup errors automatically on export. HTML tidy I think has that
capacity.

Anyway file attached.

Hope you are having a great week!

Patrick

On 01/20/2014 10:33 PM, Michael Hunger wrote:
> Patrick,
>
> the xml encoding issues for <> & etc. should be addressed.
>
> Not sure how do deal with the control characters though. The only
> thing I could think of is to write data as CDATA fields?
>
> Or strip them somehow upfront.
>
> Michael
>
> Am 21.01.2014 um 01:34 schrieb Patrick Durusau
> <pat...@durusau.net>:
>

iQIcBAEBAgAGBQJS3sUXAAoJEAudyeI2QFGo05wP/jO9MqO1iy2XV8CVPMGStxjz
nSlcY2CPLht1IS4IO/wg/4Xh7Bilk0h86lJ5V7cllmidQCl97h/OtcrUKLlddKo7
Bc15XDkq0cMRBr39OWbyQH/wblozTiyqEI9mBhyczROcY2vHomFGyBgbZazSM092
DOMMyq1rVvm5tArxiyMHbcDNob4xuxLPBBQtPfOoPRQXOHpGt+O92hg88lI4uN0c
jfcyVHEVMVg/gP4vspJpc3cFIbBCAmPKdUe2fWZYjLQAoWXqqJ6MTH44GwzjqmDJ
Fp2Qy0jHpVyVY0wYLcjJ+xvNrL1sXjCgBJsLP7Dt0innGYLzY9K1K6rC96yU0SUV
KpGE6kAvKdbJB1O3TTWXL/iIVpyeqQUa7wfKimYD751ZxADbybD20tXNDV1FUI2r
AezukYK3QBbB0sgKP2JU0HVV9JVLBQ28y2IK925yDStoqOV/b9qnQQg78Kvpzuf0
GmJU/0XdqdPumytRRlk6st5Tmd/qRku6Zy0fjNACUFB8UhmgeVATZn/ZgNnxoISH
BsCid5sjpISzbldp4qhvHrWsqGHH5YDnRBcIaELq7I9EeW42gl0Zc36BD2SONbSm
Okg5hThxEc72G9YYW7E6ys05OZZ/zpeATdWirKYnw+AaiKVVRGcBmK1nlulMmr/8
QOCDj1ZL5cgdiTrm4pM3
=Bxn1
-----END PGP SIGNATURE-----

[Michael Hunger]

Really that's weird, b/c I changed the label format to :labe1l:label2:label3
But it still shows the comma separated one in your case.

Perhaps it was not up to date? Where did you download it?

Michael

My current export looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<graphml xmlns="http://graphml.graphdrawing.org/xmlns" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
<graph id="G" edgedefault="directed">
<key id="na&lt;&gt;me" for="node" attr.name="na&lt;&gt;me" attr.type="string"/>
<key id="count" for="edge" attr.name="count" attr.type="int"/>
<node id="n0" labels=":FOO"><data key="labels">:FOO</data><data key="na&lt;&gt;me">John &amp; Dö</data></node>
<edge id="e0" source="n0" target="n0" label="BAR"><data key="label">BAR</data><data key="count">0</data></edge>
</graph>
</graphml>

> <graphml-test.txt>

[Patrick Durusau]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael,

Trying again in case I got the wrong file.

Downloading:

http://dist.neo4j.org/jexp/shell/neo4j-shell-tools-2.0.zip

Perhaps I did lower case a instead of A on the replace question?

Continuing.

Restarted server, neo4j-shell, exporting....

OK, running SGML validation shows only the upper ASCII characters as
errors.

Thanks!

I must have messed up the overwrite of the file in lib. When prompted.

Hope you are having a great week!

Patrick

PS: Tomorrow, late for me here, will run HTML Tidy (c version) to see
if it cleans out the upper ASCII stuff.

On 01/21/2014 06:58 PM, Michael Hunger wrote:
> Really that's weird, b/c I changed the label format to
> :labe1l:label2:label3 But it still shows the comma separated one in
> your case.
>
> Perhaps it was not up to date? Where did you download it?
>
> Michael
>
> My current export looks like this:
>
> <?xml version="1.0" encoding="UTF-8"?> <graphml
> xmlns="http://graphml.graphdrawing.org/xmlns"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns
> http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd"> <graph
> id="G" edgedefault="directed"> <key id="na&lt;&gt;me" for="node"
> attr.name="na&lt;&gt;me" attr.type="string"/> <key id="count"
> for="edge" attr.name="count" attr.type="int"/> <node id="n0"
> labels=":FOO"><data key="labels">:FOO</data><data

> key="na&lt;&gt;me">John &amp; D�</data></node> <edge id="e0"

> source="n0" target="n0" label="BAR"><data
> key="label">BAR</data><data key="count">0</data></edge> </graph>
> </graphml>
>
> Am 21.01.2014 um 20:06 schrieb Patrick Durusau
> <pat...@durusau.net>:
>

>> -- You received this message because you are subscribed to the
>> Google Groups "Neo4j" group. To unsubscribe from this group and
>> stop receiving emails from it, send an email to
>> neo4j+un...@googlegroups.com. For more options, visit
>> https://groups.google.com/groups/opt_out. <graphml-test.txt>
>

- --
Patrick Durusau
pat...@durusau.net
Technical Advisory Board, OASIS (TAB)
Co-Chair, OpenDocument Format TC (OASIS)
Editor, OpenDocument Format TC, Project Editor ISO/IEC 26300
Former Chair, V1 - US TAG to JTC 1/SC 34
Convener, JTC 1/SC 34/WG 3 (Topic Maps)
Co-Editor, ISO 13250-5 (Topic Maps)

Another Word For It (blog): http://tm.durusau.net
Homepage: http://www.durusau.net
Twitter: patrickDurusau
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJS3yn2AAoJEAudyeI2QFGo2iUP/17bqYLj3COK1H3kTyeVH1tA
bL0oSewBbc8mtpA/DX3ziPBGMWSkt79o8OXwvpbDQR34VkmjpxzPin7n3kw9sqEg
906zNFCAauMcZwLS4zdVOLxW677g7j41bfh5E5h8JTCau0ktCjNZcrkp15C5aYLt
cCEW9OlI418UwQLO+ixKtu5REx4hcWPtNfsPyCYn4xFqGqeZJK/nFuOV7dEK2Doo
lz3TxOpaHOLvQyEn1jc5iPJmRyzaqetV1x8NNCmdNt18helKZTyICy7obdt4bDsA
pb+2X1pkipu3Vj7pkYZj+EBG7pAFPJ+g2zkvye3+WFsOsXGmbaSu1AHVimUIOzMI
3H1zKihy/0jSCrsK4Fa0MkdI7uQOpQ93sRCK4oy8MdLEmzGwjp+g6kURWEnX7a4/
lACs3iqUBqlT4lDF0HVN5MsbFmOBSDfQpItdY9pJUOx6mIyYrcKGEaw7S1Kg6/g9
GeVVOXi8QZRiLtBiY7eS8D3N3/fNC2kQ65aKzN5cgtpaWY/joTUEseXnGxskEway
Y7AbExuJ60geUoczOgDRLpkWM0fXmpuksCMkA25hgJnmQLNNI8ASluJA7WIR0dyE
VvorWogWG/Nl94vjcUecoOat/kTEPVqj+guOTsLWCddQF8pv4kPsxv3xHeuj0fi1
AjqexDp+SHy2rNlosMXy
=OOoq
-----END PGP SIGNATURE-----