Re: [Neo4j] Neo4j on Google App Engine

[Michael Hunger]

Not that I know of.

You could give it a try. After all you just have to swap out the lower, persistence layer which is actually writing and reading from disk (or rather MMIO buffers).

Cheers,

Michael

Am 27.08.2012 um 20:15 schrieb Koen:

> Hi, thanks for this great forum and neo4j product.
>
> I have a question: is there already an (efficient) implementation of the neo4j runtime and especially the file structure mapped om googles bigtable datastore ? This would allow for running neo4j
> on google app engine while making use of its graphs structure and facilities
>
> Google App Engine does not allow for stoiring files on their file system directly.
>
> Thanks for any replies
>
> regards koen
>
>
>
>
>
> --
>
>

[Koen]

Hi, thanks, i have created a small sample data model (see attached jpg) mapping the neo4j datamodel as i understand it onto a google bigtable datamodel with one table and a few famililies grouped by property type (string etc ) , node type (for relations) and related nodes (for relations)

every new node / relation becomes a record with one or more properties and in case of a relation related nodes ...   a bigtable sorts them on the row-key

in principle i assume it will fit this way onto google bigtable ...  but i have no clue yet if practically this is modelled the right way

one could also look at it from 90 degrees and place attributes as rows and nodes / relations as famulity attributes ..

although i guess a bigtable is most efficient when one record is written at a time for a certain change... so i assume my model would be preferred..

using a google bigtable a data storage would give immense distributed scaleability i guess



any thoughts on my model ?

reg koen






Op maandag 27 augustus 2012 20:15:50 UTC+2 schreef Koen het volgende:

[Koen]

Lets call it....    "The BigGraph"

ps.. however i am not really a deep technical programmer (i am more from the extend-a-sample kind of programming still having a linear programming brain thinking still in NS diagrams .. for the people who know what that is ........)
.. so what level of coding expertise would be required here ? python... java ... c++ ..other ??    ..   and does this interest anyone out there ??

And other more general q:     starting from my limited linear programming background ...  ...    what would be the best area to invest my knowledge gaining in for building a graph based solution potentially running on google app engine ....  python or java ?
i would say python..

thanks

[Michael Hunger]

If you want to provide the neo4j-API I think it would be Java, 

for a proof of concept pyhton is cool I think.

With my limited knowledge of column stores I think your model is fine. Perhaps it would be nice to implement a thin API and see how it works / feels to fill a graph with data and then work with it / query it.

Michael

--
 
 

[James Thornton]

On Tuesday, August 28, 2012 3:49:18 AM UTC-5, Koen wrote:

Lets call it....    "The BigGraph"

ps.. however i am not really a deep technical programmer (i am more from the extend-a-sample kind of programming still having a linear programming brain thinking still in NS diagrams .. for the people who know what that is ........)
.. so what level of coding expertise would be required here ? python... java ... c++ ..other ??    ..   and does this interest anyone out there ?? 

Neo4j is in Java so if you are extending Neo4j, you would want to use Java. A key consideration is how App Engine handles transactions. 

App Engine stores entities and properties logically together like an object, and it supports blob properties up to 1 MB.  All single entity writes are transactional for that entity and its properties, and you can explicitly enable multi entity ("cross group") transactions on a per write basis. All transactions are optimistic. 

App Engine parallelizes  writes and has high write scalability, but it's limited to 1-10 writes per second per entity or entity group so this is an important consideration when modeling nodes and relationships in a transactional system like Neo4j.

You could model nodes and relationships as entities so when you write a node--relationship-->node transaction, you'll use a "cross group" transaction. The two nodes and relationship will be  limited to 1-10 writes per second, but the transaction does not affect other entities in the system. 

The other aspect is indexing. App Engine has its own indexing system so you would need to implement that as well. Until the new Neo4j indexing framework is ready, it may be easier to do this as a Blueprints implementation (https://github.com/tinkerpop/blueprints/wiki/). 

- James

[Michael Hunger]

writes to N1 should be combined in a single tx (e.g. by using a messaging system to combine/collect the appropriate messages in bulk).

Neo4j can easily combine 10-30k operations in a single TX. the smaller the transaction size the more the actual transaction overhead will cost compared to the operation time.

On google I would love to see a test of Neo4j on google compute engine, but haven't gotten my invite yet :)

Cheers

Michael

Am 07.01.2013 um 21:53 schrieb Jamie Olson:

I don't know much about the efficiency of App Engine Data Stores or how neo4j queries it's data, but I'd be somewhat concerned about performance given that write speed.  You say, "the transaction does not affect other entities in the system", but won't you be unable to make any other changes to groups involving those entities until the transaction completes?  

So if "N1 -- R1 -- N2" is an edge, you write a row with key date or some such and ancestors "N1/R1/N2", you could write multiple changes to N1 or to edge of type R1 originating at N1 as a single transaction.  You shoud get good performance only if you always make bulk changes in a single transaction, right?  You'd run into that 1-10 writes/second when making multiple changes to edges originating from N1 as separate transactions, right?

--
 
 

[Peter Neubauer]

A typical Lab-Friday or Neo4j Hackathon activity! Anyone around here
got an invite yet?

/peter

Cheers,

/peter neubauer

G: neubauer.peter
S: peter.neubauer
P: +46 704 106975
L: http://www.linkedin.com/in/neubauer
T: @peterneubauer

Neo4j 1.8 GA - http://www.dzone.com/links/neo4j_18_release_fluent_graph_literacy.html

> --
>
>

[Love Sharma]

http://stackoverflow.com/questions/26751346/how-do-you-set-up-neo4j-to-work-with-google-compute-engine/26752772#26752772