Is the issue make Neo4jGraph NOT fresh by default reintroduced in Tinkerpop blueprints 2.0?

[Michi]

Maybe I do something wrong, but when I use

Neo4jGraph neo4jGraph = new Neo4jGraph("test.db");

there is no node 0, so neoclipse don't show the database.
Whereas

GraphDatabaseService graphDb = new EmbeddedGraphDatabase("test.db");
Neo4jGraph neo4jGraph = new Neo4jGraph(graphDb, false);

Has a node 0 and shows up in neoclipse.

See: https://github.com/tinkerpop/blueprints/issues/206

How can I get the database to show up in neoclipse 1.7. Maybe creating
node[0] somehow,

I am using thinkerpop blueprints 2.0.0. and neo4j 1.8.M02

Michael

[Peter Neubauer]

That implementation is per default removing the reference node. Do the same code in gremlin as you do in java, and things should work?

Send from mobile.

[Marko Rodriguez]

Hi,

To the developers of Blueprints, Neo4j's Node 0 (reference node) is considered an awkward concept ... If you didn't create the data, why should it exist? For example, does MySQL have a row 0 for you when you create a new table?

Thus, if you create a clean (fresh directory) graph in Blueprints, Blueprints automagically removes vertex 0.
However, if you provide an already created graph, Blueprints will not delete vertex 0.

I hope that makes sense and you can move forward with confidence.

Thanks,
Marko

http://markorodriguez.com

[Michi]

I am not using Gremlin but I will use the work around with public
Neo4jGraph(final GraphDatabaseService rawGraph, boolean fresh) as
mentioned in the issue 206.
It is a little bit tricky because when I want to clear the graph I
iterate over the nodes check if it is node 0 and delete it if it is
not node 0. Without node 0 neoclipse doesn't show the graph. Is there
a better way of doing this, or a way to add node 0 programatically?

On 26 Mai, 15:20, Peter Neubauer <neubauer.pe...@gmail.com> wrote:
> That implementation is per default removing the reference node. Do the same
> code in gremlin as you do in java, and things should work?
>
> Send from mobile.

[Peter Neubauer]

Hehe,
Equally well one could argue the other way round. If I read the neo4j docs and blueprints removes data magically, is thatore right? I think imposing a unified model onto all underlying ecosystems at this level is going to end up on the danger of partly providing sub-optimal solutions (see for instance the differences in transaction semantics and the resulting performance hits in blueprints) and partly forcing an embedded, java implementation onto all kinds of data stores. I am very skeptical of that approach performance and technology wise.

Just sayin.

Send from mobile.

[Peter Neubauer]

I agree that is a shortcoming in neoclipse. Could you please raise an issue there?

Send from mobile.

[Marko Rodriguez]

Hello,

In order to have consistent behavior across all Blueprints implementations the following assumptions are made:

1. there is no data in a brand new database.

2. the transactional scope is consistent across indices and the graph.

Regarding 1., this is something that Neo4j is not going to change as its their style.

Regarding 2., these are both issues in the Neo4j issue tracker so they are known issues that will hopefully be fixed and the Blueprints fixes can be removed.

The best Blueprints does to allow variation from the spec is to support various Java Interfaces (e.g. TransactionalGraph, Graph, KeyIndexableGraph, IndexableGraph) and the notion of Features (in Blueprints 2).

We try our best to ensure that all databases have a consistent look-and-feel when viewed from the perspective of the TinkerPop stack.

Thanks,

Marko.

http://markorodriguez.com

[James Thornton]

On Saturday, May 26, 2012 11:25:05 AM UTC-5, Marko A. Rodriguez wrote:

In order to have consistent behavior across all Blueprints implementations the following assumptions are made:

1. there is no data in a brand new database.

2. the transactional scope is consistent across indices and the graph.

Regarding 1., this is something that Neo4j is not going to change as its their style.

I thought Neo4j was planning on getting rid of the reference node...

https://groups.google.com/d/msg/neo4j/1nwZk_Z4Ie0/UcnfaI8QZVgJ

https://github.com/neo4j/community/issues/203

- James

 

[Marko Rodriguez]

Hello,

Ah. I have never seen that. Great.

The "reference node" is a bit awkward in my opinion for these reasons:

1. You always have this one isolated vertex if you don't think "oh! my first vertex should be the reference node."

- you see this in the Cypher GUI "Rabbit Hole" that Neo4j promotes -- there is always the reference node just danglin' around in their examples.

2. It can cause problems when computing graph statistics (e.g. disconnected components, infinite diameter, etc.)

Anywho -- things evolve in different ways and you get variation :).

Thanks James,

Marko.

http://markorodriguez.com

[Marko Rodriguez]

Hi,

Perhaps I misread your original email -- are you saying you are passing in a "live database" and Blueprints is deleting the reference node (v[0]) or are you saying that you are creating a "fresh database" (new directory, never existed before) and its deleting the reference node (v[0]).

If the first, this is a bug.
If the second, this is expected.

Please confirm.

Thank you,
Marko.

http://markorodriguez.com

[Peter Neubauer]

Yup,
Regarding reference node there might change coming up :-)

Regarding performance, I think we need to start setting up performance regression tests. Our customers care first about performance, them about compliance, so we need to make sure TinkerPop 2 is not breaking that hopefully I can give that some love next week.

Send from mobile.

[Michi]

I ment the second, so it is expected.

[Marko Rodriguez]

Hi,

Regarding performance, I think we need to start setting up performance regression tests. Our customers care first about performance, them about compliance, so we need to make sure TinkerPop 2 is not breaking that hopefully I can give that some love next week.

Cool. You will notice that there is no "transaction checking" in indices when the developer IS NOT in a transaction. Thus, for read operations, the performance will be identical to previous Blueprints releases. However, if the user IS in a transaction, then to ensure that any deleted vertices are not visible in the index, the "transaction checking" clause kicks in. So this is where Blueprints 2.0.0 differs from previous releases around Neo4jGraph. Once Neo4j fixes the tx issues, then we can remove this "transaction checking" from Blueprints.

On a brighter note, Neo4jGraph in Blueprints 2 works much better with Neo4j native indexing. No longer do AutomaticIndices exist in Blueprints -- this has been replaced by KeyIndexableGraph and connected nicely to Neo4j's auto-indexer. Therefore, no more Index metadata from Blueprints in Neo4j indices and a better 1-to-1 mapping between the two APIs. This should be much faster as a great deal of logic was taken out of Neo4jGraph around indices.

Thanks,

Marko.

http://markorodriguez.coim

[Peter Neubauer]

Yup,
I hope things are good. Anyway, I think we need to start setting up regression tests so customers caring about performance are not getting surprised. Will report back.

Send from mobile.

[Pierre De Wilde]

Hehe,

I like threads where 2 smart guys are trying to demonstrate their approach is better than the other one.

In collective intelligençe, we're trying to go beyond the traditional competitive approach (I'm right, you're wrong: zero-sum interaction) by using the cooperative approach (I'm right, you're right: win-win interaction). Translated into geek-comprehensible language, it means AND instead OR.

Blueprints model is a very smart approach to unify disparate graph databases implementations. On top of Blueprints, other TinkerPop products show the power behind this model. And each graph database vendor has specific features that should be supported if possible.

A way to combine both views is to have an optional HashMap configuration when creating/opening a database.

For example, in this case:

g = new Neo4jGraph(pathname, [createReferenceNode=true])

Configuration HashMap should be completely vendor specific.

For instance, in OrientDB, instead of

g = new OrientGraph('local:pathname')

we would have a generic syntax like

g = new OrientGraph(pathname, [type='local'])

etc.

Default configuration may used as well. If the default type is local, we can use:

g = new OrientGraph(pathname)

Just my 2c,

Pierre

[Peter Neubauer]

Nah Pierre,
It's not about the implementation. We can make that work. Question is what happens if blueprints performance is degrading because of generalizations or standardization. I am just saying that for our customers sake we need to keep a close eye on that since that is the foremost concern for them.

So, I am just saying that, as there are changes coming in, performance needs to be regression tested before I recommend users to adopt new versions, as we do with neo4j and everything else. We are here to solve hard problems that are slow on RDBMSes, so for me performance regression is the number one concern here. We already have standards that in theory can express these use cases - called SQL. NoSQL is about better, pragmatic solutions to me, not more standards, at least not without clear performance benefits.

My 2 cents :-)

Send from mobile.

[Marko Rodriguez]

Hi,

> A way to combine both views is to have an optional HashMap configuration when creating/opening a database.
> For example, in this case:
> g = new Neo4jGraph(pathname, [createReferenceNode=true])
> Configuration HashMap should be completely vendor specific.

That is a very good idea.

Perhaps create an issue and provide a list of the types of options you seeing being needed by each respective database, where the "default" setting (e.g. no Map) is that which is considered consistent with the Blueprints specification.

Thanks Pierre,
Marko.

http://markorodriguez.com

[Luca Garulli]

+1 from OrientDB side

Lvc@

[Matthias Broecheler]

Haha, typically the "standardization is bad for performance" argument is used by monopolists who are afraid of loosing market share, it just never worked out historically. I know that's not what you were trying to get at, Peter, but the similarity was too striking to not call it out ;-)

Seriously, most of the hardware and software technology we are using today took some performance hit in favor of standardization: have a look at some of the wifi networking protocols - those are the definition of wasteful. I think the reason is that people like vendor lockin less than loosing a few percentage points of performance.

It is my understanding that Blueprints is attempting to be that standardization layer for graph databases. So, Blueprints has to make some assumptions in order to facilitate usage across implementations and as long as those are reasonable like "an empty database is empty" I don't see an issue. And even if there is a small performance degradation, standardization is a good enough selling point.

SQL is not inefficient on graphs because it is standardized, but because its the wrong tool for the job. Technically, that's not true either. You could efficiently execute a lot of SQL queries as graph traversals. RDBMS are the wrong tool for the job.

I agree with you that NoSQL is about pragmatic over dogmatic (i.e. "let the table rule 'em all"), but for NoSQL to have a sustainable future there must certainly be standardization. NoSQL will not exist in 5 years if you have to rewrite your code just because you want to migrate from DynamoDB to Hbase, both of which follow the BigTable data model, only because the two have different APIs. The cost of that is just prohibitively large.

And again, SQL is not inefficient - executing joins across distributed tables hosted by bloated RDBMS's is inefficient. And there is a lot of innovation happening right now that is overcoming this. Have a look at Daniel's recent work: he is running the TPC benchmark at 500,000 transactions per second on commodity amazon ec2 machines. That's very impressive. So, if NoSQL wants to survive the upcoming onslaught of use-case tailored and vastly improved RDBMS's it better be prepared.

That said, I am all in favor of performance regression testing. Having that would be a huge contribution. 

My 2 cents ;-)

--
Matthias Broecheler, PhD
http://www.matthiasb.com
E-Mail: m...@matthiasb.com

[Peter Neubauer]

Mattias,
If you read my last mail, I am saying exactly what you say - SQL is already able to express graphs, so I agree on that. What I don't agree on is that we lock down an API that is java centric, embedded and focused around one particular approach of data processing, take it as a standard and then force e.g. a. NET implementation to implement a java API which is the base for a graph algo in gremlin and groovy which is executed via RexterGraph over the wire (again ). All to be compliant with a blueprints client driver library.

Note that SQL is a declarative language, not a java library.

I am just saying that we run the risk of running down a path that is just wrong technically and locking innovation by mandating an implementation as a standard at the wrong abstraction level.

Wdyt?

Send from mobile.

[Matthias Broecheler]

I absolutely agree with you, Peter, that a declarative graph query language is needed. For me that is at a different abstraction layer, i.e. further to the top of the stack.

Traditionally, database system have been these big monolithic components integrating everything from the lowest disk access level (Oracle even has their own file system) up to the SQL query language level. I think that is the reason we have seen so little innovation in the RDBMS world, because it was impossible to say "hey, I have this super efficient algorithm to re-order joins" and build a company around that. For that, you would have to rebuild the entire stack, so the most you could hope for is a publication at SIGMOD. I think the NoSQL movement is finally changing that. Now, we have many key-value store, Btree, write ahead-logging and other database component implementations that people can assemble into special purpose database software. And we will have more abstraction and multiple layers inside databases which will allow people so swap out components individually - much like we are already used to with non-database software which typically have a ton of dependencies. Similarly, you are running an e-commerce site? Then run your database with this storage backend, that data model and plug-in this query optimizer.

Blueprints is one layer of abstraction at an intermediate level and I think it does have its place and role in that - independent of higher level abstractions.

For example. let's say you want to implement a query language for graph particle swarm algorithms. Then you could implement it as an ANTL script that compiles it to Blueprints and voila it runs on a bunch of graph databases.

So, I don't think Blueprints is the wrong level of abstraction. It's wrong for those users that are looking for an SQL like interface (and I agree that a lot if not most users are of that nature) but its right for those users who are at the cutting edge or those who want to build query languages and tools that exploit the power of graphs to solve complex relational problems. And I think that's a very important group of individuals that will drive innovation in the graph database space and demonstrate to the Stonebraker's of the world that graph databases are more than just a rehashing of a 50 year old idea that has already failed. 

At least I strongly hope that's what's going to happen ;-)

[Peter Neubauer]

Agreed on most of it,
Except that blueprints is a java API that is "used to run efficienrly on a bunch of java database implementations that can process data in a stepwise manner"

That doesn't sound like a standard to me, but a library useful for a certain set of tools to build. For neo4j we have never claimed any standard aspirations for exactly that reason. Usefulness! = standardization Imho.

And yes, there are other abstraction points like network protocols and other that might be far more useful to others. So this is the reason I am against a blueprints standard. Far more interesting to me is the pipes dataflow approach as a thought, currently working great with neo4j. Looping back to the original question, the reference node is an implementation details that is manageable. "you are not compliant with the standard " talk (given that Aurelius with Marko and you employed is starting to have own vested interest on that standard implementation while calling out others as vendors) at this level is just wrong wording.

Send from mobile.