Orient Database Server Transaction

[Junji Takakura]

A exception occurred during the execution of the below code.
(using Orient Database Server v0.9.22 and Orient Database Server
v0.9.23-SNAPSHOT)
And a OConcurrentModificationException occurred even if the rollback
method was called when I re-execute.
Dose it require some configurations or is it known issue?
-----
com.orientechnologies.orient.core.exception.OStorageException: Error
on commit
at
com.orientechnologies.orient.client.remote.OStorageRemote.handleException(OStorageRemote.java:
879)
at
com.orientechnologies.orient.client.remote.OStorageRemote.commit(OStorageRemote.java:
524)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.executeCommit(ODatabaseRecordTx.java:
102)
at
com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:
34)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(ODatabaseRecordTx.java:
65)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(ODatabaseRecordTx.java:
1)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.commit(ODatabaseRecordWrapperAbstract.java:
52)
at example.Example.main(Example.java:12)
Caused by: java.lang.NullPointerException: null
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:
39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:
513)
at
com.orientechnologies.orient.client.remote.OStorageRemote.createException(OStorageRemote.java:
1067)
at
com.orientechnologies.orient.client.remote.OStorageRemote.readStatus(OStorageRemote.java:
869)
at
com.orientechnologies.orient.client.remote.OStorageRemote.commit(OStorageRemote.java:
521)
... 6 more
-----
package example;

import
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx;

public class Example {

public static void main(String[] args) {
ODatabaseDocumentTx database = new
ODatabaseDocumentTx("remote:localhost/demo").open("admin", "admin");
try {
database.begin();
database.newInstance("Items").field("Name", "Pen").save();
// bellow case was also given same exception,
OStorageException.
/*
ODocument doc = database.newInstance("Cars");
doc.field("name", "Bentz");
doc.save();
*/
database.commit();
} catch (Exception ex) {
ex.printStackTrace();
database.rollback();
} finally {
database.close();
}
}
}
-----

[Luca Garulli]

Please can you re-test it with latest SVN fix?

Thank you.

bye,

Lvc@

[Junji Takakura]

I re-tested with SVN revision 1245(not the latest because build error
was occurred with latest, revision 1246).
The process was stoped when the method of commit was called.
(I waited 10 min but not returned.)

Thanks.

On 9月20日, 午後5:52, Luca Garulli <l.garu...@gmail.com> wrote:
> Please can you re-test it with latest SVN fix?
>
> Thank you.
>
> bye,
> Lvc@
>

[MarkH]

This is effecting me too. Still getting a build error with latest SVN
rev (1428):

compile:
[javac] Compiling 7 source files to D:\javalibs\graph stuff
\orientDbSvnRepos\client\target\classes
[javac] D:\javalibs\graph stuff\orientDbSvnRepos\client\src\main
\java\com\orientechnologies\orient\client\remote\OStorageRemote.java:
66: com.orientechnologi
es.orient.client.remote.OStorageRemote is not abstract and does not
override abstract method
addCluster(java.lang.String,com.orientechnologies.orient.core.storage.OStorage.CLUSTER_TYPE,java.lang.Object...)
in com.orientechnologies.orient.core.storage.OStorage [javac]
public class OStorageRemote extends OStorageAbstract {

[Luca Garulli]

Hi,

sorry for the mistake. Now compiles, but I hope you all will be happy to know that benchmarks show an improvement of 10x on massive insert involving a TreeMap (indexes, dictionaries and Key/Value engine).

bye,

Lvc@

[MarkH]

Thanks for the quick response, Luca!
Still getting a build error unfortunately:

compile:
[javac] Compiling 29 source files to D:\javalibs\graph stuff
\orientDbSvnRepos\tests\target\test-classes
[javac] D:\javalibs\graph stuff\orientDbSvnRepos\tests\src\test
\java\com\orientechnologies\orient\test\database\users
\SecMaskTest.java:94:
addCluster(java.lang.String,com.orientechnologies.orient.core.storage.OStorage.CLUSTER_TYPE,java.lang.Object...)
in com.or
ientechnologies.orient.core.storage.OStorage cannot be applied to
(java.lang.String,java.lang.String)
[javac] .createClass("Account",
database.getStorage().addCluster("account", OClusterLocal.TYPE));

[javac]
^
[javac] 1 error


Looks like its confined to a test class.

[Luca Garulli]

Yes,

it's the class where I'm working. Committed.

Only 1 test unit fails in remote configuration but everything should compile.

bye,

Lvc@

[MarkH]

Great, thanks. Compiles OK and happy to report that the
ConcurrentModificationException I was previously getting when
controlling transactions manually has gone.

I'm working on loading full IMDB (~3million nodes, 10-million edges)
via Tinkerpop so I can test performance of indexing and the typical
"Kevin Bacon" pathfinding games.
I'm getting up to speed with the various docs so any advice on
optimisations for this scale e.g. transaction batch sizes will be
gratefully received!

Cheers
Mark

[Gian Luca Farina Perseu]

> I'm working on loading full IMDB (~3million nodes, 10-million edges)
> via Tinkerpop so  I can test performance of indexing and the typical
> "Kevin Bacon" pathfinding games.

Hi Mark, sorry for the intrusion ...

I'm starting studing OrientDB .... where I can find full IMDB graph ?

And what do you mean with 'typical "Kevin Bacon" pathfinding games' ?

I need to heavly test my knowledge (currently = 0) on graph DB ....

Many thanks !

gianluca

[Luca Garulli]

Are you working to the project http://github.com/tinkerpop/graphdb-bench ?

If you post on a public repository your code & results I can help you to optimize it. However if you're making a massive insert I suggest you to:

(1) Use the "local" protocol instead of remote

(2) Declare the "massive insert" intent. OrientDB will auto-tune itself for it. At the end reset the intent to restore original parameters:

db.declareIntent( new OIntentMassiveInsert() );

...

db.declareIntent( null );

Good luck :-)

bye,

Lvc@

[Gian Luca Farina Perseu]

> At the end reset the intent to restore original parameters:
> db.declareIntent( new OIntentMassiveInsert() );
> ...
> db.declareIntent( null );
>

Cool !

;-)

glfp

[MarkH]

> Are you working to the projecthttp://github.com/tinkerpop/graphdb-bench?

No - but I may take a look at that now you've brought it to my
attention.

> If you post on a public repository your code & results I can help you to
> optimize it. However if you're making a massive insert I suggest you to:
>
> (1) Use the "local" protocol instead of remote
>
> (2) Declare the "massive insert" intent. OrientDB will auto-tune itself for
> it. At the end reset the intent to restore original parameters:

Thanks for the help. Yes, using "local".
Changing to "declareIntent" gave me an interesting new cast error when
creating an edge:
java.lang.ClassCastException:
com.orientechnologies.orient.core.id.ORecordId cannot be cast to
com.orientechnologies.orient.core.record.impl.ODocument
at
com.orientechnologies.orient.core.db.graph.OGraphVertex.getInEdges(Unknown
Source)
at
com.orientechnologies.orient.core.db.graph.OGraphVertex.link(Unknown
Source)
at
com.tinkerpop.blueprints.pgm.impls.orientdb.OrientGraph.addEdge(OrientGraph.java:
48)
at
com.detica.graph.orientdb.TestLoadOrientDbWithImdbViaTinkerpop.main(TestLoadOrientDbWithImdbViaTinkerpop.java:
92)

Presumably "massive inserts" cannot involve creating edges between new
nodes?
I'm inserting "actor-centric" data ie actors along with their related
movies in this order:
actor1
movie1
edge actor1-movie1
actor2
edge actor2->movie1
movie2
edge actor2->movie2
...

That's the way my data arrives in my apps
Cheers
Mark

>
> db.declareIntent( new OIntentMassiveInsert() );
> ...
> db.declareIntent( null );
>
> Good luck :-)
>
> bye,
> Lvc@
>

[Luca Garulli]

Hi,

one of the optimizations is in collections: they are lazy until the first access.

I've fixed the code in SVN repository. Can you re-test it with your importer app?

bye,

Lvc@

[Junji Takakura]

Hi Luca,
I re-test my app with latest(revision 1434).

1) Build OrientDB.
2) Run "create database" command by console app.
3) Run my test app.

Still getting bellow exception.
But no exception was occurred if the class was already created, for
example by console app.

Thanks.
----------
com.orientechnologies.common.io.OIOException: You can't access outside
the file size (0 bytes). You've requested portion 15-17 bytes

at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:
39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:

27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:
513)
at

com.orientechnologies.orient.client.remote.OStorageRemote.createException(Unknown
Source)
at
com.orientechnologies.orient.client.remote.OStorageRemote.readStatus(Unknown
Source)
at
com.orientechnologies.orient.client.remote.OStorageRemote.commit(Unknown
Source)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.executeCommit(Unknown
Source)
at
com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(Unknown
Source)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(Unknown
Source)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(Unknown
Source)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.commit(Unknown
Source)
at example.Example.main(Example.java:23)

On 9月21日, 午前2:44, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi,
> one of the optimizations is in collections: they are lazy until the first
> access.
>
> I've fixed the code in SVN repository. Can you re-test it with your importer
> app?
>
> bye,
> Lvc@
>

[Luca Garulli]

Hi,

could the problem be the schema that is not updated and therefore don't see the class until reopen of the database?

Lvc@

[Luca Garulli]

Please try the SVN revision: 1437

Lvc@

[MarkH]

IMDB data I think I originally got via this app: http://www.jmdb.de/

>>And what do you mean with 'typical "Kevin Bacon" pathfinding games' ?

It's a reference to a common type of test in social network analysis.
See http://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon

[MarkH]

> I've fixed the code in SVN repository. Can you re-test it with your importer
> app?

Will do

[Junji Takakura]

Hi,
I retry with the latest, rev: 1437 and it seems fixed(no exception was
occured).

Thanks.

On 9月21日, 午前3:15, Luca Garulli <l.garu...@gmail.com> wrote:
> Please try the SVN revision: 1437
>
> Lvc@
>

> On 20 September 2010 20:10, Luca Garulli <l.garu...@gmail.com> wrote:
>
>
>
> > Hi,
> > could the problem be the schema that is not updated and therefore don't see
> > the class until reopen of the database?
>
> > Lvc@
>

> > On 20 September 2010 20:06, Junji Takakura <j.takak...@gmail.com> wrote:
>
> >> Hi Luca,
> >> I re-test my app with latest(revision 1434).
>
> >> 1) Build OrientDB.
> >> 2) Run "create database" command by console app.
> >> 3) Run my test app.
>
> >> Still getting bellow exception.
> >> But no exception was occurred if the class was already created, for
> >> example by console app.
>
> >> Thanks.
> >> ----------
> >> com.orientechnologies.common.io.OIOException: You can't access outside
> >> the file size (0 bytes). You've requested portion 15-17 bytes
> >>         at
> >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> >>        at
>

> >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce ssorImpl.java:
> >> 39)
> >>        at
>
> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru ctorAccessorImpl.java:

[Gian Luca Farina Perseu]

On Mon, Sep 20, 2010 at 8:49 PM, MarkH <markh...@gmail.com> wrote:
> IMDB data I think I originally got via this app: http://www.jmdb.de/
>

ok, thanks.

>>>And what do you mean with 'typical "Kevin Bacon" pathfinding games' ?
>
> It's a reference to a common type of test in social network analysis.
> See http://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon

Ha ha !!! That's interesting !! i'd know "six degree" but not applied
on Kevin Bacon ! Thanks for the reference !

Gianluca

[MarkH]

Thanks for the update.
I've managed to get things running smoothly but unfortunately the
insert costs seem directly linear with the database size whereas to
scale I need sub-linear.
What this means is that my IMDB import is running but getting
progressively slower to insert every record and so looks like it
ultimately won't complete in my lifetime.

Pasted below is my OrientDB/Tinkerpop code based on this morning's
head revision of OrientDB. Am I doing something wrong here?

Cheers
Mark
=======================
Begin code
=======================
package com.detica.graph.orientdb;

import java.io.File;

import org.apache.lucene.document.Document;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.store.FSDirectory;

import com.orientechnologies.orient.core.db.graph.ODatabaseGraphTx;
import com.orientechnologies.orient.core.intent.OIntentMassiveInsert;
import com.tinkerpop.blueprints.pgm.Edge;
import com.tinkerpop.blueprints.pgm.Element;
import com.tinkerpop.blueprints.pgm.Index;
import com.tinkerpop.blueprints.pgm.Vertex;
import com.tinkerpop.blueprints.pgm.impls.orientdb.OrientGraph;

/**
* Imports IMDB data from a Lucene store where records appear as an
actor record followed by
* "role" records which list that actor's roles in different movies.
* Each actor record always creates a new node/vertex.
* Each "role" record either creates a new movie node or finds an
existing movie node then
* creates an edge to it.
* Explicit transaction control has been removed in favour of a
"declareIntent" hint to OrientDb
* @author MAHarwood
*
*/
public class TestLoadOrientDbWithImdbViaTinkerpop
{

private static final int NUM_RECS_PER_BATCH = 1000;

/**
* @param args
*/
public static void main(String[] args) throws Exception
{
IndexReader r=IndexReader.open(FSDirectory.open(new File("D:\\indexes
\\imdbactorcentric2")),true);

OrientGraph graph = new OrientGraph("local:C:/temp/graph/graph");
if(graph.exists())
{
System.out.println("Graph exists - clearing");
graph.open("admin", "admin");
graph.clear();
}
else
{
System.out.println("Graph does not exist - creating");
graph.create();
}


ODatabaseGraphTx gtx=graph.getRawGraph();
gtx.declareIntent(new OIntentMassiveInsert());
Index index = graph.getIndex();
long start=System.currentTimeMillis();
//Removed explicit transaction control and opted for "declareIntent"
approach shown above.
// gtx.begin();
try{


Vertex actorNode = null;

for(int i=0;i<r.maxDoc();i++)
{
Document doc=r.document(i);
String actorId=doc.get("nodeid");
if(actorId!=null)
{
//This is an actor record - always insert as new
actorNode= graph.addVertex(null);
actorNode.setProperty("id", actorId);
actorNode.setProperty("personName", doc.get("personName"));
index.put("id", actorId, actorNode);
}
else
{
//This is an movie role record - find existing movie node or
create new
// then link actor to it.
String movieId=doc.get("targetNodeid");
if(movieId!=null)
{
String movieTitle=doc.get("movieTitle");
Vertex movie=null;
Iterable<Element> elems = index.get("id", movieId);
if(elems!=null)
{
movie=(Vertex) elems.iterator().next();
}
if(movie==null)
{
movie= graph.addVertex(null);
movie.setProperty("id", movieId);
movie.setProperty("movieTitle", movieTitle);
index.put("id", movieId, movie);
}
Edge edge=graph.addEdge(null, actorNode, movie, "StarredIn");
edge.setProperty("role", doc.get("movieRole"));

}
else
{
System.err.println("Unknown type for recod#"+i);
}
} //end is movie

if(i%NUM_RECS_PER_BATCH==0)
{
long diff=System.currentTimeMillis()-start;
System.out.println("["+i+"] actor
="+actorNode.getProperty("personName") +" ( " + NUM_RECS_PER_BATCH+
" docs in "+diff+" ms)");
start=System.currentTimeMillis();
// gtx.commit();
}
}


// System.out.println("Committing changes");
// gtx.commit();

} catch( Exception e ) {

// System.err.println("Rolling back:");
// gtx.rollback();
e.printStackTrace();
}
System.out.println("Shutdown");
graph.shutdown();
System.out.println("done");
}
}

[Luca Garulli]

Hi,

to obtain real linear performance with OrientDB I suggest you to avoid transactions at all! In facts OrientDB keeps in memory all the changes until you flush it with a commit. So the bottleneck is your Heap space.

Just remove begin() and commit() and retry.

Then I suggest you to run your application using these parameters of the JVM:

-server -XX:+AggressiveOpts -XX:CompileThreshold=200

Let me know!

bye,

Lvc@

[Luca Garulli]

Ops,

I forgot to say that transactions slow down massive inserts unless you're using a "remote" connection. In that case it speeds up all the insertion because the client/server communication happens only at commit time.

However if you need light speed performance use "local" as connection type.

There are a lot of other tips I'll write in the WIKI time per time.

Bye,

Luca Garulli

[MarkH]

Hi Luca,
begin() and commit() are commented out in this code.
I'm comparing this with a neo4j insert of the same data and I have to
say neo4j is sub-linear and significantly faster (staying stable at
~100 ms per 1,000 records inserted) whereas OrientDB code above
started out at 400ms per 1000 records and climbed steadily to 10
seconds per 1,000 after 500k records inserted. Neo4j I'm using via
native API and committing every 100k records. Each of these neo4j
commits takes some time (worst so far being 2 minutes) but it does
appear to be maintaining stable (and scalable) insert costs otherwise.


Cheers
Mark

On Sep 21, 11:59 am, Luca Garulli <l.garu...@orientechnologies.com>
wrote:

> Hi,
> to obtain real linear performance with OrientDB I suggest you to avoid
> transactions at all! In facts OrientDB keeps in memory all the changes until
> you flush it with a commit. So the bottleneck is your Heap space.
>
> Just remove begin() and commit() and retry.
>
> Then I suggest you to run your application using these parameters of the
> JVM:
>
> -server -XX:+AggressiveOpts -XX:CompileThreshold=200
>
> Let me know!
>
> bye,
> Lvc@
>

[Luca Garulli]

Hi,

can you try to disable the index? I see that you manually control the index.

Try this:

index.indexAll(false);

Let me know if is faster now.

Lvc@

[Luca Garulli]

Hi,

I can't see manual management of tx. Put this after db creation/open:

graph.setTransactionMode( TransactionalGraph.Mode.MANUAL );

This avoid to create a single transaction per EACH operation ;-)

Let me know if it's faster.

bye,

Lvc@

[MarkH]

> can you try to disable the index? I see that you manually control the index.

Much better! I'll leave it running and report back...

[MarkH]

Disabling the "index all" mode kept insert speeds constant at around
200ms per 10000 records.
I'll try adding the manual transaction control line too now in the
next run.

In the last run I hit an exception on this line though after
6,014,000 inserts:
actorNode.setProperty("id", actorId);
The exception was
com.orientechnologies.orient.core.exception.ODatabaseException: Error
on saving record in cluster id: 5, position: 6014835
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.save(Unknown
Source)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(Unknown
Source)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.save(Unknown
Source)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(Unknown
Source)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(Unknown
Source)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(Unknown
Source)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(Unknown
Source)
at
com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(Unknown
Source)
at
com.orientechnologies.orient.core.record.ORecordAbstract.save(Unknown
Source)
at
com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.save(Unknown
Source)
at
com.orientechnologies.orient.core.record.ORecordSchemaAwareAbstract.save(Unknown
Source)
at
com.orientechnologies.orient.core.type.ODocumentWrapper.save(Unknown
Source)
at
com.orientechnologies.orient.core.db.graph.OGraphVertex.save(Unknown
Source)
at
com.tinkerpop.blueprints.pgm.impls.orientdb.OrientElement.save(OrientElement.java:
66)
at
com.tinkerpop.blueprints.pgm.impls.orientdb.OrientElement.setProperty(OrientElement.java:
27)
at
com.detica.graph.orientdb.TestLoadOrientDbWithImdbViaTinkerpop.main(TestLoadOrientDbWithImdbViaTinkerpop.java:
73)
Caused by: java.lang.IllegalArgumentException: Can't enlarge file
since the configured max size (500.00Mb) was reached! File:
default.odh os-size=500.00Mb, stored=500.00Mb, filled=500.00Mb,
max=500.00Mb
at
com.orientechnologies.orient.core.storage.fs.OFile.allocateSpace(Unknown
Source)
at
com.orientechnologies.orient.core.storage.impl.local.ODataLocalHole.createHole(Unknown
Source)
at
com.orientechnologies.orient.core.storage.impl.local.ODataLocal.setRecord(Unknown
Source)
at
com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.updateRecord(Unknown
Source)
at
com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.updateRecord(Unknown
Source)
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.save(Unknown
Source)

Looks like there is a config setting that needs tweaking in my app.
Many thanks for all your help.

[MarkH]

FYI - I've posted raw data, graph, settings and source in a
spreadsheet here:
https://spreadsheets.google.com/ccc?key=0AsKVSn5SGg_wdC1EVno2Z255RWgtenJ0d1kzOGo5bkE&hl=en&authkey=CKLqy5sP

[Luca Garulli]

Hi,

by the way you can also turn off the cache with:

graph.setUseCache( false );

This should speed up more again.

In your spreadsheet you've write about a commit every 10,000. Is it true?

I see there is a problem after the 742,315th. Can you share the file "imdbactorcentric2" so I can test it?

Lvc@

[Gian Luca Farina Perseu]

>Can you share the file
> "imdbactorcentric2" so I can test it?

I'm also interested in this file :-)

[MarkH]

> >Can you share the file
> > "imdbactorcentric2" so I can test it?

~2GB of data I'm afraid so sharing is not easy.
I imagine something like it may already be available to download
elsewhere?

[Luca Garulli]

Hi,

you can find original files here: http://www.imdb.com/interfaces

Lvc@

[Luca Garulli]

Hi Marko,

the stack trace in your spreadsheet shows me that there is a problem while allocating space. The problem resides when a segment is full and OrientDB uses a new one. I've tested with millions of records but this error won't happen. I'd like to have your original file in order to reproduce it.

Can you zip it and share in a ftp server?

bye,

Lvc@

On 21 September 2010 18:11, MarkH <markh...@gmail.com> wrote:

[Luca Garulli]

Hi,

I've launched your test and it's just ended without errors after changed the default maxSize to 0 (means no limits). I've committed these minor changes in SVN trunk. I suggest you to update your SVN and retry the test. Let me know.

I've changed your source to add these lines of code just after the graph database instance is retrieved:

graph.setTransactionMode(TransactionalGraph.Mode.MANUAL);

graph.setUseCache(false);

graph.getIndex().indexAll(false);

The insertion time has been always constant, but some peak sometime due to my background running applications. The average is about 0,1 ms per record (100ms was the average per 1,000 records). But consider that in this time there is also the Lucene read of the document.

I've also added a print of times:

Insertion of 13982181 docs completed in 2555091 ms

That means 0,18 ms per movie. Not bad :-) What was the results of Neo4J for this task?

bye,

Luca Garulli

[MarkH]

Good stuff! I make that just over 40 minutes.
I ran the same test with neo4j earlier and it took a couple of hours
(maybe the neo4j guys can offer some similar performance tweaking
tips).
I'll publish those results once my latest run completes.
I'll then re-run OrientDB insert tests with your suggested settings.

Then on to query benchmarks....

Thanks again for your help!

Mark

[MarkH]

OK. Neo4j results here: https://spreadsheets.google.com/ccc?key=0AsKVSn5SGg_wdGdEMjQ2ZGZ0eGZSRTlEWGxlQy1rRXc&hl=en

[MarkH]

Hmm. Still getting that crash at record #742k with OrientDB based on
latest code (rev 1460).
I've updated the source,stack trace and timings here
https://spreadsheets.google.com/ccc?key=0AsKVSn5SGg_wdC1EVno2Z255RWgtenJ0d1kzOGo5bkE&hl=en&authkey=CKLqy5sP

[Luca Garulli]

Hi,

are you sure to use the latest version of OrientDB? I've committed some more log in case of error. Can you re-run the test and send me the full stack trace?

bye,

Lvc@

[MarkH]

> log in case of error. Can you re-run the test and send me the full stack
> trace?

No - compile error:

compile:
[javac] Compiling 1 source file to D:\javalibs\graph stuff
\orientDbSvnRepos\
server\target\classes
[javac] D:\javalibs\graph stuff\orientDbSvnRepos\server\src\main
\java\com\or
ientechnologies\orient\server\network\protocol\binary
\ONetworkProtocolBinary.jav
a:227: cannot find symbol
[javac] symbol : variable DATACLUSTER_COUNT
[javac] location: class
com.orientechnologies.orient.enterprise.channel.bina
ry.OChannelBinaryProtocol
[javac] case OChannelBinaryProtocol.DATACLUSTER_COUNT: {
[javac] ^
[javac] D:\javalibs\graph stuff\orientDbSvnRepos\server\src\main
\java\com\or
ientechnologies\orient\server\network\protocol\binary
\ONetworkProtocolBinary.jav
a:241: cannot find symbol
[javac] symbol : variable DATACLUSTER_DATARANGE
[javac] location: class
com.orientechnologies.orient.enterprise.channel.bina
ry.OChannelBinaryProtocol
[javac] case OChannelBinaryProtocol.DATACLUSTER_DATARANGE: {
[javac] ^
[javac] D:\javalibs\graph stuff\orientDbSvnRepos\server\src\main
\java\com\or
ientechnologies\orient\server\network\protocol\binary
\ONetworkProtocolBinary.jav
a:252: cannot find symbol
[javac] symbol : variable DATACLUSTER_ADD
[javac] location: class
com.orientechnologies.orient.enterprise.channel.bina
ry.OChannelBinaryProtocol
[javac] case OChannelBinaryProtocol.DATACLUSTER_ADD: {
[javac] ^
[javac] D:\javalibs\graph stuff\orientDbSvnRepos\server\src\main
\java\com\or
ientechnologies\orient\server\network\protocol\binary
\ONetworkProtocolBinary.jav
a:282: cannot find symbol
[javac] symbol : variable DATACLUSTER_REMOVE
[javac] location: class
com.orientechnologies.orient.enterprise.channel.bina
ry.OChannelBinaryProtocol
[javac] case OChannelBinaryProtocol.DATACLUSTER_REMOVE: {
[javac] ^
[javac] 4 errors

[Luca Garulli]

Have you tried an

> ant -f build-db.xml clean

> ant -f build-db.xml 

or you can test the new:

> mvn install

Lvc@

[Luca Garulli]

I've commited other stuff. can you test it?

bye,

Lvc@

[MarkH]

I hope to take a look tomorrow afternoon when I'm back in the office
Cheers
Mark

[MarkH]

Synced to rev 1466.
Compile issues gone (I used ant clean as suggested)

Still have the crash after 742k records.

I've updated the spreadsheet with times, source, stacktrace.

This works for you?

I tried updating TinkerPop Blueprints to latest source but this seems
to have regressed in its version of OrientGraph class and
setTransactionMode etc methods are now missing. I'm sticking with a
Tinkerpop blueprints build from Sep 21st/


Cheers
Mark

[Luca Garulli]

Hi,

yesterday night was taken the prompt decision to release the Blueprints, and the full stack, as is. This is the reason why the new improvement of OrientDB is missing. Now I'm committing the new version but you need to wait the official inclusion but you use my own fork:

http://github.com/lvca/blueprints

bye,

Lvc@

[MarkH]

Just tried the blueprints fork (with the transactional APIs) but still
get the "invalid range" request on the oda file after 742k records.

[Luca Garulli]

The crazy thing is that with exactly the same your file works on my PC. I don't would you have a different version.

Can you send me the output? In the latest version I've improved it.

Lvc@

[MarkH]

Source, stats and stack trace updated here:
https://spreadsheets.google.com/ccc?key=0AsKVSn5SGg_wdC1EVno2Z255RWgtenJ0d1kzOGo5bkE&hl=en&authkey=CKLqy5sP
Do you have special environment variables or vm args to launch the
test?

Mark