Improving select performace

[Richard Kolb]

Hello

I am new to OrientDB and graph databases.
(I'm using OrientDB 1.4.0 with blueprints-orient-graph 2.4.0-SNAPSHOT)

Here is a sample of my code :
        final OClass node = odb.createVertexType("Node");
        node.createProperty("nodeid", OType.LONG);
        node.createProperty("lat", OType.DOUBLE);
        node.createProperty("lon", OType.DOUBLE);
        node.createIndex("nodeididx", OClass.INDEX_TYPE.UNIQUE, "nodeid");

Then I create 400K Nodes.

The performace of a getVertices is sometimes very fast (2ms) and sometime 20 seconds
                Iterable<Vertex> vs = graph.getVertices("nodeid", nodeId);
                final Iterator<Vertex> it = vs.iterator();
                if (it.hasNext()) {

I have tried other things like the following, but again it's very slow:
               Iterable<Vertex> vs = graph.query().has("nodeid", nodeId).limit(1).vertices();

Can someone help me improve the speed please. Perhaps the index it not right.

thanks
Richard.

[Luca Garulli]

Hi Richard,

you've a unique index against the nodeid, so it's weird such difference in performance. Time should be always the same for any key. When do you have 20 seconds? Maybe on startup when the index was rebuilding in background after a kill of the JVM?

Lvc@

Richard.

--
 
---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Richard Kolb]

Hi Luca

Thanks for the very fast response.

I am actually importing an OpenStreetMap OSM file, so first are all the node information and then the all relations. So I am guessing the index is not rebuilding.

Below are some times I am getting back from the following function :

            private void nd(Long nodeId) {
                Stopwatch stopwatch = new Stopwatch().start();

                Iterable<Vertex> vs = graph.query().has("nodeid", nodeId).limit(1).vertices();

                //Iterable<Vertex> vs = graph.getVertices("nodeid", nodeId);

                final Iterator<Vertex> it = vs.iterator();
                if (it.hasNext()) {

                    Vertex v = it.next();
                    vertexs.add(v);
                }
                stopwatch.stop();
                System.out.println("nd " + stopwatch.elapsed(TimeUnit.MILLISECONDS) + "ms");
            }

Thanks

Richard.

nd 732ms
nd 2748ms
nd 235ms
nd 439ms
nd 233ms
nd 259ms
nd 61ms
nd 469ms
nd 40ms
nd 3047ms
nd 37ms
nd 33ms
nd 33ms
nd 40ms
nd 1ms
nd 2ms
nd 1ms
nd 397ms
nd 1ms
nd 63ms
nd 2947ms
nd 64ms
nd 2662ms
nd 2825ms

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/pClF81KBfR4/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

[Luca Garulli]

Hi,

how did you create the index? Times should be always < 100ms with such data volume.

Lvc@

[Richard Kolb]

Hi Luca

Yes, it seems it is not hitting the index. I have checked the web gui and the index does exist.

I created the index when I created the Vertex as such.

        final OClass node = odb.createVertexType("Node");
        node.createProperty("nodeid", OType.LONG);
        node.createProperty("lat", OType.DOUBLE);
        node.createProperty("lon", OType.DOUBLE);
        node.createIndex("nodeididx", OClass.INDEX_TYPE.UNIQUE, "nodeid");

I'm trying to prove that OrientDB can handle massive complex graphs.

I've attached the source code which is one class really. Maybe I am doing something really silly.

It just loads OSM file. And for the moment, just one City with 400k nodes.
http://wiki.openstreetmap.org/wiki/OSM_Map_On_Garmin/Download

regards

Richard

[Richard Kolb]

Hi Luca

If you get that far I am looking at this OSM file.
http://download.bbbike.org/osm/bbbike/Johannesburg/Johannesburg.osm.gz (6 meg)

regards

Richard

[Luca Garulli]

Hi,

the right API is createKeyIndex():

odb.createKeyIndex("nodeid", Vertex.class, new Parameter("type", "UNIQUE"), new Parameter("class", "Node"));

Lvc@

[Richard Kolb]

Thanks!

regards

Richard

[Richard Kolb]

Hi Luca

Sorry, but the performance is better but still not below 100ms.

From the previous "void nd(Long nodeId) {" code I sent I am now getting the below times and errors:

(The Cannot ? the record #11:457 because the version is not the latest. goes away if I use a OrientGraph instead of a OrientGraphNoTx, but the performance is is still quite slow.

I am running on my Laptop

Java 1.7.0_13

Quad core i7 2.4Ghz

SSD drive

Ubuntu Linux 13.04

connect ways 481807 vertexs 50809
Cannot ? the record #11:457 because the version is not the latest. Probably you are ing an old record or it has been modified by another user (db=v3 your=v0)
nd 644ms
nd 60ms
nd 560ms
nd 559ms
nd 288ms
nd 211ms
connect ways 481817 vertexs 50815
Cannot ? the record #11:457 because the version is not the latest. Probably you are ing an old record or it has been modified by another user (db=v3 your=v0)
nd 572ms
nd 637ms
connect ways 481823 vertexs 50817
Cannot ? the record #11:457 because the version is not the latest. Probably you are ing an old record or it has been modified by another user (db=v3 your=v0)
nd 70ms
nd 568ms
nd 2728ms
connect ways 481829 vertexs 50820
Cannot ? the record #11:457 because the version is not the latest. Probably you are ing an old record or it has been modified by another user (db=v3 your=v0)
nd 1405ms
nd 1516ms

regards

Richard

[Richard Kolb]

Just FYI.

I switched to the local storage and I am getting better results.

Still quite inconsistent, but better.

I loaded the entire Africa OSM (8 gig XML) into Neo4j in a couple of hours using a modified version of this :
https://github.com/EugeneBorshch/neo4j-osm-routing

connect ways 405584 vertexs 74
way 30ms
nd 7ms
nd 7ms
nd 6ms
nd 512ms
nd 6ms
nd 534ms
nd 7ms
nd 7ms
nd 7ms
nd 1385ms
nd 7ms
nd 182ms
nd 7ms
nd 1403ms
nd 7ms
nd 1430ms
nd 7ms
nd 6ms
nd 1412ms
nd 1393ms
nd 1375ms
nd 7ms
nd 1423ms
nd 1467ms
nd 7ms
nd 1421ms
nd 1384ms
nd 1444ms
nd 1441ms
nd 1457ms
nd 7ms
nd 7ms
nd 7ms
nd 1429ms
nd 452ms
nd 7ms
nd 1369ms
nd 206ms
nd 211ms
nd 250ms
nd 1344ms
nd 212ms
nd 1423ms
nd 1399ms
nd 1405ms
nd 1366ms
nd 8ms
nd 7ms
nd 143ms
nd 1082ms
connect ways 405639 vertexs 50
way 24ms
nd 4ms
nd 825ms
nd 4ms
nd 787ms
nd 4ms
nd 4ms
nd 4ms
nd 8ms
nd 26ms
nd 26ms
nd 4ms
nd 7ms
nd 3ms
nd 26ms
nd 26ms
nd 805ms

[Luca Garulli]

Hi Richard,

what times? What do you mean for inconsistent? Can you share the code you've used to import it? Use the OrientGraphNoTx and the massive insert intent. For all the tips loko at: 

https://github.com/nuvolabase/orientdb/wiki/Performance-Tuning-Blueprints

Lvc@

[Richard Kolb]

Hi Luca

I attached the source code to the mail yesterday. It's a Maven project and in a single class. Really dirty code, but I was just messing around.

What I mean by inconsistent is sometime a lookups range between 6ms and 3 seconds.

This graph.getVertices("Account.id", "23876JS2") may actually help, thanks.

regards

Richard.

[Luca Garulli]

Hi Richard,

I don't know if this problem is present yet. Have you created the index against the property "id" against "Account" class?

graph.createKeyIndex("id", Vertex.class, new Parameter("class", "Account"));

Pay attention to the additional parameter with the "class".

Lvc@

[Richard Kolb]

Hi Luca

Previously I created an index for my class called Node which worked fine :
graph.createKeyIndex("nodeid", Vertex.class, new Parameter("type", "UNIQUE"), new Parameter("class", "Node"));


Now I am running :

        final OClass node = odb.createVertexType("Node");
        node.createProperty("nodeid", OType.LONG);
        node.createProperty("lat", OType.DOUBLE);
        node.createProperty("lon", OType.DOUBLE);

        //node.createIndex("nodeididx", OClass.INDEX_TYPE.UNIQUE, "nodeid");
        //graph.createKeyIndex("nodeid", Vertex.class, new Parameter("type", "UNIQUE"), new Parameter("class", "Node"));

       
         graph.createKeyIndex("nodeid", Vertex.class, new Parameter("class", "Node"));

From this I got a heap space issue after inserting the 389137'th node and when using local storage with OGraphDatabase. I do commit each transaction. I have not seen this error when using the old createKeyIndex.

node 1967ms
node 389136
node 3885ms
node 389137
Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:2694)
    at java.lang.String.<init>(String.java:203)
    at java.lang.String.substring(String.java:1913)
    at java.io.UnixFileSystem.parentOrNull(UnixFileSystem.java:227)
    at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:160)
    at java.io.File.getCanonicalPath(File.java:589)
    at java.util.prefs.FileSystemPreferences.lockFile(FileSystemPreferences.java:902)
    at java.util.prefs.FileSystemPreferences.sync(FileSystemPreferences.java:729)
    at java.util.prefs.FileSystemPreferences.flush(FileSystemPreferences.java:824)
    at java.util.prefs.FileSystemPreferences.syncWorld(FileSystemPreferences.java:464)
    at java.util.prefs.FileSystemPreferences.access$1200(FileSystemPreferences.java:50)
    at java.util.prefs.FileSystemPreferences$3.run(FileSystemPreferences.java:432)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)
Error after the transaction has been committed. The transaction remains valid. The exception caught was on execution of %s.onAfterTxCommit()
com.orientechnologies.orient.core.exception.OTransactionBlockedException: Error after the transaction has been committed. The transaction remains valid. The exception caught was on execution of %s.onAfterTxCommit()
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
    at com.orientechnologies.common.log.OLogManager.debug(OLogManager.java:69)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.commit(ODatabaseRecordTx.java:140)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:457)
    at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.commit(OrientTransactionalGraph.java:54)
    at za.co.enerweb.testorient.App$1.node(App.java:171)
    at za.co.enerweb.testorient.App$1.startElement(App.java:69)

regards

Richard.

[Luca Garulli]

Hi Richard,

for massive insertion use the OrientGraphNoTx because avoid tx at all.

To avoid the tx problem try reducing some values in server.sh file:

java ... -Dmvrbtree.entryPoints=8 -Dmvrbtree.optimizeThreshold=1000

Lvc@