Too slow query speed

[ladic]

Hi,

I have simple query without WHERE: select * from EXPERIMENT limit = 125000, the speed of query is between 270 - 315 sec...Why this simple query is so slow?
Query select * from EXPERIMENT limit = 50000 takes about 8 sec only.

Thanks,
L. Janák

[Luca Garulli]

Hi,

this depends by your memory settings. Probably more than 50k records the JVM starts swapping? Are you using local or remote connection?

Lvc@

--
 
---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ladislav Janák]

Hi,

yes, you are right, If i try to run this query direct from Java program it takes about 300 ms, from OrientDB console it prints out of memory error : Java heap space.

I am working on my Diploma thesis - OrientDB query speed comparison with ORACLE (the same database and model)
OrientDB database - 3.3 mil. vertices and 9.8 mil edges

this query I tried to run on this machine: Win 7 32bit, (4GB RAM) because of 32 bit - only 3GB, Core 2 Duo 2,4 GHz (is there any chance to avoid java heap error?)


then I will test query speed on the second machine: Win 7 64 bit, 8 GB RAM, Core i5 2,5 GHz

my server settings for the first machine is:

set ORIENTDB_SETTINGS=-Dprofiler.enabled=true -Dindex.auto.rebuildAfterNotSoftClose=false -Dfile.mmap.useOldManager=true -Dfile.mmap.maxMemory=2gb -Dcache.level1.enabled=true -Dcache.level2.enabled=true -Dcache.level2.strategy=1

set JAVA_OPTS_SCRIPT=-XX:+HeapDumpOnOutOfMemoryError -XX:+AggressiveOpts -XX:CompileThreshold=200 -Xmx512m

I would really appreciate if you could give me some advice how to set these server settings for both machines (having regard to their parameters)

Thanks in advance,
L. Janák
 

2013/3/28 Luca Garulli <l.ga...@gmail.com>

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/w82MzPOmiCk/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

[Ladislav Janák]

Regards the protocol and Java heap error - it does not matter which protocol I am using (local or remote)...

Thanks
L. Janák

2013/3/28 Ladislav Janák <ladis...@gmail.com>

[Luca Garulli]

Hi,
use asynchronous queries that don't consumes heap. Example:

database.command(
  new OSQLAsynchQuery<ODocument>("select * from animal where column(0) < 5 or column(0) >= 3 and column(5) < 7",
      new OCommandResultListener() {
        @Override
        public boolean result(Object iRecord) {
          System.out.println(iRecord);
          return true;
        }

        @Override
        public void end() {
        }
     })).execute());

In this case the result() method is called per record found avoiding to collecting them in the heap to return right after.

Lvc@

[Ladislav Janák]

Hi,

I tried your sollution, but server still prints this error:  Map failed...Memory mapping error, try to reduce max memory to .... and retry.

and next error is without trying to reduce max memory.

How schould I reduce Memory mapping on server? Is it possible work with these queries (select * from artefact limit=125000) on PC with 3GB RAM?

Thanks again,

[Luca Garulli]

Hi Ladislav,

you can extract millions of records with few MB of free RAM. Can you post the full error please?

Lvc@

[Ladislav Janák]

Records are extracted but it still takes about 200 sec, the same query in ORACLE in SQL developer takes 15,8 sec. Please find the picture with error in attachment.

Thanks for your help,
L. Janak

2013/3/28 Luca Garulli <l.ga...@gmail.com>

[Ladislav Janák]

My code is:

long startTime = System.currentTimeMillis();
        graph.getRawGraph().command(
                new OSQLAsynchQuery<ODocument>("select * from artefact limit=125000",

                new OCommandResultListener() {

                    @Override
                    public boolean result(Object iRecord) {

                        return true;
                    }
                })).execute();
  double estimatedTime = (System.currentTimeMillis() - startTime) / 1000.0;
  System.out.println(estimatedTime + "sec");

L. Janak

2013/3/28 Ladislav Janák <ladis...@gmail.com>

[Luca Garulli]

By looking at the stack you're using the old "OMMapManagerOld" class.

Have you set file.mmap.useOldManager=true ?

Lvc@

[Ladislav Janák]

Yes, I am using it on the server I edited this setting directly in Server.bat file:

set ORIENTDB_SETTINGS=
-Dprofiler.enabled=true

-Dindex.auto.rebuildAfterNotSoftClose=false
-Dfile.mmap.useOldManager=true
-Dfile.mmap.maxMemory=2gb
-Dcache.level1.enabled=true
-Dcache.level2.enabled=true
-Dcache.level2.strategy=1

and Java setting:
set JAVA_OPTS_SCRIPT=

-XX:+HeapDumpOnOutOfMemoryError
-XX:+AggressiveOpts
-XX:CompileThreshold=200
-Xmx512m

[Luca Garulli]

Ok, now try removing it. Then against a remote server performance are much worst than working with local.

Lvc@

[ladic]

Hi,

I wanted try to run query with local protocol from Netbeans IDE but I get this error:

28.3.2013 18:56:10 com.orientechnologies.common.log.OLogManager log
WARNING: segment file 'default.0.oda' was not closed correctly last time
28.3.2013 18:56:10 com.orientechnologies.common.log.OLogManager log
WARNING: segment file 'default.odh' was not closed correctly last time
28.3.2013 18:56:10 com.orientechnologies.common.log.OLogManager log
WARNING: [OVariableParser.resolveVariables] Error on resolving property: ORIENTDB_HOME
Exception in thread "main" com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'D:/Diplomka/database_OrientDB/server/orientdb-graphed-1.3.0/databases/EEG_ERP' with mode=rw
    at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.open(OStorageLocal.java:212)
    at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.open(ODatabaseRaw.java:96)
    at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:47)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.open(ODatabaseRecordAbstract.java:114)
    at com.orientechnologies.orient.core.db.ODatabaseWrapperAbstract.open(ODatabaseWrapperAbstract.java:47)
    at com.orientechnologies.orient.core.db.graph.OGraphDatabase.open(OGraphDatabase.java:90)
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.openOrCreate(OrientBaseGraph.java:389)
    at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.<init>(OrientBaseGraph.java:70)
    at com.tinkerpop.blueprints.impls.orient.OrientGraph.<init>(OrientGraph.java:64)
    at com.mycompany.orientdb_query_speed_test.QuerySpeed.main(QuerySpeed.java:23)
Caused by: java.io.FileNotFoundException: File: C:\Users\Ladislav Janák\Documents\NetBeansProjects\OrientDB_Query_speed_test\null\databases\EEG_ERP\default.0.oda
    at com.orientechnologies.orient.core.storage.fs.OAbstractFile.open(OAbstractFile.java:122)
    at com.orientechnologies.orient.core.storage.impl.local.OMultiFileSegment.open(OMultiFileSegment.java:89)
    at com.orientechnologies.orient.core.storage.impl.local.ODataLocal.open(ODataLocal.java:104)
    at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.open(OStorageLocal.java:165)
    ... 9 more

How can I set ORIENTDB_HOME variable? I am lost now...

Thanks,
L. Janák

Dne čtvrtek, 28. března 2013 17:22:25 UTC+1 Lvc@ napsal(a):

[Luca Garulli]

Hi,

like all the environment variables you can set it at OS level or via command line:

java ... -DORIENTDB_HOME=/Users/you/orientdb

or again via Java:

System.setProperty("ORIENTDB_HOME", "/Users/you/orientdb");

Lvc@

[Shivanandan Gupta]

Hi All,

I am a newbie to orientdb and trying with Async query, can someone post a sample java code end to end which I can refer for asynchronous query.

Thanks in advance.

-- Shivanandan Gupta

[Shivanandan Gupta]

Hi All,

I have written thhe Async select program please see the attached .java file , but when I run the program I am getting below given error , the program is not able to open the database, I can use the Oriendb browser and can access the db from there but from program I am getting below given error.

If anyone can help me out it will be great , I am almost there , I am a newbie in java and orientdb both :)  , so really will appreciate any help here.

Exception in thread "main" com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'localhost:2480/databases/eigen' with mode=rw

        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:214)

        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:243)

        at Async_query.main(Async_query.java:18)

Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot open the storage 'eigen' because it does not exist in path: localhost:2480/databases/eigen

        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:159)

        ... 2 more

Thank you,

--Shivanandan Gupta