1.7rc2 performance results
392 views
Skip to first unread message
Andrey Yesyev
Feb 7, 2014, 4:50:05 PM2/7/14
to orient-...@googlegroups.com
NOTE: The original project I got from here https://groups.google.com/forum/#!topicsearchin/orient-database/perfomance%7Csort:date%7Cspell:true/orient-database/VF_j5rGeffA . Thanks to Milen Dyankov for it! I just modified it to multithread mode and new OrientDB API
Ok, finally I've got some performance results for OrientDB.
Here is my environment:
RAM 16Gb
CPU i7-3740QM 2.7Ghz
HDD 7200rpm
Windows 7 Prof 64bit
OrientDB v1.7rc2 SNAPSHOT
DB type plocal
Connection mode - remote
Single mode (Not distributed, just one node, started by server.bat)
Default configuration
In this test I was trying to figure out if I can get higher insertion rate in multithread mode
1 thread
|-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_ONLY threads 1 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 50000 | 0 | 3562 | 14037.06 || 50000 documents with 5 fields each | 50000 | 50000 | 0 | 2881 | 17355.09 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 100000 | 0 | 4056 | 24654.83 || 100000 documents with 5 fields each | 100000 | 100000 | 0 | 5576 | 17934.00 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 1000000 | 1000000 | 0 | 50031 | 19987.61 || 1000000 documents with 5 fields each | 1000000 | 1000000 | 0 | 66602 | 15014.56 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_AND_EDGES threads 1 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 25000 | 25000 | 12711 | 3933.60 || 50000 documents with 5 fields each | 50000 | 25000 | 25000 | 14535 | 3439.97 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 50000 | 50000 | 86934 | 1150.30 || 100000 documents with 5 fields each | 100000 | 50000 | 50000 | 92906 | 1076.36 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 623000 | 500000 | 123000 |(timeout) 600928 | 1036.73 || 1000000 documents with 5 fields each | 620900 | 500000 | 120900 |(timeout) 600668 | 1033.68 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting EDGES_ONLY threads 1 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 2 | 49998 | 71424 | 700.04 || 50000 documents with 5 fields each | 50000 | 2 | 49998 | 82808 | 603.81 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 2 | 99998 | 397524 | 251.56 || 100000 documents with 5 fields each | 100000 | 2 | 99998 | 394251 | 253.65 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 125500 | 0 | 125500 |(timeout) 600655 | 208.94 || 1000000 documents with 5 fields each | 123500 | 0 | 123500 |(timeout) 600398 | 205.70 ||-----------------------------------------------------------------------------------------------------------------------------------------|
|-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_ONLY threads 2 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 50000 | 0 | 3246 | 15403.57 || 50000 documents with 5 fields each | 50000 | 50000 | 0 | 2279 | 21939.45 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 100000 | 0 | 3465 | 28860.03 || 100000 documents with 5 fields each | 100000 | 100000 | 0 | 4444 | 22502.25 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 1000000 | 1000000 | 0 | 44121 | 22664.94 || 1000000 documents with 5 fields each | 1000000 | 1000000 | 0 | 62515 | 15996.16 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_AND_EDGES threads 2 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 25000 | 25000 | 6216 | 8043.76 || 50000 documents with 5 fields each | 50000 | 25000 | 25000 | 5698 | 8775.01 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 50000 | 50000 | 21480 | 4655.49 || 100000 documents with 5 fields each | 100000 | 50000 | 50000 | 25726 | 3887.12 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 480700 | 250000 | 230700 |(timeout) 600890 | 799.98 || 1000000 documents with 5 fields each | 472000 | 250000 | 222000 |(timeout) 601410 | 784.82 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting EDGES_ONLY threads 2 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50002 | 4 | 49998 | 19705 | 2537.53 || 50000 documents with 5 fields each | 50002 | 4 | 49998 | 23434 | 2133.74 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100002 | 4 | 99998 | 105019 | 952.23 || 100000 documents with 5 fields each | 100002 | 4 | 99998 | 109623 | 912.24 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 211300 | 0 | 211300 |(timeout) 600411 | 351.93 || 1000000 documents with 5 fields each | 216900 | 0 | 216900 |(timeout) 600936 | 360.94 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|4 threads
|-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_ONLY threads 4 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 50000 | 0 | 2413 | 20721.09 || 50000 documents with 5 fields each | 50000 | 50000 | 0 | 2171 | 23030.86 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 100000 | 0 | 3600 | 27777.78 || 100000 documents with 5 fields each | 100000 | 100000 | 0 | 4124 | 24248.30 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 1000000 | 1000000 | 0 | 37901 | 26384.53 || 1000000 documents with 5 fields each | 1000000 | 1000000 | 0 | 47457 | 21071.71 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_AND_EDGES threads 4 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 25000 | 25000 | 3483 | 14355.44 || 50000 documents with 5 fields each | 50000 | 25000 | 25000 | 3623 | 13800.72 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 50000 | 50000 | 9192 | 10879.03 || 100000 documents with 5 fields each | 100000 | 50000 | 50000 | 10067 | 9933.45 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 525500 | 125000 | 400500 |(timeout) 601017 | 874.35 || 1000000 documents with 5 fields each | 522800 | 125000 | 397800 |(timeout) 601126 | 869.70 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting EDGES_ONLY threads 4 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50004 | 8 | 49996 | 7673 | 6516.88 || 50000 documents with 5 fields each | 50004 | 8 | 49996 | 8358 | 5982.77 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100004 | 8 | 99996 | 30451 | 3284.10 || 100000 documents with 5 fields each | 100004 | 8 | 99996 | 33923 | 2947.97 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 395300 | 0 | 395300 |(timeout) 600839 | 657.91 || 1000000 documents with 5 fields each | 391500 | 0 | 391500 |(timeout) 600702 | 651.74 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|Each test has 3 modes
VERTICES_ONLY - only vertices have been inserted
VERTICES_AND_EDGES - vetices and edges have been inserted. You can see the number of them in corresponding column
EDGES_ONLY
If in time column you see (timeout) - it means test haven't been done within 10mins
All edges have been added to 2 vertices. I mean all edges in any particular test connect only 2 vertices (It was easier to implement)
2 fields each - means each document (vertex or edge) has 2 properties.
Well, results are not very satisfying. Running in multithread mode doesn't tremendously improve the insertion rate.
I would appreciate any explanation here!
If edges are involved, the overall rate drops noticeably. For example neither tests with 1000000 documents including edges didn't finished before 10mins timeout.
As a matter of fact 27K/sec, the best test result is also not impressive.
Should I distribute edges among more then 2 vertices?
Anyhow, I'm absolutely sure there are some tricks (not very technical term, I know) that can improve these numbers.
I would be more than happy to hear about them.
Regards,
Andrey
PS. I'll provide the project as soon as I post it on github.
Andrey Yesyev
Feb 7, 2014, 7:55:50 PM2/7/14
to orient-...@googlegroups.com
As I promised, here is the project https://github.com/ayesyev/graphdb-tests
Your comments, critic, test results are highly appreciated.
Andrey Yesyev
Feb 7, 2014, 10:36:24 PM2/7/14
to orient-...@googlegroups.com
Here is some more results.
actually slow down insertion rate.... What's the purpose of it then?
The main change there, I got rid of properties on edges, because I don't really need that in my project.
Also, I figured out that
db.getRawGraph().declareIntent(new OIntentMassiveInsert());
|-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_AND_EDGES threads 4 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 1000 documents with 2 fields each | 1000 | 500 | 500 | 37 | 27027.03 || 1000 documents with 5 fields each | 1000 | 500 | 500 | 71 | 14084.51 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 5000 documents with 2 fields each | 5000 | 2500 | 2500 | 86 | 58139.53 || 5000 documents with 5 fields each | 5000 | 2500 | 2500 | 99 | 50505.05 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 10000 documents with 2 fields each | 10000 | 5000 | 5000 | 149 | 67114.09 || 10000 documents with 5 fields each | 10000 | 5000 | 5000 | 180 | 55555.56 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 50000 documents with 2 fields each | 50000 | 25000 | 25000 | 813 | 61500.62 || 50000 documents with 5 fields each | 50000 | 25000 | 25000 | 806 | 62034.74 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 10000 documents with 2 fields each | 10000 | 5000 | 5000 | 154 | 64935.06 || 10000 documents with 5 fields each | 10000 | 5000 | 5000 | 180 | 55555.56 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 50000 documents with 2 fields each | 50000 | 25000 | 25000 | 746 | 67024.13 || 50000 documents with 5 fields each | 50000 | 25000 | 25000 | 819 | 61050.06 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 50000 | 50000 | 1440 | 69444.44 || 100000 documents with 5 fields each | 100000 | 50000 | 50000 | 1780 | 56179.78 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 500000 documents with 2 fields each | 500000 | 250000 | 250000 | 7576 | 65997.89 || 500000 documents with 5 fields each | 500000 | 250000 | 250000 | 10560 | 47348.48 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 1000000 | 500000 | 500000 | 16061 | 62262.62 || 1000000 documents with 5 fields each | 1000000 | 500000 | 500000 | 21154 | 47272.38 ||-----------------------------------------------------------------------------------------------------------------------------------------|As you can see, the performance is way better.
This is the results of test where vertices don't have properties and edges have 2 or 5 properties.
|-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_AND_EDGES threads 1 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 1000 documents with 2 fields each | 1000 | 500 | 500 | 287 | 3484.32 || 1000 documents with 5 fields each | 1000 | 500 | 500 | 215 | 4651.16 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 5000 documents with 2 fields each | 5000 | 2500 | 2500 | 723 | 6915.63 || 5000 documents with 5 fields each | 5000 | 2500 | 2500 | 596 | 8389.26 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 10000 documents with 2 fields each | 10000 | 5000 | 5000 | 977 | 10235.41 || 10000 documents with 5 fields each | 10000 | 5000 | 5000 | 1041 | 9606.15 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 50000 documents with 2 fields each | 50000 | 25000 | 25000 | 12470 | 4009.62 || 50000 documents with 5 fields each | 50000 | 25000 | 25000 | 11266 | 4438.13 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 50000 | 50000 | 78769 | 1269.53 || 100000 documents with 5 fields each | 100000 | 50000 | 50000 | 68145 | 1467.46 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 500000 documents with 2 fields each | 374500 | 250000 | 124500 | (timeout) 617106 | 606.86 || 500000 documents with 5 fields each | 370100 | 250000 | 120100 | (timeout) 620044 | 596.89 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 614100 | 500000 | 114100 | (timeout) 626038 | 980.93 |The result is way worse.
Starting with 500000 docs, test stopped by timeout, so I didn't wait for the whole test suit to complete.
I think we found the bottle neck of OrientDB.
Avoid properties in edges!
Would be great to hear why edges properties are so slow!
Andrey Lomakin
Feb 8, 2014, 1:46:02 AM2/8/14
to orient-database
Hi Andrey,
Several notes.
1. According to your previous answers I suppose that you did not set amount of memory which is consumed by disk cache (the more the better of course) , default value is 4Gb so you do not use 75% of your memory during this benchmark, amount of memory for disk cache has the same importance as -Xmx for Java. To set this parameter the simplest way is to provide system property in server.sh so if you have 16 Gb of RAM then I think you can freely use 14GB on Windows environment without any problems. So to set this parameter set system property in server.sh -Dstorage.diskCache.bufferSize=14336 . After start open console and issue command "config get storage.diskCache.bufferSize" (after connect as root of course) to check that this parameter was set correctly.
2. Your note about bottleneck of insertion of edges with properties is not completely true. I looked into your code.
You do following:
OrientEdge e = db.addEdge(null, v1, v2, "E");
for (int i = 0; i < numberOfProperties; i++)
e.setProperty("property" + i, "value" + i);
So you add edge at first and then properties.
In such case following steps are performed:
a. Lightweight edge is created. It means direct link between 2 vertexes.
b. Lightweight edge is removed (which means that links on each of 2 vertexes is removed), document to hold properties is created, new links are added.
So we have to perform 3 times more operations on collection of links than we do without properties on edges.
To avoid this either do not use lightweight edges (typically users do not have properties on edges so it is good choice to use lightweight edges), or modify your code as following:
Map<String, String> properties = new HashMap<String, String>();
for (int i = 0; i < numberOfProperties; i++)
properties.put("property" + i, "value" + i);
OrientEdge e = db.addEdge(null, v1, v2, "E", properties);
In such case document for edge will be created at the same time as you create edge so we will add links to edge document at once, and will not do add/remove/add operations instead.
3. You use following code
OrientVertex vertex = db.addVertex("OGraphVertex",null);
But you did not create schema which means that new cluster is created during benchmark and full db flush and fsync is performed which provides good system slowdown:
So could you either change code to the following one:
a. OrientVertex vertex = db.addVertex(null);
b. or create "OGraphVertex" class before benchmark.
And at last about your note that multi threaded test is not much faster, sorry to agree with this but plocal storage is very new, give us several months and we will provide multicore support https://github.com/orientechnologies/orientdb/issues/1678 .
BTW may I ask you to run the same tests not on remote but on plocal storage and publish difference ?
Could you change your benchmark according suggestions above and rerun test ?
--
---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Best regards,
Andrey Lomakin.
Orient Technologies
the Company behind OrientDB
Andrey Lomakin.
Orient Technologies
the Company behind OrientDB
Andrey Lomakin
Feb 8, 2014, 2:33:32 AM2/8/14
to orient-database
Andrey,
If you will have time to do insertion benchmark on plocal embedded storage could you also set following parameters
OGlobalConfiguration.CACHE_LEVEL2_ENABLED.setValue(false);
OGlobalConfiguration.CACHE_LEVEL2_SIZE.setValue(0);
and do not forget to set
OGlobalConfiguration.DISK_CACHE_SIZE(amout of ram in megabytes to consume by disk cache)
it should be done before you open database in embedded mode.
If you will have time to do insertion benchmark on plocal embedded storage could you also set following parameters
OGlobalConfiguration.CACHE_LEVEL2_ENABLED.setValue(false);
OGlobalConfiguration.CACHE_LEVEL2_SIZE.setValue(0);
and do not forget to set
OGlobalConfiguration.DISK_CACHE_SIZE(amout of ram in megabytes to consume by disk cache)
it should be done before you open database in embedded mode.
Andrey Yesyev
Feb 8, 2014, 9:31:38 AM2/8/14
to orient-...@googlegroups.com
Great hints!
Thanks! I'll gladly add this to my tests and post results here ASAP.
Andrey Yesyev
Feb 8, 2014, 10:01:55 AM2/8/14
to orient-...@googlegroups.com
The first problem is
I don't have method db.addEdge(null, v1, v2, "E", properties);
What version of blueprints do I have to use?
For now I have blueprints-orient-graph-2.5.0-SNAPSHOT.jar
Andrey Yesyev
Feb 8, 2014, 11:40:09 AM2/8/14
to orient-...@googlegroups.com
The second problem is with this parameter storage.diskCache.bufferSize.
I run tests on my Windows laptop.
So I added to server.bat this parameter as
set ORIENTDB_SETTINGS=-Dprofiler.enabled=true -Dcache.level1.enabled=false -Dcache.level2.enabled=false -Dcache.level2.size=0 -Dstorage.diskCache.bufferSize=14336
then I start the server using this script.
Connect to the server using console.bat
Run the command config get storage.diskCache
The result is 4096
I also added this parameter in System Variables. Still can't see it set.
Andrey Yesyev
Feb 8, 2014, 11:47:33 AM2/8/14
to orient-...@googlegroups.com
I figured out that I can set in console this parameter
config set storage.diskCache.bufferSize 14336
but after I reopen console, it sets to 4096 again.
Luca Garulli
Feb 8, 2014, 1:23:16 PM2/8/14
to orient-database
Hi Andrey,
remove the blueprints-orient-graph-2.5.0-SNAPSHOT.jar file in your classpath: OrientDB's Blueprints implementation has been merged with OrientDB distribution now.
About the addEdge use this instead:
OrientVertex.addEdge(String label, final OrientVertex inVertex, final String iClassName, final String iClusterName, final Object... fields)
Example:
v1.addEdge( null, v2, "E", null, properties);
About usage of console to set parameters, first you've to connect against remote server with:
orientdb> connect remote:localhost root myrootpassword
Lvc@
--
Andrey Lomakin
Feb 8, 2014, 2:59:35 PM2/8/14
to orient-database
Hi Andrey ,
If you used blueprints-oriendb that means that you used version which was dropped in 1.7 and all tests have to be reran to get numbers related to new implementation .
You did not use new data structures .
08 февр. 2014 г. 20:23 пользователь "Luca Garulli" <l.ga...@gmail.com> написал:
Andrey Lomakin
Feb 8, 2014, 3:15:33 PM2/8/14
to orient-database
About the system settings it is strange . I tried on fedora without any problems will look at Monday .You should set this setting before starting the server .
08 февр. 2014 г. 21:59 пользователь "Andrey Lomakin" <lomakin...@gmail.com> написал:
Andrey Yesyev
Feb 8, 2014, 3:55:21 PM2/8/14
to orient-...@googlegroups.com
Luca&Andrey,
Thanks! Everything looks good now.
I'll post results in a bit.
Andrey Yesyev
Feb 9, 2014, 9:59:26 AM2/9/14
to orient-...@googlegroups.com
Ok, here we go!
I added all Andrey's tips to the project.
storage.diskCache.bufferSize set to 14336
All edges have appropriate number of properties and added this way
protected OrientEdge createEdge(Vertex v1, Vertex v2) { Map<String, String> properties = new HashMap<String, String>(); for (int i = 0; i < numberOfProperties; i++) properties.put("property" + i, "value" + i); OrientEdge e = ((OrientVertex)v1).addEdge(null, (OrientVertex)v2, "E", null, properties); e.save();
return e;}Results are attached for remote and embedded (both using plocal storage type).
On Monday I'll try to make my conclusions.
All changes are committed to github project.
Andrey Lomakin
Feb 9, 2014, 10:44:49 AM2/9/14
to orient-database
Hi Andrey,
Could you provide instructions how to run these tests to see statistic results ?
--
---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Andrey Lomakin
Feb 9, 2014, 10:47:05 AM2/9/14
to orient-database
Got it ! ))
Andrey Lomakin
Feb 9, 2014, 10:50:34 AM2/9/14
to orient-database
Andrey,
I do not see any commits in project. https://github.com/ayesyev/graphdb-tests/commits/master
Did you push them ?
Andrey Yesyev
Feb 9, 2014, 10:56:10 AM2/9/14
to orient-...@googlegroups.com
I guess I just did it :)
Sorry for that... didn't use to github.
Andrey Yesyev
Feb 9, 2014, 10:59:04 AM2/9/14
to orient-...@googlegroups.com
Please post your results!
Again, any comments regarding source code are very welcome!
Andrey Lomakin
Feb 9, 2014, 2:27:19 PM2/9/14
to orient-database
Hi Andrey,
I started benchmark on my side and while it is running I investigated it.
I think that I should note that this benchmark does not reproduce real test cases (dunno what performance data you get on other DBs).
I mean what this benchmark does.
Lets suppose that we have to insert 1 000 000 documents vertexes and edges.
Then it creates 500 000 vertexes and then takes 2 of them, and creates 500 000 edges between them.
And everything in one transaction.
So we have graph database with 499 998 unconnected vertexes and 2 vertexes which have 500 000 edges and everything is committed in single transaction.
Did I miss something ?
I mean that I think you do not suppose users to commit such data structure and commit it using single transaction.
Usually data structures are way different and changes are committed in following way users load data, change them, commit them.
It is my personal opinion but may be you will be interested in performance test which loads real wikipedia data by loading and committing them by small batches ?
Also this tests uses index which is very typical for db usage.
We used such test case so I can change and publish it as maven project and because it is tinkerpop based you can test all dbs which you are interested in.
Our load test does not have properties on vertexes only relations and index by page key,but it is simple to add additional properties.
What do you think ?
Dexter Pratt
Feb 9, 2014, 2:49:48 PM2/9/14
to orient-...@googlegroups.com
Hi Andrey,
I'd definitely be interested in seeing the wikipedia example of loading a large structure in small batch commits. Especially one that used a tinker pop frames data model, like our application.
Thanks
Dexter Pratt
Director, NDEx project
Ideker Lab UCSD / Cytoscape Consortium
Message has been deleted
Andrey Yesyev
Feb 9, 2014, 2:56:03 PM2/9/14
to orient-...@googlegroups.com
Hi Andrey,
I'm totally agree with you, it's not a benchmark, moreover it's a real test case either.
What I'm trying to do is
1. Get skills to work with particular DB. With Orient in this case.
2. To compare it with others, doing the same things with all of them.
I forgot to mention, but I was really looking for an optimal number of documents to commit (definitely 1000000 docs are way to many). Depending on my results it's about 25K docs per commit.
Also, 500000 edges between 2 vertices doesn't look like real world, but I warned about it when posted my results. It's just the easiest way to implement it :)
My next goal to try it with indices and in a distributed mode (as soon as you'll respond my posts regarding issues with distributed mode).
My next goal to try it with indices and in a distributed mode (as soon as you'll respond my posts regarding issues with distributed mode).
I would love to see your performance test project. However, I don't really understand the concept of vertices without properties? Could you please explain how you store data there?
Quick overlook of our use-case.
We need a DB to handle 40K/sec insertion rate (Vertices and Edges). We need 2 properties for a Vertex and no properties for Edges, we can use different Classes for them.
This insertion rate should be achievable in a distributed mode. We might have up to 40 nodes in cluster (I mean h/w cluster).
Milen Dyankov
Feb 9, 2014, 3:23:45 PM2/9/14
to orient-...@googlegroups.com
Hello Andrey Lomakin,
as I write the original tests that Andrey Yesyev is basing his on, I thought I need to step in with a word of explanation.
Let me start by saying your findings are correct, the test indeed inserts given amount of vertices and the a given amount of edges between the first two vertices. Generally speaking you are also right saying "this benchmark does not reproduce real test cases". However it was never meant to be a general purpose benchmark (please have a look at the disclaimer of my original post https://groups.google.com/forum/#!topicsearchin/orient-database/perfomance%7Csort:date%7Cspell:true/orient-database/VF_j5rGeffA).
The purpose of this test was to illustrate the fact that I found OrientDB to be very slow on inserting edges. In fact getting slower and slower as the amount of edges increases. I also compared it to Neo4j just because I wasn't sure whether this is something OrientDB specific or it's due to the nature of the graph databases in general.
As far as transactions are concern, my original code did not use transactions at all (at least not explicitly). According tho the docs (back then) the was supposed execute each operation instantly. I don't know (didn't have the time to examine Andrey's code) why he introduced transactions and while I agree inserting millions of documents in a single transaction is not a good idea, I just wanted to point out the original test was demonstrating the problem with no transactions at all. I'm pretty sure Andrey can easily change the code to commit data in smaller chunks but honestly speaking I don't expect huge improvements (comparing to the no transaction).
As far as the structure of the data is concerned, I fail to see how can that cause performance degradation. Are you saying that if the test was to create edges between every 2 vertices for example (instead of just first 2) it would be faster? I highly doubt it. In fact I think the way the test is written should actually allow OrientDB to perform better than average as it can utilize cache and doesn't have to look for edges.
Finally, I have to admin I gave up on OrientDB half a year ago (don't get me wrong, nothing personal, I just found it not to be mature enough for the project I was working on) and while I'm still trying to keep an eye on this list, I'm not fully aware of all the optimizations that have happened since then. It may me the case that the test is no longer valid for the current version or needs to be rewritten completely. If I find some spare time I will try to update my original tests to use the latest version and post some results here.
Regards,
Milen
Andrey Lomakin
Feb 10, 2014, 5:54:54 AM2/10/14
to orient-database
HI all,
Thank you all for answers.
The main mine concern here is that for benchmarks we should use cases which are close to real.
About edges distribution, we use cache to optimize loops in graph, I mean if vertex is created, and then loaded to create edge there is good probability that it will be in cache.
Any way I gathered links to benchmarks which we used or are going to use.
Here is load test of Wikipedia data https://github.com/laa/orientdb-wikipedia-benchmark
and there is very interesting benchmark here https://github.com/Morro/GraphDBBenchmark
So if you publish your data using them I will very appreciate it.
Giraldo Rosales
Feb 13, 2014, 3:19:43 PM2/13/14
to orient-...@googlegroups.com
Would be great if someone would benchmark OrientDB, MySQL (with Joins), and MySQL/Neo4J. To get some speed tests. Notice there were some out there with older versions of OrientDB (1.3).
Andrey Yesyev
Feb 13, 2014, 3:23:57 PM2/13/14
to orient-...@googlegroups.com
Do you mean compare them?
If so, what exactly? Insertion rate? Query?
In my opinion, it's not technically correct to compare relational and graph DBs.
Giraldo Rosales
Feb 13, 2014, 3:36:30 PM2/13/14
to orient-...@googlegroups.com
Yes, a comparison. Knowing some biz refuse to leave relational in cases where a graph should exist. I do know Neo4J is a graph but they recommend doing a double query, the first in neo4j to get the relationship and return the ids, and a second with mysql to get the data. OrientDB would have both combined.
Tests would be large use case scenarios. Complex selects, and high inserts/updates. For example, 1 million users linked to an object and traversing down a selected group. The same data between each database. Which is fastest? Would also help developers in OrientDB see where performance may be needed, if any.
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/QscZaIK5JPU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.
Andrey Yesyev
Feb 13, 2014, 3:40:00 PM2/13/14
to orient-...@googlegroups.com
I'm about to implement that test for my use-case. I'll post my results as soon as possible. I'm not sure they'll be interesting to many people though.
Giraldo Rosales
Feb 13, 2014, 3:40:44 PM2/13/14
to orient-...@googlegroups.com
That would be great! Thanks!!
pieter
Feb 14, 2014, 1:55:04 AM2/14/14
to orient-...@googlegroups.com
I am interested!
Thanks
Pieter
Thanks
Pieter
Xiaobing LIU
Nov 30, 2014, 9:16:15 PM11/30/14
to orient-...@googlegroups.com
I see this is a very old post, but it cann't work when I get it from the github. Is this the version problem?
On Saturday, February 8, 2014 5:50:05 AM UTC+8, Andrey Yesyev wrote:
On Saturday, February 8, 2014 5:50:05 AM UTC+8, Andrey Yesyev wrote:
NOTE: The original project I got from here https://groups.google.com/forum/#!topicsearchin/orient-database/perfomance%7Csort:date%7Cspell:true/orient-database/VF_j5rGeffA . Thanks to Milen Dyankov for it! I just modified it to multithread mode and new OrientDB API
Ok, finally I've got some performance results for OrientDB.Here is my environment:
RAM 16GbCPU i7-3740QM 2.7GhzHDD 7200rpm
Windows 7 Prof 64bitOrientDB v1.7rc2 SNAPSHOTDB type plocalConnection mode - remoteSingle mode (Not distributed, just one node, started by server.bat)Default configuration
In this test I was trying to figure out if I can get higher insertion rate in multithread mode
1 thread
|-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_ONLY threads 1 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 50000 | 0 | 3562 | 14037.06 || 50000 documents with 5 fields each | 50000 | 50000 | 0 | 2881 | 17355.09 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 100000 | 0 | 4056 | 24654.83 || 100000 documents with 5 fields each | 100000 | 100000 | 0 | 5576 | 17934.00 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 1000000 | 1000000 | 0 | 50031 | 19987.61 || 1000000 documents with 5 fields each | 1000000 | 1000000 | 0 | 66602 | 15014.56 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_AND_EDGES threads 1 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 25000 | 25000 | 12711 | 3933.60 || 50000 documents with 5 fields each | 50000 | 25000 | 25000 | 14535 | 3439.97 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 50000 | 50000 | 86934 | 1150.30 || 100000 documents with 5 fields each | 100000 | 50000 | 50000 | 92906 | 1076.36 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 623000 | 500000 | 123000 |(timeout) 600928 | 1036.73 || 1000000 documents with 5 fields each | 620900 | 500000 | 120900 |(timeout) 600668 | 1033.68 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting EDGES_ONLY threads 1 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 2 | 49998 | 71424 | 700.04 || 50000 documents with 5 fields each | 50000 | 2 | 49998 | 82808 | 603.81 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 2 | 99998 | 397524 | 251.56 || 100000 documents with 5 fields each | 100000 | 2 | 99998 | 394251 | 253.65 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 125500 | 0 | 125500 |(timeout) 600655 | 208.94 || 1000000 documents with 5 fields each | 123500 | 0 | 123500 |(timeout) 600398 | 205.70 ||-----------------------------------------------------------------------------------------------------------------------------------------|
2 threads
|-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_ONLY threads 2 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 50000 | 0 | 3246 | 15403.57 || 50000 documents with 5 fields each | 50000 | 50000 | 0 | 2279 | 21939.45 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 100000 | 0 | 3465 | 28860.03 || 100000 documents with 5 fields each | 100000 | 100000 | 0 | 4444 | 22502.25 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 1000000 | 1000000 | 0 | 44121 | 22664.94 || 1000000 documents with 5 fields each | 1000000 | 1000000 | 0 | 62515 | 15996.16 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_AND_EDGES threads 2 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 25000 | 25000 | 6216 | 8043.76 || 50000 documents with 5 fields each | 50000 | 25000 | 25000 | 5698 | 8775.01 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 50000 | 50000 | 21480 | 4655.49 || 100000 documents with 5 fields each | 100000 | 50000 | 50000 | 25726 | 3887.12 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 480700 | 250000 | 230700 |(timeout) 600890 | 799.98 || 1000000 documents with 5 fields each | 472000 | 250000 | 222000 |(timeout) 601410 | 784.82 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting EDGES_ONLY threads 2 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50002 | 4 | 49998 | 19705 | 2537.53 || 50000 documents with 5 fields each | 50002 | 4 | 49998 | 23434 | 2133.74 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100002 | 4 | 99998 | 105019 | 952.23 || 100000 documents with 5 fields each | 100002 | 4 | 99998 | 109623 | 912.24 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 211300 | 0 | 211300 |(timeout) 600411 | 351.93 || 1000000 documents with 5 fields each | 216900 | 0 | 216900 |(timeout) 600936 | 360.94 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|
4 threads
|-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_ONLY threads 4 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 50000 | 0 | 2413 | 20721.09 || 50000 documents with 5 fields each | 50000 | 50000 | 0 | 2171 | 23030.86 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 100000 | 0 | 3600 | 27777.78 || 100000 documents with 5 fields each | 100000 | 100000 | 0 | 4124 | 24248.30 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 1000000 | 1000000 | 0 | 37901 | 26384.53 || 1000000 documents with 5 fields each | 1000000 | 1000000 | 0 | 47457 | 21071.71 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting VERTICES_AND_EDGES threads 4 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50000 | 25000 | 25000 | 3483 | 14355.44 || 50000 documents with 5 fields each | 50000 | 25000 | 25000 | 3623 | 13800.72 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100000 | 50000 | 50000 | 9192 | 10879.03 || 100000 documents with 5 fields each | 100000 | 50000 | 50000 | 10067 | 9933.45 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 525500 | 125000 | 400500 |(timeout) 601017 | 874.35 || 1000000 documents with 5 fields each | 522800 | 125000 | 397800 |(timeout) 601126 | 869.70 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|| Test inserting EDGES_ONLY threads 4 ||-----------------------------------------------------------------------------------------------------------------------------------------|| test | documents | vertices | edges | time (ms) | avg (doc/sec) || ---------------------------------------- | --------------- | --------------- | --------------- | -------------------- | --------------- || 50000 documents with 2 fields each | 50004 | 8 | 49996 | 7673 | 6516.88 || 50000 documents with 5 fields each | 50004 | 8 | 49996 | 8358 | 5982.77 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 100000 documents with 2 fields each | 100004 | 8 | 99996 | 30451 | 3284.10 || 100000 documents with 5 fields each | 100004 | 8 | 99996 | 33923 | 2947.97 ||-----------------------------------------------------------------------------------------------------------------------------------------|| 1000000 documents with 2 fields each | 395300 | 0 | 395300 |(timeout) 600839 | 657.91 || 1000000 documents with 5 fields each | 391500 | 0 | 391500 |(timeout) 600702 | 651.74 ||-----------------------------------------------------------------------------------------------------------------------------------------||-----------------------------------------------------------------------------------------------------------------------------------------|
Some explanations to these resultsEach test has 3 modes VERTICES_ONLY - only ve...Reply all
Reply to author
Forward
0 new messages