How to improve binary data upload

21 views
Skip to first unread message

Kor

unread,
Jan 29, 2015, 3:33:15 AM1/29/15
to orient-...@googlegroups.com
Hello everybody,

i've read documentation about binary data, everything works fine except commit duration.
It never takes the same amount of time, but it's often too long.

Here is the code.
        OrientGraphFactory graphFactory = new OrientGraphFactory("remote:localhost/db", "admin", "admin").setupPool(0, 10);
       
       
OrientGraph graph = graphFactory.getTx();
       
try {
           
File file = new File("/path/to/my/file");
           
InputStream in = new BufferedInputStream( new FileInputStream(file) );
           
            graph
.declareIntent( new OIntentMassiveInsert() );
           
           
List<ORID> chunks = new ArrayList<ORID>();

           
int size = 0;
           
int chunkSize = 8192;

           
ODatabaseDocumentTx rawGraph = graph.getRawGraph();
           
while (in.available() > 0) {
               
final ORecordBytes chunk = new ORecordBytes();

                size
+= chunk.fromInputStream(in, chunkSize);

                rawGraph
.save(chunk);

                chunks
.add(chunk.getIdentity());
           
}

           
OrientVertex vertex = graph.addVertex(null);
            vertex
.setProperty("chunks", chunks);
            vertex
.setProperty("size", size);
           
            graph
.declareIntent( null );
           
           
long time = System.currentTimeMillis();
            graph
.commit();
           
long dt = System.currentTimeMillis() - time;
           
System.out.println("Time : " + dt / 1000.0f);
           
       
} catch (IOException e) {
            graph
.rollback();
       
} finally {
            graph
.shutdown();
       
}


For my test, i use a video file (70MB).
I expect no more than 7 sec (10MB/sec which sounds acceptable).

Here is my results (with ODB v2.0, no network app in background) :
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:40:51 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=563 887MB)
Time 52.84
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:41:52 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=563 625MB)
Time 16.816
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:42:17 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=563 334MB)
Time 22.637
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:43:02 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 989MB)
Time 13.024
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:43:28 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=563 236MB)
Time 11.74
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:43:48 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 987MB)
Time 22.274
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:48:08 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 625MB)
Time 3.013
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:48:21 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 897MB)
Time 12.335
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:48:56 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 573MB)
Time 17.509
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 6:49:31 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 274MB)
Time 28.329
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv
. 29, 2015 7:06:36 PM com.orientechnologies.common.log.OLogManager log
INFOS
: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 882MB)
Time 166.533

Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv. 29, 2015 7:12:02 PM com.orientechnologies.common.log.OLogManager log
INFOS: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 505MB)
Time 228.504
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv. 29, 2015 7:17:49 PM com.orientechnologies.common.log.OLogManager log
INFOS: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 387MB)
Time 17.534
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv. 29, 2015 7:18:14 PM com.orientechnologies.common.log.OLogManager log
INFOS: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 132MB)
Time 21.962
Mac-mini-de-Kor:Desktop kor$ java -jar  ODBUpload.jar
janv. 29, 2015 7:18:50 PM com.orientechnologies.common.log.OLogManager log
INFOS: OrientDB auto-config DISKCACHE=1 137MB (heap=910MB os=4 096MB disk=562 152MB)
Time 17.493

Best case : 3 sec
Worse case : 228 sec

What's wrong with this code ? How to improve it ?

Thank you very much.
ODBUpload.java
Reply all
Reply to author
Forward
0 new messages