Released OrientDB v0.9.25: new memory model, new graph api, much more stable

[Luca Garulli]

Hi all,

after about 3 months the long waited version 0.9.25 is ready. This is the direct URL to download it: http://orient.googlecode.com/files/orientdb-0.9.25.zip.

The contribution of the users is every day more important. I appreciate so much when I receive test cases and sometime SVN patches to apply. This saves me a lot of work and speed up the roadmap of OrientDB. So... Thank you to all of you!

Main news:

- Brand new memory model with level-1 and level-2 caches (Issue #242)

- SQL prepared statement (Issue #49)

- SQL Projections with the support of links (Issue #15)

- Graphical editor for documents in OrientDB Studio app (Issue #217)

- Graph representation in OrientDB Studio app

- Support for JPA annotation by the Object Database interface (Issue #102)

- Smart Console under bash: history, auto completition, etc. (Issue #228)

- Operations to work with GEO-spatial points (Issue #182)

- @rid support in SQL UPDATE statement (Issue #72)

- Range queries against Indexes (Issue #231)

- 100% support of TinkerPop Blueprints 0.5

- Many bugs fixed

For the complete list: http://code.google.com/p/orient/issues/list?can=1&q=label:v0.9.25

The next release will be the 1.0RC1 and is planned for April 2011 where the following issues will be implemented: http://code.google.com/p/orient/issues/list?q=label:v1.0RC1

Issues

To report an issue to the Mailing List and to the Issue Tracker please give me always the following information:

• OrientDB version
• Operative System
• 32-bit or 64-bit
• JVM version
• RAM

To open a new issue: http://code.google.com/p/orient/issues/entry

Professional Services

Every month OrientDB gains the trust of new users that want to benefit of the professional services to speed-up the development of their applications:

• Support for servers in production: http://www.orientechnologies.com/professional.htm#support
• On-site consulting, to help companies to build software using OrientDB: http://www.orientechnologies.com/professional.htm#consulting
• Training, with 6 courses: http://www.orientechnologies.com/training.htm

Events

Next events where there will be a talk about OrientDB are:

• Codemotion on March 5th 2011 in Rome, Italy, the most important Italian conference about development (ex-Javaday): http://www.codemotion.it/evento/talk-perch%C3%A8-potresti-aver-bisogno-di-un-database-nosql-anche-se-non-sei-google-o-facebook
• NoSQL Day on March 25th 2011 in Brescia, Italy: http://www.nosqlday.it

If you'd like to see a live presentation of OrientDB in your city please contact me via email or twitter (http://www.twitter.com/lgarulli).

Enjoy with OrientDB!

Luca Garulli

http://www.twitter.com/lgarulli

[David Forslund]

I'm trying my test I did a while back with 0.9.24 with a 100Mbyte RDF file and reported to this list.   I modified my code to use the "improved" api with OGraphDatabase.  Now when I try to run it, it blows memory when set at 4Gbytes and has written only about 300Mbytes of data.   The earlier code created an 8.8GB database.    I'm not sure what to do about this.  I've set the declareIntent(new OIntentMassiveInsert()).

I do get the message:
2011-03-02 11:20:55:737 WARN [] Low memory (2.39Gb of 2.67Gb), calling listeners to free memory...Size after free: 11251
Size after free: 4

after a short while, but the "GC overhead limit exceeded"  error occurs a few minutes later.   It appears that it is not able to
do any garbage collection and dies.   This happens with or without the OIntentMassiveInsert() is specified.   When the GC
limit is hit, it has hit the 4GB memory for quite some time.

Any suggestions?

Thanks,

Dave

[Michael Widmann]

Hi Dave

We investigate the same problem here - we work tight together with luca to solve the issue...
Maybe you can jump into and help narrow down the problem?

We do find it when we write binary data in bigger chunks - system dies / or silent stops...

When it ooms - could you help with the stacktrace - and  jstack  <PID> if it silent "stops"?

Think would be great ...

Michael
(bayoda)

2011/3/3 David Forslund <fors...@mail.com>

--
bayoda.com - Professional Online Backup Solutions for Small and Medium Sized Companies

[Luca Garulli]

Hi Dave,

the message "WARN [] Low memory (2.39Gb of 2.67Gb), calling listeners to free memory..." could be normal since OrientDB uses all the resources at the maximum, but when the free heap is less than 15% than the memory is freed.

Can you give me your settings? What about your OS, RAM and JVM version?

Lvc@

[Luca Garulli]

Yeah,

Michael and his staff is contributing a lot to the behaviour of OrientDB with large data (several millions), mostly binary chunks.

Today there is some problem yet on using OrientDB in this scenario, but I/we hope to fix ASAP.

Lvc@

[David Forslund]

On 3/3/2011 1:27 AM, Luca Garulli wrote:

Hi Dave,

the message "WARN [] Low memory (2.39Gb of 2.67Gb), calling listeners to free memory..." could be normal since OrientDB uses all the resources at the maximum, but when the free heap is less than 15% than the memory is freed.

Can you give me your settings? What about your OS, RAM and JVM version?

Vista 64bit, 12GB Ram 1.6.0_22 JVM.  I set the heap max to -Xmx4096m

When it crashes it has been at the maximum heap memory usage for a while.

This is the stacktrace:

2011-03-02 11:20:55:737 WARN [] Low memory (2.39Gb of 2.67Gb), calling listeners to free memory...Size after free: 11251
Size after free: 4

com.orientechnologies.orient.core.exception.ODatabaseException: Error on saving record #6:9469 of class 'OGraphEdge'
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at com.orientechnologies.common.log.OLogManager.exception(OLogManager.java:172)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:100)
    at dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:99)
Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Error on saving record in cluster #24
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:509)
    at com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
    at com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:148)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:31)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:645)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:233)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:221)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
    at com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:103)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
    at com.orientechnologies.orient.core.tx.OTransactionNoTx.save(OTransactionNoTx.java:51)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:156)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.save(ODatabaseRecordTx.java:33)
    at com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.save(ODatabaseRecordWrapperAbstract.java:135)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:88)
    ... 1 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
    at java.lang.StringBuilder.<init>(StringBuilder.java:68)
    at com.orientechnologies.orient.core.id.ORecordId.generateString(ORecordId.java:167)
    at com.orientechnologies.orient.core.id.ORecordId.toString(ORecordId.java:90)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.linkToStream(ORecordSerializerCSVAbstract.java:667)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerCSVAbstract.fieldToStream(ORecordSerializerCSVAbstract.java:296)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerSchemaAware2CSV.toString(ORecordSerializerSchemaAware2CSV.java:221)
    at com.orientechnologies.orient.core.serialization.serializer.record.string.ORecordSerializerStringAbstract.toStream(ORecordSerializerStringAbstract.java:70)
    at com.orientechnologies.orient.core.record.ORecordAbstract.toStream(ORecordAbstract.java:103)
    at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeSaveRecord(ODatabaseRecordAbstract.java:436)
    ... 18 more

[Luca Garulli]

You should fix it by disabling cache. The issue has been fixed in 1.0rc1-SNAPSHOT but without making any upgrade try to set those (for more information look at http://code.google.com/p/orient/wiki/PerformanceTuning#Configuration):

storage.cache.size = 0

db.cache.size = 0

db.cache.enabled = false

Let me know if works,

Lvc@

[David Forslund]

I believe I have a different problem that I need to resolve before I deal with the cache.   I just have to make sure my code is working correctly before dealing with the memory issues.   The changes I made to use the new API may have broken something else.  More later.

Thanks,

Dave

[David Forslund]

In using the new OGraphDatabase API, I now get the following error:
java.lang.ClassCastException: com.orientechnologies.orient.core.db.record.ORecordLazyList cannot be cast to com.orientechnologies.orient.core.db.record.ORecordLazySet
    at com.orientechnologies.orient.core.db.graph.OGraphDatabase.createEdge(OGraphDatabase.java:235)
    at com.orientechnologies.orient.core.db.graph.OGraphDatabase.createEdge(OGraphDatabase.java:214)
    at dataManager.orientdb.BuildGraphDatabase.createGraphObject(BuildGraphDatabase.java:272)
    at dataManager.orientdb.BuildGraphDatabase.main(BuildGraphDatabase.java:93)

when I get a ODocument (Vertex) from the database and then call createEdge() to link it to another ODocument I've created. 

I can't tell why this is happening.

I was caching the objects I created in memory besides saving them to the database, but figured that this is part of my memory problems so I
went back to finding the object in the database (hopefully from its cache), and this is the result.

Thanks,

Dave

[Luca Garulli]

Hi David,

are you using a db created with previous release of OrientDB? If this is the case probably the auto-migration code doesn't work:

// @COMPATIBILITY 0.9.25

if (vertexBaseClass.getProperty(VERTEX_FIELD_OUT_EDGES).getType() == OType.LINKLIST)

  vertexBaseClass.getProperty(VERTEX_FIELD_OUT_EDGES).setType(OType.LINKSET);

if (vertexBaseClass.getProperty(VERTEX_FIELD_IN_EDGES).getType() == OType.LINKLIST)

  vertexBaseClass.getProperty(VERTEX_FIELD_IN_EDGES).setType(OType.LINKSET);

If this is the case is due to the recent change from LIST to SET as base class to handle Edges since it's much faster with many edges (see some post ago on this ML)

Lvc@

[David]

No.  I'm creating a fresh database.  

I"m doing some simple test cases to make sure I understand the new API, and so far, I'm not doing too well.

I create a small database with linked objects.   I do a query on an object type and get an ODocument back.  It says it has some inEdges as it should as a vertex.  When I call database.getInEdges(object),  it says the object isn't a vertex and aborts.  If I call database.getVertex(object), it says it isn't an edge and aborts.  

So I don't really understand this new api at all, I guess.

Thanks,

Dave