Wrong chunk in embedded mode

[Sebastian Sdorra]

Hi,
I'm trying to use Orientdb 1.0rc8 (from maven repository) in embedded
mode, but the database seems to be corrupt after creation.
Here is a short program which causes the problem in my environment:

public class Main
{

 public static final String CONFIG =
     "<orient-server>"
   +   "<network>"
   +     "<protocols />"
   +     "<listeners />"
   +   "</network>"
   +   "<users>"
   +     "<user name=\"root\" password=\"secret\" resources=\"*\"/>"
   +   "</users>"
   +   "<properties>"
   +     "<entry name=\"server.cache.file.static\" value=\"false\"/>"
   +     "<entry name=\"log.console.level\" value=\"debug\"/>"
   +   "</properties>"
   + "</orient-server>";

 public static void main(String[] args) throws Exception
 {
   File directory = new File("/tmp/db");
   OServer server = OServerMain.create();

   server.startup(CONFIG);
   server.activate();

   ODatabaseDocumentTx connection =
     new ODatabaseDocumentTx("local:".concat(directory.getPath()));

   connection.create();

   OSchema schema = connection.getMetadata().getSchema();
   OClass tuser = schema.createClass("TUser");
   tuser.createProperty(
     "id", OType.STRING
   ).createIndex(INDEX_TYPE.UNIQUE);
   tuser.createProperty(
     "name", OType.STRING
   );

   schema.save();

   connection.close();

   server.shutdown();
 }
}



After the program is finished, i check the database with the console command:


OrientDB console v.1.0rc8 (build 11421) www.orientechnologies.com
Type 'help' to display all the commands supported.

> connect local:/tmp/db admin admin
Connecting to database [local:/tmp/db] with user 'admin'...OK

> check database

Checking database 'db'...

- Checking cluster coherence...
 +- Checking cluster 'internal' (id=0)...
 +- Checking cluster 'index' (id=1)...
 +- Checking cluster 'default' (id=2)...
 +- Checking cluster 'orole' (id=3)...
 +- Checking cluster 'ouser' (id=4)...
 +- Checking cluster 'tuser' (id=5)...

- Checking data chunks integrity...
Mar 20, 2012 2:49:59 PM com.orientechnologies.common.log.OLogManager log
WARNING: [OStorageLocal.check] Found wrong chunk 2386, cause:
com.orientechnologies.orient.core.exception.OStorageException: Error
on reading record from file 'default.0.oda', position 2386, size
1.76Gb: the record size is bigger then the file itself (3.11Kb).
Probably the record is dirty due to a previous crash. It is strongly
suggested to restore the database or export and reimport this one.
       at com.orientechnologies.orient.core.storage.impl.local.ODataLocal.getRecord(ODataLocal.java:168)
       at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.check(OStorageLocal.java:546)
       at com.orientechnologies.orient.console.OConsoleDatabaseApp.checkDatabase(OConsoleDatabaseApp.java:1030)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:208)
       at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:115)
       at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:83)
       at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:127)

Check of database completed in 14ms:
- Total records checked: 11
- Total chunks checked.: 10
- Warnings.............: 0
- Errors...............: 1


Is there a error in the way i create the database or is this a bug of
orientdb? I'm running Mac OS 10.6.8 and Java 1.6.0_29.

[Sebastian Sdorra]

Hi,

I have the same problem with the version 1.0rc9 and i can reproduce the problem without the java program from my first post.

$ ./console.sh  

OrientDB console v.1.0rc9 (build 11668) www.orientechnologies.com

Type 'help' to display all the commands supported.

> create database local:/tmp/db root F8692440147081DDF711C5386C69AC6D5EFC9D264BB3AAF296DD02A23B60C5A4 local document

Creating database [local:/tmp/db] using the storage type [local]...

Database created successfully.

Current database is: local:/tmp/db

> check database

Checking database 'db'...

- Checking cluster coherence...

 +- Checking cluster 'internal' (id=0)...

 +- Checking cluster 'index' (id=1)...

 +- Checking cluster 'default' (id=2)...

 +- Checking cluster 'orole' (id=3)...

 +- Checking cluster 'ouser' (id=4)...

 +- Checking cluster 'orids' (id=5)...

- Checking data chunks integrity...

Mar 27, 2012 4:45:35 PM com.orientechnologies.common.log.OLogManager log

WARNING: [OStorageLocal.check] Deleted chunk at position 66 (recordSize=-673) points to the valid RID #11364:7306634602413835116 instead of #-1:-1

Check of database completed in 6ms:

- Total records checked: 10

- Total chunks checked.: 11

- Warnings.............: 1

- Errors...............: 0

> create class tuser

Class created successfully with id=4

> check database

Checking database 'db'...

- Checking cluster coherence...

 +- Checking cluster 'internal' (id=0)...

 +- Checking cluster 'index' (id=1)...

 +- Checking cluster 'default' (id=2)...

 +- Checking cluster 'orole' (id=3)...

 +- Checking cluster 'ouser' (id=4)...

 +- Checking cluster 'orids' (id=5)...

 +- Checking cluster 'tuser' (id=6)...

- Checking data chunks integrity...

Mar 27, 2012 4:45:47 PM com.orientechnologies.common.log.OLogManager log

WARNING: [OStorageLocal.check] Found wrong chunk 822, cause: 

com.orientechnologies.orient.core.exception.OStorageException: Error on reading record from file 'default.0.oda', position 822, size 1.01Gb: the record size is bigger then the file itself (2.62Kb). Probably the record is dirty due to a previous crash. It is strongly suggested to restore the database or export and reimport this one.

at com.orientechnologies.orient.core.storage.impl.local.ODataLocal.getRecord(ODataLocal.java:168)

at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.check(OStorageLocal.java:546)

at com.orientechnologies.orient.console.OConsoleDatabaseApp.checkDatabase(OConsoleDatabaseApp.java:1063)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:222)

at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:116)

at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:84)

at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:127)

Check of database completed in 8ms:

- Total records checked: 10

- Total chunks checked.: 4

- Warnings.............: 0

- Errors...............: 1

> exit

[Luca Garulli]

Hi,
The check database command was finished when you tried it. Coupled you retry it with latest snapshot?

Lvc@

Sent from a touch phone: sorry for this fruit of crappy keyboard..

[Sebastian Sdorra]

Hi,
I've downloaded the latest snapshot from https://oss.sonatype.org
(orientdb-1.0-20120426.223106-16-distribution.tar.gz). When i try to
start the console i get the following exception:

Exception in thread "main" java.lang.NoClassDefFoundError:
com/orientechnologies/orient/core/command/OCommandOutputListener
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Caused by: java.lang.ClassNotFoundException:
com.orientechnologies.orient.core.command.OCommandOutputListener
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 12 more

After that i build orientdb from source (Revision 5397), but i get the
same result.

Sebastian

2012/4/27 Luca Garulli <l.ga...@gmail.com>:

[Luca Garulli]

Are you using ant or mvn to build it?

Lvc@

[Sebastian Sdorra]

I've tested both (for maven i had to skip the tests).

[Luca Garulli]

Try:

> ant clean install

Jetkins says that it's all fine:  http://datastorm.com.ua/jenkins/ 

Lvc@

[Sebastian Sdorra]

The build is fine, but the console does not work:

ant clean install
...
cd distribution/target
tar xvfz orientdb-1.0-SNAPSHOT-distribution.tar.gz
orientdb-1.0-SNAPSHOT/bin/console.sh

Exception in thread "main" java.lang.NoClassDefFoundError:
com/orientechnologies/orient/core/command/OCommandOutputListener
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Caused by: java.lang.ClassNotFoundException:
com.orientechnologies.orient.core.command.OCommandOutputListener
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 12 more

2012/4/27 Luca Garulli <l.ga...@gmail.com>:

[Luca Garulli]

Probably the problem is on .sh. Can you check if any variable is unset?

Lvc@

[Sebastian Sdorra]

I think the classpath is missing. I've changed the following line from:

java -client -Dfile.encoding=utf-8
-Dorientdb.build.number="trunk@r5219; 2012-04-27 09:05:11+0200" -jar
$ORIENTDB_HOME/lib/orientdb-tools-1.0-SNAPSHOT.jar $*

to:
java -client -Dfile.encoding=utf-8
-Dorientdb.build.number="trunk@r5219; 2012-04-27 00:27:58+0200" -cp
"$ORIENTDB_HOME/lib/*"
com.orientechnologies.orient.console.OConsoleDatabaseApp $*

With this change the console works.

2012/4/27 Luca Garulli <l.ga...@gmail.com>:

[Sebastian Sdorra]

Now i could run the check database command and the output looks better:

Check of database completed in 50ms:
- Total records checked: 13
- Total chunks checked.: 14

- Warnings.............: 1
- Errors...............: 0

There is a warning, but the output does not show what kind warning it is.

Sebastian

2012/4/27 Sebastian Sdorra <s.sd...@gmail.com>:

[Luca Garulli]

Use:

> check database -v

Lvc@

Sent from a touch phone: sorry for this fruit of crappy keyboard..

[Sebastian Sdorra]

Hi,
Ok the database check works. But now i have an other problem. After a
restart of the jvm with embedded server where clauses in queries not
longer work. For example i create an entry (tuser) and fetch it with a
select:

select tuser where name = "test"

It works, but when i restart the jvm the query returns null. Is the
problem related to the warning of the database check below? Perhaps
it is the same problem like this
https://groups.google.com/forum/?fromgroups#!starred/orient-database/KNj2367Pf-c
one.

> check database -v

Checking database 'db'...

(1) Checking data-clusters. This activity checks if pointers to data
are coherent.
- data-cluster #0 internal -> OK
- data-cluster #1 index -> OK
- data-cluster #2 default -> OK
- data-cluster #3 orole -> OK
- data-cluster #4 ouser -> OK
- data-cluster #5 orids -> OK
- data-cluster #6 tuser -> OK

(2) Checking data chunks integrity. In this phase data segments are
scanned to check the back reference into the clusters.
- data-segment default (id=0) size=2719/2719...
-- found 1 holes:
--- hole #0 offset=2005 size=136
-- checking chunks:
--- chunk #0 offset=0 size=4 -> OK
--- chunk #1 offset=18 size=50 -> OK
--- chunk #2 offset=82 size=132 -> OK
--- chunk #3 offset=228 size=266 -> OK
--- chunk #4 offset=508 size=33 -> OK
--- chunk #5 offset=555 size=33 -> OK
--- chunk #6 offset=602 size=413 -> OK
--- chunk #7 offset=1029 size=841 -> OK
--- chunk #8 offset=1884 size=27 -> OK
--- chunk #9 offset=1925 size=66 -> OK
--- chunk #10 offset=2005 size=892879409 -> WARN: Chunk
default:10 (offset=2005 size=892879409) it's between the holes (hole
#0) even if has no negative recordSize. Jump the content OK
--- chunk #11 offset=2141 size=133 -> OK
--- chunk #12 offset=2288 size=270 -> OK
--- chunk #13 offset=2572 size=133 -> OK

Check of database completed in 16ms:

- Total records checked: 13
- Total chunks checked.: 14

- Warnings.............: 1
- Errors...............: 0

[Sebastian Sdorra]

When i rebuild the index on startup the query works. It seems to be

the same problem like this

https://groups.google.com/forum/?fromgroups#!starred/orient-database/KNj2367Pf-c.

Sebastian

2012/4/29 Sebastian Sdorra <s.sd...@gmail.com>:

[Luca Garulli]

Hi,

probably you don't close the database correctly.

Set these parameters (via jvm, System.property or OGlobalConfiguration.xxx.setValue()):

• -Dstorage.keepOpen=false This prevent OrientDB to leave the database open
• -Dmvrbtree.lazyUpdates=1 This avoid OrientDB to buffer the index changes

For more information look at: http://code.google.com/p/orient/wiki/PerformanceTuning#Parameters 

Lvc@

[Sebastian Sdorra]

Hi,
Thank you that seems to fix the problem.

Sebastian

2012/4/29 Luca Garulli <l.ga...@gmail.com>: