Orient-DB query limit?

[secmask]

Hi
I've create a database with about 50 millions records, a query to
select a document with 1000 for example give me an exception:

com.orientechnologies.orient.core.exception.ODatabaseException: Record
position #524288000 was bound to file #1 that is out of limit
(current=0)
at
com.orientechnologies.orient.core.storage.impl.local.OMultiFileSegment.getRelativePosition(OMultiFileSegment.java:
252)
at
com.orientechnologies.orient.core.storage.impl.local.ODataLocal.getRecord(ODataLocal.java:
107)
at
com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.readRecord(OStorageLocal.java:
872)
at
com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.browseCluster(OStorageLocal.java:
531)
at
com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.browse(OStorageLocal.java:
501)
at
com.orientechnologies.orient.core.query.nativ.ONativeAsynchQuery.run(ONativeAsynchQuery.java:
78)
at
com.orientechnologies.orient.core.query.nativ.ONativeSynchQuery.run(ONativeSynchQuery.java:
42)
at
com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:
34)
at
com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.query(ODatabaseRecordAbstract.java:
220)
at
com.orientechnologies.orient.core.db.ODatabaseRecordWrapperAbstract.query(ODatabaseRecordWrapperAbstract.java:
84)
at me.test.TestDB.testShowData(TestDB.java:119)

Is this a limitation?

[Luca Garulli]

Hi,
no it's a bug. OrientDB support theorically 9.223.372.036 Billions of
records. I'm saying theoretically because I don't know if anyone could
never test it.

Can you copy and paste your database directory structure here? A "ls
-al" should be enough.

Lvc@

[secmask]

Hi Luca, this is it:

E:\mydb>dir
Volume in drive E is New Volume
Volume Serial Number is 8200-A440

Directory of E:\mydb

08/19/2010 05:10 PM <DIR> .
08/19/2010 05:10 PM <DIR> ..
08/19/2010 05:53 PM 524,289,019 account.0.ocl
08/19/2010 05:53 PM 226,565,449 account.1.ocl
08/19/2010 05:53 PM 262,144 account.och
08/19/2010 05:53 PM 1,049,600 default.0.ocl
08/19/2010 05:53 PM 524,289,024 default.0.oda
08/19/2010 05:21 PM 10,485,760 default.1.oda
08/19/2010 04:58 PM 524,289,024 default.10.oda
08/19/2010 05:02 PM 524,289,024 default.11.oda
08/19/2010 05:04 PM 524,289,024 default.12.oda
08/19/2010 05:06 PM 524,289,024 default.13.oda
08/19/2010 05:08 PM 524,289,024 default.14.oda
08/19/2010 05:10 PM 524,289,024 default.15.oda
08/19/2010 05:12 PM 479,603,342 default.16.oda
08/19/2010 04:44 PM 524,289,024 default.2.oda
08/19/2010 04:46 PM 524,289,024 default.3.oda
08/19/2010 04:48 PM 524,289,024 default.4.oda
08/19/2010 04:49 PM 524,289,024 default.5.oda
08/19/2010 04:51 PM 524,289,024 default.6.oda
08/19/2010 04:53 PM 524,289,024 default.7.oda
08/19/2010 04:54 PM 524,289,024 default.8.oda
08/19/2010 04:56 PM 524,289,024 default.9.oda
08/19/2010 05:53 PM 262,144 default.och
08/19/2010 05:53 PM 263,168 default.odh
08/19/2010 05:53 PM 1,048,576 index.0.ocl
08/19/2010 05:53 PM 262,144 index.och
08/19/2010 05:53 PM 1,049,600 internal.0.ocl
08/19/2010 05:53 PM 262,144 internal.och
08/19/2010 05:53 PM 1,000,000 txlog.otx
28 File(s) 9,110,738,450 bytes
2 Dir(s) 27,904,212,992 bytes free


Schema create with:

OClass account = database
.getMetadata()
.getSchema()
.createClass(
"Account",
database.getStorage().addCluster("account",
OClusterLocal.TYPE));
account.createProperty("id", OType.LONG);
account.createProperty("val1", OType.DOUBLE);
account.createProperty("val2", OType.DOUBLE);
account.createProperty("val3", OType.FLOAT);
account.createProperty("val4", OType.SHORT);
account.createProperty("val5", OType.STRING);
account.createProperty("name", OType.STRING);

Insert data code:

ODatabaseDocumentTx db = new ODatabaseDocumentTx(url).open("admin",
"admin");
db.declareIntent(new OIntentMassiveInsert());
db.begin(TXTYPE.NOTX);
long ndoc = 50000000;
ODocument doc = new ODocument();
long start = System.nanoTime();
long block = System.nanoTime();
for (long i = 1; i <= ndoc; i++) {
doc.field("id", i);
doc.field("val1", 4.0d);
doc.field("val2", 5.0d);
doc.field("val3", 6.0f);
doc.field("val4", 255);
doc.field("val5",
"this is the description for a long comic books -" + i);
doc.field("name", "this is secmask put on top - " + i);
doc.setClassName("Account");
doc.setDatabase(db);
doc.save();
doc.reset();
if (i % 100000 == 0) {
double time = (double) (System.nanoTime() - block) / 1000000;
System.out.println(i * 100 / ndoc + "%.\t" + time + "\t"
+ 100000.0d / time + " docs/ms");
block = System.nanoTime();
}
}
db.commit();


and the query:

List<ODocument> result = db
.query(new ONativeSynchQuery<ODocument,
OQueryContextNativeSchema<ODocument>>(
db, "Account",
new OQueryContextNativeSchema<ODocument>()) {
@Override
public boolean filter(
OQueryContextNativeSchema<ODocument> iRecord) {
return iRecord.field("id").eq(1000l).field("name").go();
}
});
for (ODocument o : result) {
System.out.println("id=" + o.field("id") + "\tname="
+ o.field("name"));
}


and other thing is that, i using for-loop to create 50 millions
record, but when I call long c = db.countClass("Account"); , it
returns value is 49.999.999 .

secmask.

On Aug 19, 6:53 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi,
> no it's a bug. OrientDB support theorically 9.223.372.036 Billions of
> records. I'm saying theoretically because I don't know if anyone could
> never test it.
>
> Can you copy and paste your database directory structure here? A "ls
> -al" should be enough.
>
> Lvc@
>

[Luca Garulli]

Hi,
seems that OrientDB doesn't reload the multi files... It seems to be a
bug. I'm trying to write a test case with your code.

For the counting you insert 49.999.999 and not 50.000.000 because the
last .reset(). Call .reset() just after the for() and it should work

bye,
Luca Garulli
http://www.linkedin.com/in/garulli
OrientDB (www.OrienTechnologies.com)

[Luca Garulli]

I've found and fixed the bug: no data loss, multiple file segments was
not loaded on open().

I've slightly improved your test to use the index after the insertion
of 50.000.000 of record (it's faster than create the index and
insert):

database.getMetadata().getSchema().getClass("Account").getProperty("id").createIndex(INDEX_TYPE.UNIQUE);

And, in order to use the index, I've used a SQL query in place of your:

List<ODocument> result = database.query(new
OSQLSynchQuery<ODocument>("SELECT FROM Account WHERE id = " + 1000l));

You can find it at:
http://code.google.com/p/orient/source/browse/trunk/tests/src/test/java/com/orientechnologies/orient/test/database/users/SecMaskTest.java

bye,
Lvc@

[secmask]

Hi Luca!
When does it actually save the Document?, I do the test and saw that,
It won't save the Document if I didn't call doc.reset(), or something
strange here.
I have an other test like this:

ODatabaseDocumentTx db = new ODatabaseDocumentTx(url).open("admin",
"admin");
db.declareIntent(new OIntentMassiveInsert());
db.begin(TXTYPE.NOTX);

ODocument doc = new ODocument();

int i = 1;

doc.field("id", i);
doc.field("val1", 4.0d);
doc.field("val2", 5.0d);
doc.field("val3", 6.0f);
doc.field("val4", 255);
doc.field("val5", "this is the description for a long comic books -"
+ i);
doc.field("name", "this is secmask put on top - " + i);
doc.setClassName("Account");
doc.setDatabase(db);
doc.save();

i++;
ODocument doc2 = new ODocument();
doc2.field("id", i);
doc2.field("val1", 4.0d);
doc2.field("val2", 5.0d);
doc2.field("val3", 6.0f);
doc2.field("val4", 255);
doc2.field("val5", "this is the description for a long comic books
-"
+ i);
doc2.field("name", "this is secmask put on top - " + i);
doc2.setClassName("Account");
doc2.setDatabase(db);
doc2.save();

db.commit();

System.out.println("docCount: "+db.countClass("Account"));
db.close();

with a fresh DB, I create 2 document object, set fields value, save()
it and commit() the database, but db.countClass("Account") always
return 0.
The previous test with 50 millions records, I print the document info,
and I saw that it didn't save the first document ( #id = 1).

On Aug 20, 10:10 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> I've found and fixed the bug: no data loss, multiple file segments was
> not loaded on open().
>
> I've slightly improved your test to use the index after the insertion
> of 50.000.000 of record (it's faster than create the index and
> insert):
>
> database.getMetadata().getSchema().getClass("Account").getProperty("id").createIndex(INDEX_TYPE.UNIQUE);
>
> And, in order to use the index, I've used a SQL query in place of your:
>
> List<ODocument> result = database.query(new
> OSQLSynchQuery<ODocument>("SELECT FROM Account WHERE id = " + 1000l));
>

> You can find it at:http://code.google.com/p/orient/source/browse/trunk/tests/src/test/ja...
>
> bye,
> Lvc@

>
> On 20 August 2010 16:02, Luca Garulli <l.garu...@gmail.com> wrote:
>
> > Hi,
> > seems that OrientDB doesn't reload the multi files... It seems to be a
> > bug. I'm trying to write a test case with your code.
>
> > For the counting you insert 49.999.999 and not 50.000.000 because the
> > last .reset(). Call .reset() just after the for() and it should work
>
> > bye,
> > Luca Garulli
> >http://www.linkedin.com/in/garulli
> > OrientDB (www.OrienTechnologies.com)
>

[secmask]

hei, are u there?

> ...
>
> read more »

[Luca Garulli]

Hi,
The sequence inside a loop should be:
1) Reset the document
2) Set the database
3) Set the class
4) Set all the fields
5) Save it

Let me know
Lvc@

2010/8/24, secmask <sec...@gmail.com>:

--
Inviato dal mio dispositivo mobile

[secmask]

Hi
I don't have any loop, I assume that reset() only need when reuse
Document. This case, when I save singular Document, I don't need to
call reset(), but strange, it did not save the document.

On Aug 24, 11:52 pm, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi,

> The sequence inside a loop should be:
> 1) Reset the document
> 2) Set the database
> 3) Set the class
> 4) Set all the fields
> 5) Save it
>
> Let me know
> Lvc@
>

> 2010/8/24, secmask <secm...@gmail.com>:

> ...
>
> read more »

[Luca Garulli]

Hi,
Even if you don't need to reuse the document try my sequence but reset.

Bye,
Lvc@

2010/8/25, secmask <sec...@gmail.com>:

--

[secmask]

Hi, it works now :), it should in order as you said.

On Aug 26, 2:54 am, Luca Garulli <l.garu...@gmail.com> wrote:
> Hi,

> Even if you don't need to reuse the document try my sequence but reset.
>
> Bye,
> Lvc@
>

> 2010/8/25, secmask <secm...@gmail.com>:

> ...
>
> read more »

[Luca Garulli]

Ok,
Maybe I need to explain better that the setDatabase() and
setClassname() reset the field contents.

Bye,
Lvc@

2010/8/26, secmask <sec...@gmail.com>:

--