Big Tables
davidthings
I've been doing a little investigation into Persevere performance and
behavior with big tables. Mostly I've been finding that there are a
few places where there are assumptions that might not be optimal:
At least in the version r388, it seems like all objects in all Classes
are loaded into memory at startup. I have my Java VM set to max out
at around 400MB, and with that it looks like somewhere around 400,000
single integer attribute objects is about as many as I can get.
This means
a) that start up can be very slow - O( n x m ) where n is number of
records, m is number of attributes. (>20mins!)
b) that start up can fail when there are a lot of objects in memory
(crashing with heap errors)
It seems like possibly this load all objects thing might happen in
other places as well (index updates?). Around the same limit (400,000
single integer objects) creation of new objects gets a bit tricky and
at least once I've seen a failure with memory problems.
It seems like indexing all attributes by default is maybe not quite
the right thing. If I have a lot of objects that are simply { value:
[integer value] }, I might prefer that the DB not index them until I
issue a request that might indicate that I'm every going to access
'value' in any search-like way. I wonder about assuming no index on
an attribute until someone does a query that might use it, unless the
'index' meta-value is set in which case it is respected.
From a philosophical point of view, should we take databases that can
fit in memory as the upper bound on database size for the forseable
future? The application we're evaluating Persevere for, and I submit,
many others will want a DB that can size up without hard limits.
I see r389 might work with these issues a bit and may change things...
I'm doing some comparisons now.
davidthings
140B per single integer change) makes a few more things come to mind:
1) Is the reason for the long start up time that object states are
built up from replaying all the transactions that have occurred since
the database was created?
2) How would this work in a system where lots of data changes all the
time? In our system much of the user's data changes second to
second. Would there be a transaction compactor, or would there be a
separate state table?
Kris Zyp
davidthings wrote:
> Hi,
>
> I've been doing a little investigation into Persevere performance and
> behavior with big tables. Mostly I've been finding that there are a
> few places where there are assumptions that might not be optimal:
>
> At least in the version r388, it seems like all objects in all Classes
> are loaded into memory at startup. I have my Java VM set to max out
> at around 400MB, and with that it looks like somewhere around 400,000
> single integer attribute objects is about as many as I can get.
>
No, that shouldn't be true. I have created tables with +1000000 objects
with multiple properties, and it starts in just a couple seconds with
about 23MB of memory (the normal usage). Persevere should only load
objects into memory as needed. Are you sure you don't have anything in
your startup that is somehow iterating through and loading every object?
> This means
> a) that start up can be very slow - O( n x m ) where n is number of
> records, m is number of attributes. (>20mins!)
> b) that start up can fail when there are a lot of objects in memory
> (crashing with heap errors)
>
> It seems like possibly this load all objects thing might happen in
> other places as well (index updates?). Around the same limit (400,000
> single integer objects) creation of new objects gets a bit tricky and
> at least once I've seen a failure with memory problems.
>
Rebuilding the indexes requires that all the objects be processed, but
they don't need to all be in memory at one time for this to take place.
They should be freed after they are processed.
> It seems like indexing all attributes by default is maybe not quite
> the right thing. If I have a lot of objects that are simply { value:
> [integer value] }, I might prefer that the DB not index them until I
> issue a request that might indicate that I'm every going to access
> 'value' in any search-like way. I wonder about assuming no index on
> an attribute until someone does a query that might use it, unless the
> 'index' meta-value is set in which case it is respected.
>
It actually is adaptive. Indexes that aren't used will become inactive,
and won't be updated.
> From a philosophical point of view, should we take databases that can
> fit in memory as the upper bound on database size for the forseable
> future? The application we're evaluating Persevere for, and I submit,
> many others will want a DB that can size up without hard limits.
>
Absolutely, Persevere databases should be able to far exceed memory, and
in the load tests I have performed I have done extensive queries,
updates, and more with tables that far exceed the amount of memory
allocated to Persevere without any problem.
It is certainly to get in situations where all excessive memory can be
consumed, especially with queries and application code that might load
all the objects into memory to carry out an action.
Kris
davidthings
Here's what I'm doing. Hopefully there's something dumb in there!
Define the class Nano8 as included below.
Start persevere from a clean database.
Then create 100,000 objects
> Nano8.create( 100000 )
Understandably, this takes a little bit (approx 1.28ms/object)
Then, shutdown and restart persevere.
After about 4 minutes and 40s... DB is up again.
Class( {
"extends":"Object",
"id":"Nano8",
"create":function(count){
base = load( "/Nano8.length" );
console.log("Creating " + count + " Nano8s. Already " + base );
var i = 0;
var start = new Date();
while(i++ < count)
{
var n = new Nano8( );
commit();
}
var elapsed = new Date() - start;
console.log(" Elapsed time: " + elapsed );
console.log(" Mean: " + (elapsed / count) + " ms per object");
},
"access":function(count){
var start = new Date();
var min = 1000000;
var max = 0;
var i = 0;
var base = load( "/Nano8.length" );
console.log(" Access " + count + " Tinys " );
while(i++ < count)
{
index = Math.floor( Math.random() * base ) + 1
var startThis = new Date();
var s = load( "/Nano8/" + index );
s.access( );
commit();
var elapsedThis = new Date() - startThis;
if ( ( count < 1000 ) || ( ( i % 1000 ) == 0 ) )console.log("
Accessed Nano8 " + i + ":" + index + " " + elapsedThis + " ms" );
if ( elapsedThis > max ) max = elapsedThis;
if ( elapsedThis < min ) min = elapsedThis;
// console.log(" Tiny " + s.name );
}
var elapsed = new Date() - start;
console.log(" Elapsed time: " + elapsed );
console.log(" Mean: " + (elapsed / count) + " ms per
object)");
console.log(" Min: " + min + " ms, Max:" + max + " ms");
},
"show":function( count ){
var start = new Date();
var i = 0;
var min = 1000000;
var max = 0;
base = load( "/Nano8.length" );
if ( count != null && count <= base )
base = count;
while(i++ < base )
{
console.log(" Nano8 " + i );
var startThis = new Date();
var n = load( "/Nano8/" + i );
var elapsedThis = new Date() - startThis;
if ( elapsedThis > max ) max = elapsedThis;
if ( elapsedThis < min ) min = elapsedThis;
n.output();
}
var elapsed = new Date() - start;
console.log(" Elapsed time: " + elapsed );
console.log(" Mean: " + (elapsed / count) + " ms per
object");
console.log(" Min: " + min + " ms, Max:" + max + " ms");
},
"prototype":{
initialize:function( ) {
this.accesses = 0;
this.p2 = 0;
this.p3 = 0;
this.p4 = 0;
this.p5 = 0;
this.p6 = 0;
this.p7 = 0;
this.p8 = 0;
},
access:function( ) {
this.accesses += 1;
this.p2 += Math.floor( Math.random() * 100 );
this.p3 += Math.floor( Math.random() * 100 );
this.p4 += Math.floor( Math.random() * 100 );
this.p5 += Math.floor( Math.random() * 100 );
this.p6 += Math.floor( Math.random() * 100 );
this.p7 += Math.floor( Math.random() * 100 );
this.p8 += Math.floor( Math.random() * 100 );
},
output:function( ) {
console.log( this.id + " : " + this.accesses + " accesses " );
console.log( " P2:" + this.p2 + " P3:" + this.p3 + " P4:" +
this.p4 + " P5:" + this.p5 + " P6:" + this.p6 + " P7:" + this.p7 + "
P8:" + this.p8 );
}
},
"properties":{
"accesses":{index: false, "type":"integer", "default":0 },
"p2":{index: false, "type":"integer", "default":0 },
"p3":{index: false, "type":"integer", "default":0 },
"p4":{index: false, "type":"integer", "default":0 },
"p5":{index: false, "type":"integer", "default":0 },
"p6":{index: false, "type":"integer", "default":0 },
"p7":{index: false, "type":"integer", "default":0 },
"p8":{index: false, "type":"integer", "default":0 }
}
} );
davidthings
At 60,000 Nano8 objects, startup takes approximately 134s. (At
100,000 startup is a little unstable on my machine - memory ceiling?)
During the last 30s of this startup process, the transaction.psv file
grows by 8733696B, which is consistent with a single value change for
each of the 60,000 records.
D.
P.S. My machine is Quad 2.4GHz 2GB Mem Windows Vista
Kris Zyp
problem is that when the Class function/constructor is executed it
updates the internal schema, triggering an operation that goes through
every instance of the class to ensure that it is valid by the schema,
and coerces properties if it is not (this was not an issue when classes
were defined in the data source, because those changes were seen as
application initiated). This can actually be very useful if you make a
schema change, and you want to ensure all the instances are valid, but
with a big table, this is obviously far too expensive. I have therefore
made this a configurable capability, and disabled by default (some
regrettably, as I really liked that feature). The trunk version of
Persevere should start up without loading all the objects now.
Kris
Kris Zyp
davidthings wrote:
> Seeing also that the transaction table grows with each access (approx
> 140B per single integer change) makes a few more things come to mind:
>
> 1) Is the reason for the long start up time that object states are
> built up from replaying all the transactions that have occurred since
> the database was created?
>
No, transactions are only replayed if the index needs to be rebuilt. If
the server crashes before indexes are finished updating, there is also
incremental replay, but that should obviously be very minimal.
> 2) How would this work in a system where lots of data changes all the
> time? In our system much of the user's data changes second to
> second. Would there be a transaction compactor, or would there be a
> separate state table?
>
A database compactor would be a nice feature. However, for databases
where the number of updates on objects is not significantly greater than
the number of objects, compaction would have limited value. On the other
hand, in many situations where data changes extremely rapidly, in may
not even be necessary to persist the data (and use the InMemorySource),
and it may be more efficient to simply rebuild the table when the server
is restarted (if the data is no longer relevant after a few minutes). Of
course, there is certainly an area in between where the transaction
compactor would be useful. In the meantime, the old internal storage
system (the DynaObjectDBSource) might be useful if the JavaScriptDB's
transaction file is growing too fast for the storage limits.
Kris
davidthings
Thanks for the clarification. I think monotonic db growth from
updates alone might be a bit of a problem for a lot of our data. We
maintain access timestamps, running totals, etc., both of which would
be miserable in a db gets bigger every write situation. To say
nothing of algorithms that run O( t ), where t is the number of
transactions in the DB.
Anyway, regardless of this, perhaps it is a good time to put in a
request for an update to the Architecture and Roadmap docs. They're
starting to look a little old. How does the data store really work,
what's really standing out as not complete, what kinds of data is
Persevere optimal for, etc.
It might help us with expectations, plans, etc.
D.