SimpleDBM support, maintenance and scalability

[John Gymer]

I am looking at using SimpleDBM to manage a local data store, which makes use of indexing and transactional capabilities.

First question is whether SimpleDBM is going to be supported and maintained into the future, so if we hit problems/bugs, are they likely to be fixed?

In terms of scalability, are they any experiences that can be shared?  Should SimpleDBM be scalable to support large row counts? (see example below)...

To test, I have created a few tables in SimpleDBM and loaded them from another data source.

For small sets of data this is working OK, but for larger row counts I hit problems.

My first table is defined as follows:

// DOBJ

TypeDescriptor dobj_rowtype[] = {

ff.getLongType(), // 0: ObjId Num(10)

ff.getIntegerType(), // 1: OTC Num(5)

ff.getVarcharType(1), // 2: ChgStatus Char(1)

ff.getVarcharType(32), // 3: Name Char(32)

ff.getVarcharType(1), // 4: Obj_Char_Prop_1 Char(1)

ff.getVarcharType(1), // 5: Obj_Char_Prop_2 Char(1)

ff.getVarcharType(1), // 6: Obj_Char_Prop_3 Char(1)

ff.getVarcharType(1), // 7: Obj_Char_Prop_4 Char(1)

ff.getVarcharType(1), // 8: Obj_Char_Prop_5 Char(1)

ff.getVarcharType(1), // 9: ;Obj_Char_Prop_6 Char(1)

ff.getVarcharType(1), // 10: Obj_Char_Prop_7 Char(1)

ff.getVarcharType(1), // 11: Obj_Char_Prop_8 Char(1)

ff.getVarcharType(1), // 12: Obj_Char_Prop_9 Char(1)

ff.getVarcharType(1), // 13: Obj_Char_Prop_10 Char(1)

ff.getVarcharType(1), // 14: Obj_Char_Prop_11 Char(1)

ff.getVarcharType(1), // 15: Obj_Char_Prop_12 Char(1)

ff.getVarcharType(1), // 16: Obj_Char_Prop_13 Char(1)

ff.getVarcharType(1), // 17: Obj_Char_Prop_14 Char(1)

ff.getVarcharType(1), // 18: Obj_Char_Prop_15 Char(1)

ff.getVarcharType(1), // 19: Obj_Char_Prop_16 Char(1)

ff.getVarcharType(1), // 20: Obj_Char_Prop_17 Char(1)

ff.getVarcharType(1), // 21: Obj_Char_Prop_18 Char(1)

ff.getVarcharType(1), // 22: Obj_Char_Prop_19 Char(1)

ff.getVarcharType(1), // 23: Obj_Char_Prop_20 Char(1)

ff.getVarcharType(1), // 24: Obj_Char_Prop_21 Char(1)

ff.getVarcharType(1), // 25: Obj_Char_Prop_22 Char(1)

ff.getVarcharType(1), // 26: Obj_Char_Prop_23 Char(1)

ff.getVarcharType(1), // 27: Obj_Char_Prop_24 Char(1)

ff.getVarcharType(1), // 28: Obj_Char_Prop_25 Char(1)

ff.getVarcharType(1), // 29: Obj_Char_Prop_26 Char(1)

ff.getVarcharType(1), // 30: Obj_Char_Prop_27 Char(1)

ff.getVarcharType(1), // 31: Obj_Char_Prop_28 Char(1)

ff.getLongType(), // 32: Obj_Int_Prop_1 Num(10)

ff.getLongType(), // 33: Obj_Int_Prop_2 Num(10)

ff.getLongType(), // 34: Obj_Int_Prop_3 Num(10)

ff.getLongType(), // 35: Obj_Int_Prop_4 Num(10)

ff.getLongType(), // 36: Obj_Int_Prop_5 Num(10)

ff.getLongType(), // 37: Obj_Int_Prop_6 Num(10)

ff.getLongType(), // 38: Obj_Int_Prop_7 Num(10)

ff.getLongType(), // 39: Obj_Int_Prop_8 Num(10)

ff.getLongType(), // 40: Obj_Int_Prop_9 Num(10)

ff.getLongType(), // 41: Obj_Int_Prop_10 Num(10)

ff.getLongType(), // 42: Obj_Int_Prop_11 Num(10)

ff.getLongType(), // 43: Obj_Int_Prop_12 Num(10)

ff.getLongType(), // 44: Obj_Int_Prop_13 Num(10)

ff.getIntegerType(), // 45: Obj_SInt_Prop_1 Num(5)

ff.getIntegerType(), // 46: Obj_SInt_Prop_2 Num(5)

ff.getIntegerType(), // 47: Obj_SInt_Prop_3 Num(5)

ff.getIntegerType(), // 48: Obj_SInt_Prop_4 Num(5)

ff.getIntegerType(), // 49: Obj_SInt_Prop_5 Num(5)

ff.getIntegerType(), // 50: Obj_SInt_Prop_6 Num(5)

ff.getIntegerType(), // 51: Obj_SInt_Prop_7 Num(5)

ff.getIntegerType(), // 52: Obj_SInt_Prop_8 Num(5)

ff.getIntegerType(), // 53: Obj_SInt_Prop_9 Num(5)

ff.getIntegerType() // 54: Obj_SInt_Prop_10 Num(5)

};

try {

TableDefinition dobjTableDefinition = db.newTableDefinition("dobj.dat", containerIdDOBJ, dobj_rowtype);

// INDEX creation: (containerId, name, cols (by 0-based pos), primary, unique)

dobjTableDefinition.addIndex(containerIdDOBJIdx1, "dobj1.idx", new int[] { 0 }, true, true); // ObjId

dobjTableDefinition.addIndex(containerIdDOBJIdx2, "dobj2.idx", new int[] { 1, 3, 0 }, false, true); // OTC, Name, ObjId

db.createTable(dobjTableDefinition);

System.out.println("DOBJ Table/Index created OK");

} catch (Exception e) {

System.out.println("ERROR: DOBJ Table/Index could not be created - " + e);

}

As you can see, quite a 'wide' table with many columns.

I then process my input record set to create rows in this table:

System.out.println("Starting DOBJ insert at " + getTS());

Transaction trx = null;

int totalRows = 0;

try {

@SuppressWarnings("rawtypes")

Iterator itDOBJ = dobjList.iterator();

int batchRowCount = 0;

Table tableDOBJ = null;

while (itDOBJ.hasNext()) {

if (trx == null) {

trx = db.startTransaction(IsolationMode.READ_COMMITTED);

}

if (tableDOBJ == null) tableDOBJ = db.getTable(trx, containerIdDOBJ);

DOBJ dobj = (DOBJ)itDOBJ.next();

Row tableDOBJRow = tableDOBJ.getRow();

tableDOBJRow.setLong(0, dobj.objId);

tableDOBJRow.setInt(1, (int)dobj.type);

tableDOBJRow.setString(2, ""+dobj.chgStatus);

if (dobj.name != null) tableDOBJRow.setString(3, dobj.name);

tableDOBJRow.setString(4, ""+dobj.charProp1);

tableDOBJRow.setString(5, ""+dobj.charProp2);

tableDOBJRow.setString(6, ""+dobj.charProp3);

tableDOBJRow.setString(7, ""+dobj.charProp4);

tableDOBJRow.setString(8, ""+dobj.charProp5);

tableDOBJRow.setString(9, ""+dobj.charProp6);

tableDOBJRow.setString(10, ""+dobj.charProp7);

tableDOBJRow.setString(11, ""+dobj.charProp8);

tableDOBJRow.setString(12, ""+dobj.charProp9);

tableDOBJRow.setString(13, ""+dobj.charProp10);

tableDOBJRow.setString(14, ""+dobj.charProp11);

tableDOBJRow.setString(15, ""+dobj.charProp12);

tableDOBJRow.setString(16, ""+dobj.charProp13);

tableDOBJRow.setString(17, ""+dobj.charProp14);

tableDOBJRow.setString(18, ""+dobj.charProp15);

tableDOBJRow.setString(19, ""+dobj.charProp16);

tableDOBJRow.setString(20, ""+dobj.charProp17);

tableDOBJRow.setString(21, ""+dobj.charProp18);

tableDOBJRow.setString(22, ""+dobj.charProp19);

tableDOBJRow.setString(23, ""+dobj.charProp20);

tableDOBJRow.setString(24, ""+dobj.charProp21);

tableDOBJRow.setString(25, ""+dobj.charProp22);

tableDOBJRow.setString(26, ""+dobj.charProp23);

tableDOBJRow.setString(27, ""+dobj.charProp24);

tableDOBJRow.setString(28, ""+dobj.charProp25);

tableDOBJRow.setString(29, ""+dobj.charProp26);

tableDOBJRow.setString(30, ""+dobj.charProp27);

tableDOBJRow.setString(31, ""+dobj.charProp28);

tableDOBJRow.setLong(32, dobj.intProp1);

tableDOBJRow.setLong(33, dobj.intProp2);

tableDOBJRow.setLong(34, dobj.intProp3);

tableDOBJRow.setLong(35, dobj.intProp4);

tableDOBJRow.setLong(36, dobj.intProp5);

tableDOBJRow.setLong(37, dobj.intProp6);

tableDOBJRow.setLong(38, dobj.intProp7);

tableDOBJRow.setLong(39, dobj.intProp8);

tableDOBJRow.setLong(40, dobj.intProp9);

tableDOBJRow.setLong(41, dobj.intProp10);

tableDOBJRow.setLong(42, dobj.intProp11);

tableDOBJRow.setLong(43, dobj.intProp12);

tableDOBJRow.setLong(44, dobj.intProp13);

tableDOBJRow.setInt(45, (int)dobj.sIntProp1);

tableDOBJRow.setInt(46, (int)dobj.sIntProp2);

tableDOBJRow.setInt(47, (int)dobj.sIntProp3);

tableDOBJRow.setInt(48, (int)dobj.sIntProp4);

tableDOBJRow.setInt(49, (int)dobj.sIntProp5);

tableDOBJRow.setInt(50, (int)dobj.sIntProp6);

tableDOBJRow.setInt(51, (int)dobj.sIntProp7);

tableDOBJRow.setInt(52, (int)dobj.sIntProp8);

tableDOBJRow.setInt(53, (int)dobj.sIntProp9);

tableDOBJRow.setInt(54, (int)dobj.sIntProp10);

tableDOBJ.addRow(trx, tableDOBJRow);

totalRows++;

batchRowCount++;

if (batchRowCount >= ROW_INSERT_BATCH_SIZE) {

trx.commit();

// trx = null;

batchRowCount = 0;

System.out.println("   committed DOBJ batch at " + getTS() + " with totalRows so far: " + totalRows);

}

}

if (trx != null) trx.commit();

System.out.println("Inserts to DOBJ OK at " + getTS());

} catch (Exception e) {

System.out.println("ERROR: Insert DOBJ failed - " + e);

e.printStackTrace();

if (trx != null) trx.abort();

    return;

}

The idea is to add 500 rows at a time (ROW_INSERT_BATCH_SIZE) and commit the transaction for each batch.

This logic works OK until I get to about 457000 rows, at which point I receive this exception:

java.lang.ClassCastException: org.simpledbm.rss.impl.sp.SlottedPageImpl cannot be cast to org.simpledbm.rss.impl.fsm.FreeSpaceManagerImpl$SpaceMapPageImpl

at org.simpledbm.rss.impl.fsm.FreeSpaceManagerImpl.doExtendContainer(Unknown Source)

at org.simpledbm.rss.impl.fsm.FreeSpaceManagerImpl.extendContainer(Unknown Source)

at org.simpledbm.rss.impl.tuple.TupleManagerImpl.locateEmptyPage(Unknown Source)

at org.simpledbm.rss.impl.tuple.TupleManagerImpl$TupleInserterImpl.doStartInsert(Unknown Source)

at org.simpledbm.rss.impl.tuple.TupleManagerImpl$TupleInserterImpl.startInsert(Unknown Source)

at org.simpledbm.rss.impl.tuple.TupleManagerImpl$TupleContainerImpl.insert(Unknown Source)

at org.simpledbm.database.impl.TableImpl.addRow(TableImpl.java:114)

at bug.snippet.Main.populate(Main.java:473)

at bug.snippet.Main$1.widgetSelected(Main.java:162)

at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)

at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)

at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)

at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)

at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)

at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)

at bug.snippet.Main.main(Main.java:145)

at bug.snippet.SWTEntryPoint.main(SWTEntryPoint.java:6)

It consistently crashes at the same point with the same test data.

My database properties are defined like the samples I've found:

Properties properties = new Properties();

properties.setProperty("log.ctl.1", "ctl.a");

properties.setProperty("log.ctl.2", "ctl.b");

properties.setProperty("log.groups.1.path", ".");

properties.setProperty("log.archive.path", ".");

properties.setProperty("log.group.files", "3");

properties.setProperty("log.file.size", "5242880");

properties.setProperty("log.buffer.size", "5242880");

properties.setProperty("log.buffer.limit", "4");

properties.setProperty("log.flush.interval", "30");

properties.setProperty("log.disableFlushRequests", "true");

properties.setProperty("storage.createMode", "rw");

properties.setProperty("storage.openMode", "rw");

properties.setProperty("storage.flushMode", "noforce");

properties.setProperty("bufferpool.numbuffers", "1500");

properties.setProperty("bufferpool.writerSleepInterval", "60000");

properties.setProperty("transaction.ckpt.interval", "60000");

properties.setProperty("logging.properties.type", "jdk");

properties.setProperty("logging.properties.file", "classpath:simpledbm.logging.properties");

properties.setProperty("lock.deadlock.detection.interval", "3");

properties.setProperty("storage.basePath", "c:\\MyDBs\\MyTestDB.dbm");

If you can shed any light on the cause, or how to fix, I'd appreciate it.

Thanks for your help on this!

John

[Dibyendu Majumdar]

Hi John,

Are you able to send me a test case so that I can reproduce the issue?

Might be best to log it as an issue at https://github.com/dibyendumajumdar/simpledbm/issues.

The error you are getting is strange so I do not know offhand what might be causing it.

I am definitely interested in fixing any bugs found in SimpleDBM, although enhancements are on hold due to lack of time.

Regards

Dibyendu