How to ensure that DiskPersistence is closed safely

[Amir Eslampanah]

Hey,

I'm currently using CQEngine in testing as a datastore for storing/indexing/query API logs.

I noticed that if the JVM process is killed its possible for the underlying SQLite .dat file to be corrupted such that upon the startup again the adding of an index can fail.

My question is how do we ensure that the DiskPersistence is safely closed for shutdown scenarios?

Is calling this on the persistence object sufficient?

persistence.close();

[Amir Eslampanah]

Also, to add to this; I've noticed that

persistence.compact();

On a disk-persistence seems to wipe all records?

Is this method not intended to be used?

How do I compact the underlying database then?

To add some context I've got a runnable task that runs every 3 hours to do pruning on certain filters (I've tested this successfully with shorter intervals fine without compact being called and it didn't wipe everything).

persistence = DiskPersistence.onPrimaryKeyInFileWithProperties((SimpleAttribute<LogMessage, String>) LogMessage.MESSAGE_GUID_ATTRIB, persistenceFile, SQLITE_PROPERTIES);
logMessages = new ConcurrentIndexedCollection<>(persistence);

ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1);

// Prune the logs on a regular basis
// <= DEBUG = 72 Hours Pruning
// = INFO = 7 Days Pruning
// <= ERROR = 30 Days Pruning
Runnable pruneLogsTask = () -> {
LogMessageFilter debugFilter = new LogMessageFilter.Builder().timestampBefore(Timestamp.from(Instant.now().minus(72, ChronoUnit.HOURS))).level(LEVEL.DEBUG.ordinal()).build();
LogMessageFilter infoFilter = new LogMessageFilter.Builder().timestampBefore(Timestamp.from(Instant.now().minus(7, ChronoUnit.DAYS))).level(LEVEL.INFO.ordinal()).build();
LogMessageFilter errorFilter = new LogMessageFilter.Builder().timestampBefore(Timestamp.from(Instant.now().minus(30, ChronoUnit.DAYS))).level(LEVEL.ERROR.ordinal()).build();

Query<LogMessage> debugQuery = debugFilter.getAsQuery(true);
Query<LogMessage> infoQuery = infoFilter.getAsQuery(true);
Query<LogMessage> errorQuery = errorFilter.getAsQuery(true);

this.pruneByQuery(debugQuery);
this.pruneByQuery(infoQuery);
this.pruneByQuery(errorQuery);
persistence.compact();
};

ScheduledFuture<?> scheduledFuture = executorService.scheduleAtFixedRate(pruneLogsTask, 1, 3, TimeUnit.HOURS);