MVStore files becomes unreasonable big with small amount of transaction data

[mikko]

Hi,

the enclosed zip-file H2TrMapTest1.zip includes:

1)trMapData.dat = input data file for the test
2)rootdb_big_data_ih_empty.maps = empty MVStore for the test
3)H2TrMapTest1.java = the below test program that reads input from trMapData.dat and writes output to rootdb_big_data_ih.maps

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.util.HashMap;
import java.util.Map;

import org.h2.mvstore.MVStore;
import org.h2.mvstore.db.TransactionStore;
import org.h2.mvstore.db.TransactionStore.Transaction;
import org.h2.mvstore.db.TransactionStore.TransactionMap;

public class H2TrMapTest1 {

    private static TransactionStore ts;
    private static Map<String,TransactionMap> mapNameToStoreMap;
     
    public static void main(String[] args) {

        mapNameToStoreMap = new HashMap<>();
       
        try {
            String parentDir = System.getProperty("java.io.tmpdir");
            File file = new File(parentDir, "trMapData.dat");
           
            MVStore s = openMVStoreWritable(parentDir, "rootdb_big_data_ih", 500);
            if (s==null) {
                return;
            }

            ObjectInputStream ois = new ObjectInputStream(new FileInputStream(file));
           
            TransactionStore ts = new TransactionStore(s);
            ts.init();
           
            Transaction tx = null;
            int trNumber = 0;
            try {
                while (true) {
                    Object object = ois.readObject();
                    if (object instanceof String) {
                        String trType = (String) object;
                        if (trType.equals("beginTR")) {
                            tx = ts.begin();
                            System.out.println("beginTR trNumber=" + trNumber);
                        }
                        else if (trType.equals("commitTR")) {
                            mapNameToStoreMap.clear();
                            tx.commit();
                            System.out.println("commitTR trNumber=" + trNumber++);
                        }
                    }
                    else if (object instanceof Map)  {
                        Map<Object,Object> inputMap = (Map<Object, Object>) object;
                        String mapName = (String) inputMap.get("mapName");
                        Long id = (Long) inputMap.get("key");
                        Map<Short,Object> map = (Map<Short, Object>) inputMap.get("value");
                       
                        TransactionMap<Long, Map<Short,Object>> storeMap = mapNameToStoreMap.get(mapName);
                        if (storeMap==null) {
                            storeMap = tx.openMap(mapName);
                            mapNameToStoreMap.put(mapName, storeMap);
                        }
                        storeMap.put(id, map);
                        System.out.println("id="+id+", map="+map+", mapName="+mapName);
                    }
                }
            } catch (IOException | ClassNotFoundException e) {
                ois.close();
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static MVStore openMVStoreWritable(String basePath, String storeName, Integer cacheSize) throws IOException {

        MVStore s = null;
       
        String sourceFileName = storeName + "_empty" + ".maps";
        String targetFileName = storeName + ".maps";

        String sourceFilePath = basePath + sourceFileName;
        String targetFilePath = basePath + targetFileName;
       
        Path sourcePath = Paths.get(sourceFilePath);
        Path targetPath = Paths.get(targetFilePath);
       
        Files.copy(sourcePath, targetPath, StandardCopyOption.REPLACE_EXISTING);
       
        File file_realPath = new File(targetFilePath);
        if (file_realPath.exists()) {
            s = new MVStore.Builder().
                    fileName(targetFilePath).
                    cacheSize(cacheSize).
                    autoCommitDisabled().
                    open();
        }
       
        return s;
    }
   
}

The files trMapData.dat and rootdb_big_data_ih_empty.maps are put to directory "java.io.tmpdir" for running the test program H2TrMapTest1.java.

In the beginning the test program H2TrMapTest1.java copies an empty MVStore file from rootdb_big_data_ih_empty.maps to rootdb_big_data_ih.maps which is used to write the data read from file trMapData.dat.

The data in file trMapData.dat describes transactions by:
1) "beginTR" = start a transactions
2) map, which has key value pairs:
   "mapName"=name of a TransactionMap in rootdb_big_data_ih.maps
   "key"=unique id
   "value"=Map with data to be saved to "mapName" TransactionMap by key=unique id.
3) "commitTR" = ends a transaction

Each transaction does about 15 writes to three different TransactionMaps.

The test program in H2TrMapTest1.java works correctly but the size of the result file rootdb_big_data_ih.maps grows to 577868KB bytes which is a huge size for small amount of data, i.e. it is completely unreasonable big size for saving only data of 10000 Java objects (trMapData.dat has data of 10000 Java objects).

When the 577868KB bytes rootdb_big_data_ih.maps is compacted by a command:

java -cp h2-1.4.193.jar org.h2.mvstore.MVStoreTool -compact rootdb_big_data_ih.maps

then the compacted file size is reduced to 10980KB which is a very reasonable size. Why the file size is so huge after the run of H2TrMapTest1.java?

The rapidly growing size of MVStore maps-file is a severe problem when big data is tried to save in transactions. For example, in saving 100000 objects the size grows to about 5GB and the speed of writing to the MVStore maps file becomes very slow.

Do you know any ways to prevent this unreasonable and unnecessary size growth of the MVStore file that is also harmful for the performance in writing data to the MVStore file?