Cache politics
21 views
Skip to first unread message
PL
Jan 7, 2016, 7:05:45 AM1/7/16
to Sparksee
Hi,
I wrote a program to load data to graph. Also I implemented program, which enable to querying the database. All it is good such as: performance, useful api :), but I have a problem. I didn't find solution for this problem in documentation. I use C++ API.
I observe in my implementation poor cache politics. Firstly, algorithm to load data construct database in ram memory and result is saved in HDD by Sparsee destructor. Secondly, algorithm to querying database work good. Problem it is because loading all data from database to RAM memory when this algorithm work. It are problems for me, because I'll plan work with data more than my RAM memory. Is possible part result save in HDD during work? Is possible get only part database from HDD during work query ?
I wrote a program to load data to graph. Also I implemented program, which enable to querying the database. All it is good such as: performance, useful api :), but I have a problem. I didn't find solution for this problem in documentation. I use C++ API.
I observe in my implementation poor cache politics. Firstly, algorithm to load data construct database in ram memory and result is saved in HDD by Sparsee destructor. Secondly, algorithm to querying database work good. Problem it is because loading all data from database to RAM memory when this algorithm work. It are problems for me, because I'll plan work with data more than my RAM memory. Is possible part result save in HDD during work? Is possible get only part database from HDD during work query ?
c3po.ac
Jan 11, 2016, 5:35:10 AM1/11/16
to Sparksee
Hi,
By default Sparksee tries to use as much memory as possible and it doesn't use the disk until it has to. That may improve the performance for certain use cases but can sometimes lead to other not desired behaviours like in your specific case.
The solution to the first issue you are encountering is enabling the recovery system. With the recovery enabled the data is periodically written to disk and a recovery log is created to guarantee the persistence. Therefore the destructor will no longer need to write much information because Sparksee already has the information on disk.
You can use the config file option:
Or The SparkseeConfig method:
About your second issue, the amount of cache memory that Sparksee uses by default is probably too much for your case. Sparksee is designed specifically to work with databases bigger than your available memory, so there is no problem in setting a lower memory limit so it will use more the disk.
You can use the config file option:
Or The SparkseeConfig method:
Best regards.
El dijous, 7 gener de 2016 13:05:45 UTC+1, PL va escriure:
By default Sparksee tries to use as much memory as possible and it doesn't use the disk until it has to. That may improve the performance for certain use cases but can sometimes lead to other not desired behaviours like in your specific case.
The solution to the first issue you are encountering is enabling the recovery system. With the recovery enabled the data is periodically written to disk and a recovery log is created to guarantee the persistence. Therefore the destructor will no longer need to write much information because Sparksee already has the information on disk.
You can use the config file option:
sparksee.io.recovery=TRUE
Or The SparkseeConfig method:
SetRecoveryEnabled( true )
About your second issue, the amount of cache memory that Sparksee uses by default is probably too much for your case. Sparksee is designed specifically to work with databases bigger than your available memory, so there is no problem in setting a lower memory limit so it will use more the disk.
You can use the config file option:
sparksee.io.cache.maxsize=SIZE_IN_MB
Or The SparkseeConfig method:
SetCacheMaxSize( SIZE_IN_MB )
Best regards.
El dijous, 7 gener de 2016 13:05:45 UTC+1, PL va escriure:
Reply all
Reply to author
Forward
0 new messages