memory increasing all the time when using leveldb with LRUCache in different threads

[赵琦]

I'm using leveldb with large LRUcache in multithread environment.
Db is started in the main thread. Then I started a new thread and scan some data into LRUCache in the thread using seek() and next() api. Without the former thread exit, I do repeatelly starting a new thread and loading the same data to LRUCache. The werid thing is the memory of the process never stop increasing, and far exceeded the capcity of LRUCache i had set. BTW, I had closed the mmap by modify src of leveldb. So mmap is not account for ceaseless incremental of memory usage.

Can any one tell me what happend?
I have been bothered by this problem for weeks...

[赵琦]

test code is as below. Memory usage continue to increase when starting new thread and loading data.



#include <cassert>
#include <vector>
#include <iostream>
#include <string>
#include <unistd.h>
#include <stdio.h>
#include <pthread.h>
#include "leveldb/db.h"
#include "leveldb/cache.h"

using namespace std;

leveldb::DB* pldb;
leveldb::Cache* pblock_cache = NULL;

void* loadhotThread(void* arg){
    cout<<"new thread start"<<endl;
    int cnt = 0;

    leveldb::Iterator* it = pldb->NewIterator(leveldb::ReadOptions());
    for (it->SeekToFirst();
        it->Valid();
        it->Next()) {
        it->value()[0];
        cnt++;
    }
    delete it;
    cout<<cnt<<" data load;"<<endl;

    //thread not exit
    cout<<"thread begin sleep"<<endl;
    while(1){
        sleep(10);
    }

    printf("thread exit\n");
    fflush(stdout);
    pthread_detach(pthread_self());
    return NULL;
}

int loadhotMultiThread(int argc, char **argv)
{
    int cnt = 0;
    while(cnt<10){
        pthread_t pid;
        pthread_create(&pid, NULL, loadhotThread, NULL);
        //wait the thread load data
        sleep(60);
        cnt++;
    }

    cout<<"main thread sleep"<<endl;
    while(1){
        sleep(10);
    }
    return 0;
}

int main(int argc, char **argv)
{
    if (argc != 2){
        cout<<"usage:"<<argv[0]<<" /path/to/leveldb/data/"<<endl;
        return -1;
    }
    string dir = argv[1];
    leveldb::Options options;
    //2G cache size
    uint64_t a = 2000000000;
    options.block_cache = leveldb::NewLRUCache(a);
    pblock_cache = options.block_cache;
    options.max_open_files = 65535;
    options.create_if_missing = true;
    options.block_size = 1024*32;
    options.write_buffer_size = 128000000;
    leveldb::Status s = leveldb::DB::Open(options, dir, &pldb);

    loadhotMultiThread(argc, argv);

    delete pldb;
    delete pblock_cache;
    return 0;
}

//g++ test_levelcache.cpp /data/pavelliu/extern_source/leveldb-1.14.0/libleveldb.a -o test_levelcache -I /data/pavelliu/extern_source/leveldb-1.14.0/include/ -L/data/pavelliu/local/lib/ -I/data/pavelliu/local/include/ -g -lsnappy -lpthread

[Xingxiao Zh]

Hi,

What's the size of your testing db, and how much memory increased after each full scan of db?

Btw, how about using some memory profiler, e.g. gperftools heap profiler, to identify which exactly part of code is eating your memory?

Best Regards.

--
You received this message because you are subscribed to the Google Groups "leveldb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leveldb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[赵琦]

The size of my testing db is 2.5GB. I set LRUCache size to 2GB.
And each time data was loaded, about 2GB memory increased.

haven't using any memory profiler. i'll give a try.
thanks for your advice.


在 2014年2月13日星期四UTC+8下午12时05分38秒，Xingxiao Zh写道：

[txu...@gmail.com]

what you set is the number of LRUHandle and each LRUHandle contains a pointer to one block, so the LRUCache size should be 2TB at least since the minum block size is 1KB.

在 2014年2月12日星期三 UTC+8下午2:35:03，赵琦写道：