Support for big data set?
88 views
Skip to first unread message
harry liu
Feb 4, 2016, 2:04:40 AM2/4/16
to pmem, Dong Li
Hello, everyone. Thanks for viewing this topic.
Currently, I'm working on testing some benchmarks with pmem library. However, once the data set is big, the whole system will crush. More specifically, got a core dumped message.
For an instance, there is a string copy example in ./src/example/libpmemobj/write.c. The original size of the string is just 10 (#define MAX_BUF_LEN 10;), and the system works just fine. However, if I changed the length to more than 100*1024*1024 (#define MAX_BUF_LEN 100*1024*1024 //100mb), the system crushes after trying to map the memory.
Does anybody come with this situation? If someone can explain the reason for me I'll be so grateful!
-Wei
Piotr Balcer
Feb 4, 2016, 4:42:27 AM2/4/16
to pmem, dl...@ucmerced.edu
Hi Harry,
The string store example is not a good base for a benchmark. For starters, it uses standard input for data, and even if you manage to make it work properly all you will be benchmarking is the speed of memcpy operation.
The reason the example crashes after changing the MAX_BUF_LEN variable to 100mb is because there's a local array (allocated from stack) in the main functions of both writer.c and reader.c that uses this define as it's size. You are simply overrunning the application stack.
Take a look at our benchmarks (/src/benchmarks/) where we developed quite a comprehensive set of performance measuring tools.
Piotrek
The string store example is not a good base for a benchmark. For starters, it uses standard input for data, and even if you manage to make it work properly all you will be benchmarking is the speed of memcpy operation.
The reason the example crashes after changing the MAX_BUF_LEN variable to 100mb is because there's a local array (allocated from stack) in the main functions of both writer.c and reader.c that uses this define as it's size. You are simply overrunning the application stack.
Take a look at our benchmarks (/src/benchmarks/) where we developed quite a comprehensive set of performance measuring tools.
Piotrek
harry liu
Feb 5, 2016, 6:48:55 PM2/5/16
to pmem, dl...@ucmerced.edu
Hi Piotrek,
I'm so glad you answered my question!
Actually, I didn't treat the write.c so seriously. I'm working on using CG benchmark to test the performance. However, As I said I got segment fault because the data is relatively big. But even I set the size of stack enough big. The crush is still there. I'm very new about programming here. Could you please explain more?
Thank you very much
Harry
Piotr Balcer
Feb 8, 2016, 9:10:13 AM2/8/16
to pmem, dl...@ucmerced.edu
Hi,
If I had to guess pmemobj_root(...) returns OID_NULL because the pool is not big enough to allocate 100 megabyte structure. So you are doing a NULL dereference.
As I said - this is a blind alley. There's no point in modifying this example. Please read the tutorial series on pmem.io which I hope will clear some thing for you.
Piotrek
If I had to guess pmemobj_root(...) returns OID_NULL because the pool is not big enough to allocate 100 megabyte structure. So you are doing a NULL dereference.
As I said - this is a blind alley. There's no point in modifying this example. Please read the tutorial series on pmem.io which I hope will clear some thing for you.
Piotrek
Reply all
Reply to author
Forward
0 new messages