running redis on emulated nvdimm

[muneendra kumar]

Hi ,

I wanted to run Redis on emulated NVDIMM.

And  i used the below link to build and  test the same.

https://github.com/pmem/pmem-redis

When i try to run the below test ,specified in the doc i am getting the below error

 ./redis-server --nvm-maxcapacity 1 --nvm-dir /mnt/pmem0 --nvm-threshold 64

*** FATAL CONFIG FILE ERROR ***

Reading the configuration file, at line 2

>>> 'nvm-maxcapacity "1"'

Bad directive or wrong number of arguments

Any guidance here will help me a lot.

Regards,

Muneendra.

[Steve Scargall]

Hello Muneendra,

How did you build the source?  To get NVM support, you need to compile/make it using "make USE_NVM=yes" which automatically pulls down PMDK and dependencies, then compiles it.  PMDK has several dependencies which you may be missing.  Instructions for installing pre-requisites can be found at https://docs.pmem.io/getting-started-guide/installing-pmdk/compiling-pmdk-from-source

If PMDK is being used or installed in a different location to the default, you may also need to update your PKG_CONFIG_PATH and LD_LIBRARY_PATH to something like this:

$ export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig:/lib64/pkgconfig

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/local/lib64

HTH

    Steve

[Peifeng Si]

Hello Muneendra,

As Steve said, you need to compile the source code using "make USE_NVM=yes", otherwise some newly added options can't be parsed by redis-server and then you will get the error message "Bad directive or wrong number of arguments".

Thanks,

Peifeng

在 2018年12月5日星期三 UTC+8下午7:56:03，muneendra kumar写道：

[muneendra kumar]

Hi Peifeng,

Thanks for the reply and i recompile the code with make USE_NVM=yes and it worked fine.

And when i run the below command i got the o/p saying  "Ready to accept connections"

src/redis-server --nvm-maxcapacity 1 --nvm-dir /mnt/mem --nvm-threshold 64

Is there any standard test which gives me the benchmark with pmem  device v's scsi device on redis.

i ran src/redis-benchmark -h 127.0.0.1 -p 6379 -t set,lpush -n 100000 -q  -d 20 which gave me the below o/p

SET: 82781.46 requests per second

LPUSH: 95602.30 requests per second

will this test  be sufficient ?

Regards,

Muneendra.

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/c60c21bc-56b9-47d7-9680-08a0f525284a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peifeng Si]

Hi Muneendra,

For a basic performance comparison, I think set and lpush is ok. 

And here are two things you need to notice:

1. Since you start redis-server with "--nvm-threshold 64" which means only value size larger than 64 bytes will be stored in NVM, hence, you need to specify a large value in redis-benchmark options otherwise nothing will be stored in NVM.

e.g.,  redis-benchmark -d 1024 .......

2. You'd better specify the keyspace length in redis-benchmark options. if not, all set and lpush requests will use the same key.

e.g.,  redis-benchmark -r 100000 ....

Thanks,

Peifeng

在 2018年12月6日星期四 UTC+8下午2:27:34，muneendra kumar写道：

[muneendra kumar]

Hi Peifeng,

Thanks for the input and i tried to test the same on pmem device mounting with and without dax options.

But surprisingly i din't  see any major performance difference between the two (at a max of 4k requests difference),As mounting with dax should bypass the cache.

Not sure this is the expected result or iam missing something.

Below are the steps which i followed.

For Mounted with DAX;

 mkfs.ext4 -F /dev/pmem0

 mount -o dax /dev/pmem0 /mnt/pmem4

./src/redis-server --nvm-maxcapacity 1 --nvm-dir /mnt/pmem4 --nvm-threshold 64

./redis-benchmark -h 127.0.0.1 -p 6379 -t set,lpush -n 100000 -q  -d 1024 -r 100000

And i tested for variable sizes from 1024 to 64k (-d 1024,... -d 65536)

For mounted without dax

mkfs.ext4 -F /dev/pmem0

 mount  /dev/pmem0 /mnt/pmem4

./src/redis-server --nvm-maxcapacity 1 --nvm-dir /mnt/pmem4 --nvm-threshold 64

./redis-benchmark -h 127.0.0.1 -p 6379 -t set,lpush -n 100000 -q  -d 1024 -r 100000

And i tested for variable sizes from 1024 to 64k (-d 1024,... -d 65536)

Attached the test results.

Could you please let me know if iam doing anything wrong ?

Regards,
Muneendra.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/21a5a201-c3cc-420a-b067-6eca33db0cec%40googlegroups.com.

[Andy Rudoff]

Hi Muneendra,

I believe you're still using emulated pmem, so let's look at the difference between your two runs.  Peifeng can confirm, but I think the configuration you ran is using pmem as additional volatile capacity, using it to hold values greater than 64 in size.  That means that in your test runs, redis will use malloc (actually jemalloc) for most memory allocation, which of course just asks the system for DRAM, and when storing a value greater than 64, it will use memory obtained by mapping a file on pmem (which for you is also DRAM).  If you use DAX, that's the end of the story, but if you don't use DAX, the mapped file will be cached in the page cache (which is also DRAM).  So you're comparing slightly different code paths of accessing things in DRAM.  Your results are very close, but perhaps without DAX they are a little worse because you're measuring the impact of copying data into the page cache, where it will have the same access performance after that.

You might be wondering what's the point?  In fact, since Optane media is said to be slower than DRAM, you might wonder why use these changes?  Won't the result actually slow down compared to DRAM?  There are two reasons:

- The volatile use case: Optane DC persistent memory is higher capacity than DRAM, up to 3TB per socket, and is expected to be cheaper.  For large memory use cases where applications have huge datasets in DRAM (like Redis using DRAM to hold a large in-memory database), using DCPMM for additional volatile capacity means you can build the system cheaper (less DRAM) or free up some DRAM for other things.  In this case, you're replacing the faster DRAM capacity with the slower-but-cheaper pmem capacity, so you expect a slowdown, but we find many use cases where the slowdown is not significant so the application still meets the needs of the use case while lowering the total cost of ownership for the solution.

- The persistent use case: Optane DC persistent memory is slower than DRAM, but it is significantly faster than storage and it is persistent.  If you replace some of the storage in a use case with pmem, those accesses to persistence will run faster.  For Redis, there are a few places where it accesses persistent media, and the "append-only file" (AOF) is one such case.  If you have a use case where Redis is bottlenecked on the AOF writes, moving the AOF to pmem will improve performance.

in summary: sometimes we use pmem to get better performance, sometimes we use it to lower TCO (and sometimes both).

Hope that helps,

-andy

To unsubscribe from this group and stop receiving emails from it, send an email to pmem+unsubscribe@googlegroups.com.

[muneendra kumar]

Hi Andy,

Thanks for the info.

One quick question in without dax option even though the redis uses malloc for memory allocation(which gets it from DRAM),

at the time of persistent operation the data needs to be pushed from cache to backend storage device.

In this case there will be 2 copies involved one to the page cache and other from page cache to backend device (emulated PMEM (DRAM)).

In dax case we are avoiding one copy so we should see a very minimal performance improvement which we are seeing now.

Is my understanding correct?

Is there any test case available in the git hub where i can really see the performance improvement as you mentioned above?

On the same test can i replace a pmem device with scsi device to check the performance.

Theoretically it  still uses the page cache(DRAM) for memory allocation.

The only difference should be the time it took to write into the backend device(scsi).

I need a test case which validates this theory .

If any of such test case is available please let me know so that i can test the same.

Regards,

Muneendra.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/21a5a201-c3cc-420a-b067-6eca33db0cec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/fbe6d5b3-b568-47fa-b43d-3aa6f9c1cd18%40googlegroups.com.

[muneendra kumar]

Hi Andy,

I was going through couple of  slides presented by you and Peifeng where you mentioned the different use cases as below

Volatile Usage (libmemkind):

     Persistent memory used for capacity

     Don’t care it is persistent, just another tier, high capacity

Persistent Usage (libpmemobj):

    Redis preserves the data

    Most data (95%) stored in pmem, rest in DRAM

    Memory already “warm” on startup

Are the below  links provides the corresponding code for volatile Usage and Persistent Usage ?

Persistent Usage (libpmemobj):   https://github.com/pmem/redis/tree/3.2-nvml

Volatile Usage (libmemkind): https://github.com/pmem/pmem-redis 

And iam using the code of https://github.com/pmem/pmem-redis for my current testing.

If i use the other link (https://github.com/pmem/redis/tree/3.2-nvml) can i see any performance improvement as most data is stored in pmem?

Regards,

Muneendra.

[muneendra kumar]

Hi Andy/ Peifeng  ,

Can you just confirm whether my understanding is correct or not ?

Regards,

Muneendra.

[Zhiming Li]

pmem-redis covers both volatile (memkind) and persistent (libpmem) usage.

The persistency is achieved using AOF stored in SSD.  The "Pointer Based AOF" option allow you to leverage the persistency nature of NVM to reduce the performance overhead of AOF during write operation.

Zhiming