Redis support for Persistent Memory

nicola.c...@bsc.es

unread,

Mar 11, 2016, 6:35:28 AM3/11/16

to pmem

Hi everyone,

I saw that there is a repository for an enhanced version of Redis to use NVML's libpmemlog and I would like to know if there is a plan to expand the usage of NVML to store into persistent memory not only logs but also data.

Have a nice day,
Nicola.

Krzysztof Czurylo

unread,

Mar 11, 2016, 8:49:39 AM3/11/16

to pmem

Hi,

Not only a plan - it's already done. :-)

See this pull request: https://github.com/pmem/redis/pull/3

This version is built on libpmemobj. No need to enable RDB nor AOF,

as the entire dataset is stored in PM, not in RAM.

The main dictionary (hashtable) is still in DRAM, so it must be rebuilt after

each server restart. At the moment, only simple strings are supported as values.

Due to some conflicts it's not yet merged upstream. We need to rebase it first,

and possibly do some clean up. I believe, the performance could still

be improved, as it seems that in some cases data is flushed twice for no reason.

Anyway, you can grab the code from this repo (branch "2.8"):

https://github.com/jschmieg/redis/tree/2.8

Please, let me know if you have any problems with pulling the code or with

compilation.

Regards,

Krzysztof

Christian S. Perone

unread,

Mar 11, 2016, 9:11:25 AM3/11/16

to Krzysztof Czurylo, pmem

That's an amazing work, being able to use Redis with PMEM is awesome ! Congrats.

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/38e45076-ce06-4d30-bf1e-f1502867a7ba%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Blog | Github | Twitter

"Forgive, O Lord, my little jokes on Thee, and I'll forgive Thy great big joke on me."

nicola.c...@bsc.es

unread,

Mar 12, 2016, 5:11:26 AM3/12/16

to pmem

Oh nice job!

Keep up the good work
Nicola.

sanketh nalli

unread,

Jun 21, 2016, 5:19:42 PM6/21/16

to pmem

Hi everyone,

I am trying to use redis 3.2-nvml from github.com/pmem/redis and having trouble compiling it.

After placing nvml in redis/deps/nvml, I believe nvml libs have to be first compiled since it isn't a header only

lib followed by compiling redis.

The issue is while compiling nvml. Some je_malloc routines aren't found or defined.

$ cd deps/nvml

$ make EXTRA_CFLAGS="-Wno-error"

vmem.c: In function ‘vmem_init’:

vmem.c:101:26: error: ‘je_vmem_navsnprintf’ undeclared (first use in this function)

out_set_vsnprintf_func(je_vmem_navsnprintf);

^

vmem.c:101:26: note: each undeclared identifier is reported only once for each function it appears in

vmem.c:107:3: error: ‘je_vmem_malloc_message’ undeclared (first use in this function)

je_vmem_malloc_message = print_jemalloc_messages;

^

make[1]: *** [../nondebug/libvmem/vmem.o] Error 1

make[1]: Leaving directory `Desktop/redis/deps/nvml/src/libvmem'

make: *** [libvmem] Error 2

Any suggestions ???

Krzysztof Czurylo

unread,

Jun 22, 2016, 3:04:54 AM6/22/16

to pmem

Hi,

In README file you can find the proper command to build Redis:

$ make USE_NVML=yes STD=-std=gnu99

Hope this helps,

K.

sanketh nalli

unread,

Jun 22, 2016, 11:17:49 PM6/22/16

to pmem

Thanks ! I got around it.

So is all of Redis (ie. geospatial indexes, sets etc. + any other data structures) ported on NVML or

just the key value part ? Reading the code, it looks like its just the key value codepath or am I missing something ?

Krzysztof Czurylo

unread,

Jun 23, 2016, 4:57:33 PM6/23/16

to pmem

In our prototype implementation only the strings are supported as values. Hashes, lists, sets, etc. are not supported (yet).

K.

Andrea Pellegrini

unread,

Sep 26, 2016, 6:26:00 PM9/26/16

to pmem

Hi,

I am trying your version of redis on emulated pmem as well, but I am hitting a runtime error as soon as a client connects to it (see log and backtrace below).

I am using the most recent nvml (commit 65c3da7c7e1b9ce92b5647531d4965195163cf00) and redis-pmem (commit d7e1c85dbb693fd841c91cdc714ee7c96138f812) git repos. Everything seems to be working just fine if I comment out the "pmfile" variable in the redis.config file.

The pmem partition (emulated in DRAM) is 192GB, formatted with xfs and mounted:

/dev/pmem0 on /mnt/pmem type xfs (rw,relatime,attr2,inode64,noquota)

Thanks!

-Andrea

Log:

# redis-server redis.conf

14248:C 26 Sep 17:18:57.612 * Start init Persistent memory file /mnt/pmem/redis.pm size 3.00G

14248:C 26 Sep 17:18:57.618 * Init Persistent memory file /mnt/pmem/redis.pm size 3.00G time 0.006 seconds

14248:M 26 Sep 17:18:57.619 * Increased maximum number of open files to 10032 (it was originally set to 1024).

_._

_.-``__ ''-._

_.-`` `. `_. ''-._ Redis 3.1.103_NVML (d7e1c85d/1) 64 bit

.-`` .-```. ```\/ _.,_ ''-._

( ' , .-` | `, ) Running in standalone mode

|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379

| `-._ `._ / _.-' | PID: 14248

`-._ `-._ `-./ _.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' | http://redis.io

`-._ `-._`-.__.-'_.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' |

`-._ `-._`-.__.-'_.-' _.-'

`-._ `-.__.-' _.-'

`-._ _.-'

`-.__.-'

14248:M 26 Sep 17:18:57.620 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

14248:M 26 Sep 17:18:57.620 # Server started, Redis version 3.1.103_NVML

14248:M 26 Sep 17:18:57.620 . Unrecognized RDB AUX field: 'redis-ver'

14248:M 26 Sep 17:18:57.620 . Unrecognized RDB AUX field: 'redis-bits'

14248:M 26 Sep 17:18:57.620 . Unrecognized RDB AUX field: 'ctime'

14248:M 26 Sep 17:18:57.620 . Unrecognized RDB AUX field: 'used-mem'

14248:M 26 Sep 17:18:57.620 * DB loaded from disk: 0.000 seconds

14248:M 26 Sep 17:18:57.620 * The server is now ready to accept connections on port 6379

14248:M 26 Sep 17:18:57.620 - DB 0: 1 keys (0 volatile) in 4 slots HT.

14248:M 26 Sep 17:18:57.620 - 0 clients connected (0 slaves), 762640 bytes in use

14248:M 26 Sep 17:19:00.069 - Accepted 10.118.96.81:33432

Aborted (core dumped)

GDB BT:

(gdb) bt
#0 0x00007ffff731a267 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007ffff731beca in __GI_abort () at abort.c:89
#2 0x0000000000503b22 in pmemobj_tx_free ()
#3 0x000000000043c454 in decrRefCountPM (o=o@entry=0x7ffff6ea7470) at object.c:432
#4 0x0000000000426973 in dictObjectDestructorPM (privdata=<optimized out>,
val=0x7ffff6ea7470) at server.c:473
#5 0x0000000000425bd2 in dictReplacePM (d=0x7ffff6e171e0, key=<optimized out>,
val=0x7fff003c0600) at dict.c:486
#6 0x000000000043e890 in setKeyPM (db=0x7ffff6e24800, key=key@entry=0x7ffff6e1b5a0,
val=0x7fff003c0600) at db.c:190
#7 0x0000000000448a37 in setGenericCommand (c=c@entry=0x7ffff6f41680, flags=flags@entry=0,
key=0x7ffff6e1b5a0, val=0x7ffff6ea7460, expire=expire@entry=0x0, unit=unit@entry=0,
ok_reply=0x0, abort_reply=0x0) at t_string.c:97
#8 0x0000000000448c76 in setCommand (c=0x7ffff6f41680) at t_string.c:158
#9 0x00000000004299b1 in call (c=c@entry=0x7ffff6f41680, flags=flags@entry=15)
at server.c:2265
#10 0x000000000042cc57 in processCommand (c=c@entry=0x7ffff6f41680) at server.c:2545
#11 0x000000000043a22f in processInputBuffer (c=0x7ffff6f41680) at networking.c:1296
#12 0x0000000000423347 in aeProcessEvents (eventLoop=eventLoop@entry=0x7ffff6e2e050,
flags=flags@entry=3) at ae.c:412
#13 0x00000000004236ab in aeMain (eventLoop=0x7ffff6e2e050) at ae.c:455
#14 0x0000000000420334 in main (argc=2, argv=0x7fffffffe708) at server.c:4168
(gdb)

Krzysztof Czurylo

unread,

Sep 28, 2016, 10:57:00 AM9/28/16

to pmem

Hi Andrea,

I've just compiled the version you mentioned and everything seems to work fine on my machine. However, I haven't tested it on XFS.

Did you try to run it on another filesystem? Could be just ramdisk (/dev/shm).

K.

Andrea Pellegrini

unread,

Sep 28, 2016, 3:17:55 PM9/28/16

to Krzysztof Czurylo, pmem

Hi Krzysztof,

I am having exactly the same issue when I mount the emulated pmem with ext4.

This is the kernel version I am using:

Linux 2p2650v3-1 4.4.6-ast1-00009-gfe4c1ea #3 SMP Wed Apr 6 09:19:24 CDT 2016 x86_64 x86_64 x86_64 GNU/Linux

And this is how the pmem partition is mounted:

/dev/pmem0 on /mnt/pmem type ext4 (rw,relatime,dax,data=ordered)

root@2p2650v3-1:~/redis-pm#

Is there anything else I should check?

I will try a standard (and more recent) kernel to see what happens then. I will let you know.

Thanks,

-Andrea

--
You received this message because you are subscribed to a topic in the Google Groups "pmem" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pmem/An8skSTgjO4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pmem+unsubscribe@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/c62f161f-7158-432f-b66c-61dcb53cff3d%40googlegroups.com.

Andrea Pellegrini

unread,

Sep 28, 2016, 3:29:32 PM9/28/16

to pmem, krzyszto...@intel.com

I don't know if this helps, but I am getting an error when I do "make test".

See log below.

Thanks,

-Andrea

The End

Execution time of different units:

0 seconds - unit/printver

0 seconds - unit/type/incr

1 seconds - unit/scan

1 seconds - unit/auth

1 seconds - unit/protocol

0 seconds - unit/quit

2 seconds - unit/keyspace

2 seconds - unit/multi

4 seconds - unit/type/hash

6 seconds - unit/type/set

6 seconds - unit/type/zset

6 seconds - unit/sort

1 seconds - integration/rdb

1 seconds - integration/convert-zipmap-hash-on-load

7 seconds - unit/type/string

0 seconds - integration/logging

1 seconds - unit/pubsub

8 seconds - unit/other

0 seconds - unit/introspection

1 seconds - unit/slowlog

1 seconds - unit/limits

3 seconds - integration/aof

10 seconds - unit/type/list-2

11 seconds - unit/expire

2 seconds - unit/bitops

3 seconds - unit/maxmemory

2 seconds - unit/memefficiency

6 seconds - unit/scripting

13 seconds - unit/type/list

13 seconds - integration/replication-2

15 seconds - unit/type/list-3

10 seconds - unit/hyperloglog

19 seconds - integration/replication-3

23 seconds - unit/dump

24 seconds - integration/replication-4

47 seconds - unit/obuf-limits

54 seconds - unit/aofrw

106 seconds - integration/replication-psync

158 seconds - integration/replication

!!! WARNING The following tests failed:

*** [err]: Test replication partial resync: ok psync (diskless: yes, reconnect: 1) in tests/integration/replication-psync.tcl

Expected condition '[s -1 sync_partial_ok] > 0' to be true ([s -1 sync_partial_ok] > 0)

Cleanup: may take some time... OK

Makefile:225: recipe for target 'test' failed

make[1]: *** [test] Error 1

make[1]: Leaving directory '/root/redis-pm/src'

Makefile:6: recipe for target 'test' failed

make: *** [test] Error 2

root@2p2650v3-1:~/redis-pm#

Andrea Pellegrini

unread,

Sep 28, 2016, 5:59:44 PM9/28/16

to pmem, krzyszto...@intel.com

Hi Krzysztof,

A few more info on my bug.

It is failing here:

if (server.persistent) {

oid.off = (uint64_t)o - (uint64_t)server.pm_pool;

oid.pool_uuid_lo = server.pool_uuid_lo;

pmemobj_tx_free(oid);

} else {

And these are the values of the variables involved in this routine:

(gdb) print o

$3 = (robj *) 0x7ffff6ea7470

(gdb) print *o

$4 = {type = 0, encoding = 0, lru = 15481572, refcount = 1, ptr = 0x7ffff6f35003}

(gdb) print *(server.pm_pool)

$5 = <incomplete type>

(gdb) print server.pm_pool

$6 = (PMEMobjpool *) 0x7fff00000000

(gdb) f 0

#0 0x0000000000503960 in pmemobj_tx_free ()

(gdb) print server.pool_uuid_lo

$7 = 1958689296212718681

(gdb) p/x server.pool_uuid_lo

$8 = 0x1b2ea9898e1e7459

(gdb) n

Single stepping until exit from function pmemobj_tx_free,

which has no line number information.

Program received signal SIGABRT, Aborted.

0x00007ffff731a267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55

55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

(gdb)

and this is the info for the file redis is working on:

root@2p2650v3-1:~/redis-pm/deps/nvml# pmempool info /mnt/pmem/redis.pm

POOL Header:

Signature : PMEMOBJ

Major : 2

Mandatory features : 0x0

Not mandatory features : 0x0

Forced RO : 0x0

Pool set UUID : ec7069d2-3f19-4717-a7a4-634aee629837

UUID : e89e4ed3-55c6-4e3a-a142-911aa7a03189

Previous part UUID : e89e4ed3-55c6-4e3a-a142-911aa7a03189

Next part UUID : e89e4ed3-55c6-4e3a-a142-911aa7a03189

Previous replica UUID : e89e4ed3-55c6-4e3a-a142-911aa7a03189

Next replica UUID : e89e4ed3-55c6-4e3a-a142-911aa7a03189

Creation Time : Wed Sep 28 2016 14:05:30

Alignment Descriptor : 0x000007f737777310[OK]

Class : ELF64

Data : 2's complement, little endian

Machine : AMD X86-64

Checksum : 0x7b5da60ad7be7809 [OK]

PMEM OBJ Header:

Layout : store_db

Lanes offset : 0x2000

Number of lanes : 1024

Heap offset : 0x302000

Heap size : 3218071552

Checksum : 0xc5c7889b94d3f7d8 [OK]

Root offset : 0x3c0580

Andrea Pellegrini

unread,

Sep 29, 2016, 10:29:37 PM9/29/16

to pmem, Krzysztof Czurylo

Hi Krzysztof,

It seems that there are some issues with this version of redis and Ubuntu.

When I compile it there, I get:

...

CC redis-check-rdb.o

CC geo.o

LINK redis-server

../deps/nvml/src/nondebug/libpmemobj.a(libpmemobj_all.o): In function `util_remote_unload_core':

set.c:(.text+0x18f1): undefined reference to `dlclose'

../deps/nvml/src/nondebug/libpmemobj.a(libpmemobj_all.o): In function `util_dl_check_error.part.5':

set.c:(.text+0x1ea5): undefined reference to `dlerror'

../deps/nvml/src/nondebug/libpmemobj.a(libpmemobj_all.o): In function `util_remote_load':

set.c:(.text+0x2605): undefined reference to `dlopen'

set.c:(.text+0x2627): undefined reference to `dlsym'

set.c:(.text+0x2646): undefined reference to `dlsym'

set.c:(.text+0x2665): undefined reference to `dlsym'

set.c:(.text+0x2684): undefined reference to `dlsym'

set.c:(.text+0x26a6): undefined reference to `dlsym'

collect2: error: ld returned 1 exit status

Makefile:185: recipe for target 'redis-server' failed

make[1]: *** [redis-server] Error 1

make[1]: Leaving directory '/root/redis-pm2/src'

Makefile:6: recipe for target 'all' failed

make: *** [all] Error 2

root@2p2650v3-1:~/redis-pm2#

While everything compiles fine on CentOS (I eventually managed to compile it under Ubuntu, but it does not seem very stable). However, Redis with PMEM in CentOS aborts (tmpfs on /mnt/pmem type tmpfs (rw,relatime,size=4194304k)):

[root@diablo-1 redis-pmem]# redis-server redis.conf

3894:C 29 Sep 21:15:47.894 * Start init Persistent memory file /mnt/pmem/redis.pm size 3.00G

3894:C 29 Sep 21:15:47.900 * Init Persistent memory file /mnt/pmem/redis.pm size 3.00G time 0.006 seconds

3894:M 29 Sep 21:15:47.901 * Increased maximum number of open files to 10032 (it was originally set to 1024).

_._

_.-``__ ''-._

_.-`` `. `_. ''-._ Redis 3.1.103_NVML (d7e1c85d/0) 64 bit

.-`` .-```. ```\/ _.,_ ''-._

( ' , .-` | `, ) Running in standalone mode

|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379

| `-._ `._ / _.-' | PID: 3894

`-._ `-._ `-./ _.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' | http://redis.io

`-._ `-._`-.__.-'_.-' _.-'

|`-._`-._ `-.__.-' _.-'_.-'|

| `-._`-._ _.-'_.-' |

`-._ `-._`-.__.-'_.-' _.-'

`-._ `-.__.-' _.-'

`-._ _.-'

`-.__.-'

3894:M 29 Sep 21:15:47.903 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

3894:M 29 Sep 21:15:47.903 # Server started, Redis version 3.1.103_NVML

3894:M 29 Sep 21:15:47.903 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

3894:M 29 Sep 21:15:47.903 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

3894:M 29 Sep 21:15:47.965 * DB loaded from disk: 0.061 seconds

3894:M 29 Sep 21:15:47.965 * The server is now ready to accept connections on port 6379

Aborted (core dumped)

[root@diablo-1 redis-pmem]#

Could you please share your set up (at least OS version and how you mount the pmem) so I could reproduce your environment and get redis to work?

Thanks!

-Andrea

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/5a915f8a-dd9c-48c0-83d0-20e95957f270%40googlegroups.com.

Hu Wan

unread,

Oct 2, 2016, 10:15:57 AM10/2/16

to pmem, krzyszto...@intel.com

This is because pmemobj_tx_free() cantnot be called outside of a transaction. dictGenericDelete() and _dictClear() in redis/src/dict.c call pmemobj_tx_free(), but there are no transactions.

Here is an example of pmemobj_tx_free() being used correctly:

 redis/src/server.c
 486 #ifdef USE_NVML
 487 void dictSdsDestructorPM(void *privdata, void *val)
 488 {
 489     DICT_NOTUSED(privdata);
 490     /* TODO: TX_BEGIN() */
 491     pmemobj_tx_begin(server.pm_pool, NULL, TX_LOCK_NONE);
 492     sdsfreePM(val); // sdsfreePM(val) calls pmemobj_tx_free().
 493     pmemobj_tx_commit();
 494     pmemobj_tx_end();
 495 }
 496 #endif

--

Hu Wan

在 2016年9月29日星期四 UTC+8上午5:59:44，Andrea Pellegrini写道：

To unsubscribe from this group and all its topics, send an email to pmem+uns...@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

Andrea Pellegrini

unread,

Oct 3, 2016, 11:31:17 AM10/3/16

to Hu Wan, pmem, Krzysztof Czurylo

Thank you, Hu.

I just applied the extra code you suggested to isolate transactions before deleting them (see code below for an example), and redis still crashes (back trace below as well).

Did you get it to run redis w/ pmem properly? Any other changes you had to make?

It seems that redis does not crash if I disable all calls to pmemobj_tx_free.

Thanks!

-Andrea

#ifdef USE_NVML

/* Free an sds string. No operation is performed if 's' is NULL. */

void sdsfreePM(sds s) {

PMEMoid oid;

if (s == NULL) return;

if (server.persistent) {

oid.off = (uint64_t)((char*)s-sdsHdrSize(s[-1])) - (uint64_t)server.pm_pool;

oid.pool_uuid_lo = server.pool_uuid_lo;

pmemobj_tx_begin(server.pm_pool, NULL, TX_LOCK_NONE);

pmemobj_tx_free(oid);

pmemobj_tx_commit();

pmemobj_tx_end();

} else {

s_free((char*)s-sdsHdrSize(s[-1]));

}

#endif

#0 0x00007ffff732c5f7 in raise () from /lib64/libc.so.6

#1 0x00007ffff732dce8 in abort () from /lib64/libc.so.6

#2 0x00000000004ec442 in pmemobj_tx_free ()

#3 0x000000000042e660 in sdsfreePM (s=<optimized out>) at sds.c:253

#4 0x000000000042706e in dictSdsDestructorPM (privdata=<optimized out>, val=0x7ffff08156d1) at server.c:492

#5 0x0000000000425c90 in dictGenericDelete (d=0x7ffff08171e0, key=0x7fffeee70213, nofree=nofree@entry=0) at dict.c:529

#6 0x0000000000426127 in dictGenericDelete (nofree=0, key=<optimized out>, d=<optimized out>) at dict.c:559

#7 dictDelete (ht=<optimized out>, key=<optimized out>) at dict.c:558

#8 0x000000000043d7fb in dbDelete (db=<optimized out>, key=0x7fffeee70200) at db.c:233

#9 0x0000000000453fdb in hdelCommand (c=0x7fffeee72340) at t_hash.c:685

#10 0x0000000000429ce5 in call (c=c@entry=0x7fffeee72340, flags=flags@entry=15) at server.c:2265

#11 0x000000000042cd2f in processCommand (c=c@entry=0x7fffeee72340) at server.c:2545

#12 0x0000000000439af7 in processInputBuffer (c=0x7fffeee72340) at networking.c:1296

#13 0x0000000000424170 in aeProcessEvents (eventLoop=eventLoop@entry=0x7ffff082e050, flags=flags@entry=3) at ae.c:412

#14 0x00000000004243ab in aeMain (eventLoop=0x7ffff082e050) at ae.c:455

#15 0x0000000000421304 in main (argc=2, argv=0x7fffffffe678) at server.c:4168

To unsubscribe from this group and all its topics, send an email to pmem+unsubscribe@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/18cf245b-64e5-4267-abca-4cf3ddf2f825%40googlegroups.com.

Hu Wan

unread,

Oct 4, 2016, 10:14:20 PM10/4/16

to pmem, wan...@gmail.com, krzyszto...@intel.com

Hi Andrea,

For simplicity's sake, I chose to add transactions directly to pmemobj_tx_free() in src/dict.c. It seemed to work. But i'm not sure if it's correct.

Here is the diff output:

diff --git a/src/dict.c b/src/dict.c

index d3cd47f..2206858 100644

--- a/src/dict.c

+++ b/src/dict.c

@@ -533,7 +533,10 @@ static int dictGenericDelete(dict *d, const void *key, int nofree)

if (server.persistent) {

oid.off = (uint64_t)he - (uint64_t)server.pm_pool;

oid.pool_uuid_lo = server.pool_uuid_lo;

+ pmemobj_tx_begin(server.pm_pool, NULL, TX_LOCK_NONE);

pmemobj_tx_free(oid);

+ pmemobj_tx_commit();

+ pmemobj_tx_end();

} else {

zfree(he);

}

@@ -581,7 +584,10 @@ int _dictClear(dict *d, dictht *ht, void(callback)(void *)) {

if (server.persistent) {

oid.off = (uint64_t)he - (uint64_t)server.pm_pool;

oid.pool_uuid_lo = server.pool_uuid_lo;

+ pmemobj_tx_begin(server.pm_pool, NULL, TX_LOCK_NONE);

pmemobj_tx_free(oid);

+ pmemobj_tx_commit();

+ pmemobj_tx_end();

} else {

zfree(he);

}

Also, you can add NVML_DEBUG=yes option to build Redis, and you'll see more error details.

$ make USE_NVML=yes NVML_DEBUG=yes STD=-std=gnu99

Hope this helps.

Hu Wan.

Krzysztof Czurylo

unread,

Oct 5, 2016, 10:54:54 AM10/5/16

to pmem, wan...@gmail.com, krzyszto...@intel.com

Well, I'm almost sure it's not correct. Basically, it would not guarantee the atomicity k/v pair deletion - it may happen that in case of a failure, an obsoleted value is left or a value w/o a key.

I believe, pmemobj_tx_begin() should be added somewhere in processCommand() (around the command execution).

K.

Andrea Pellegrini

unread,

Oct 5, 2016, 10:57:51 AM10/5/16

to Hu Wan, pmem, Krzysztof Czurylo

Hi Hu Wan,

Below is the diff of my redis-pmem repo -- file src/dict.c matches exactly with your suggestion.

Same result...

Thanks,

-Andrea

Client:

# redis-cli -h 10.118.96.42

10.118.96.42:6379> HSET myhash field1 "Hello"

(integer) 1

10.118.96.42:6379> HGET myhash field1

"Hello"

10.118.96.42:6379> HDEL myhash field1

Could not connect to Redis at 10.118.96.42:6379: Connection refused

not connected> quit

Server:

(gdb) bt

#0 0x00007ffff732c5f7 in raise () from /lib64/libc.so.6

#1 0x00007ffff732dce8 in abort () from /lib64/libc.so.6

#2 0x00000000004dd4b4 in out_fatal (file=0x527ba0 "tx.c", line=1853, func=0x528740 <__func__.4606> "pmemobj_tx_free", fmt=0x527c7a "assertion failure: %s")

at ../../src/../src/common/out.c:541

#3 0x00000000004fe887 in pmemobj_tx_free (oid=...) at tx.c:1853

#4 0x000000000042e320 in sdsfreePM (s=<optimized out>) at sds.c:253

#5 0x0000000000426d2e in dictSdsDestructorPM (privdata=<optimized out>, val=0x7ffff08156d1) at server.c:492

#6 0x0000000000425950 in dictGenericDelete (d=0x7ffff08171e0, key=0x7fffeee70213, nofree=nofree@entry=0) at dict.c:529

#7 0x0000000000425de7 in dictGenericDelete (nofree=0, key=<optimized out>, d=<optimized out>) at dict.c:559

#8 dictDelete (ht=<optimized out>, key=<optimized out>) at dict.c:558

#9 0x000000000043d4bb in dbDelete (db=<optimized out>, key=0x7fffeee70200) at db.c:233

#10 0x0000000000453c9b in hdelCommand (c=0x7fffeee72340) at t_hash.c:685

#11 0x00000000004299a5 in call (c=c@entry=0x7fffeee72340, flags=flags@entry=15) at server.c:2265

#12 0x000000000042c9ef in processCommand (c=c@entry=0x7fffeee72340) at server.c:2545

#13 0x00000000004397b7 in processInputBuffer (c=0x7fffeee72340) at networking.c:1296

#14 0x0000000000423e30 in aeProcessEvents (eventLoop=eventLoop@entry=0x7ffff082e050, flags=flags@entry=3) at ae.c:412

#15 0x000000000042406b in aeMain (eventLoop=0x7ffff082e050) at ae.c:455

#16 0x00000000004210a4 in main (argc=2, argv=0x7fffffffe678) at server.c:4168

(gdb)

# git diff

diff --git a/redis.conf b/redis.conf

index 5eb669d..b723bef 100644

--- a/redis.conf

+++ b/redis.conf

@@ -58,7 +58,8 @@

# IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES

# JUST COMMENT THE FOLLOWING LINE.

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-bind 127.0.0.1

+#bind 127.0.0.1

+bind 10.118.96.42

# Protected mode is a layer of security protection, in order to avoid that

# Redis instances left open on the internet are accessed and exploited.

@@ -190,7 +191,7 @@ databases 16

# If the file does not exists, it will be created with given size. Otherwise,

# the size is ignored.

#

-# pmfile /mnt/pmem/redis.pm 3gb

+pmfile /mnt/pmem/redis.pm 15gb

################################ SNAPSHOTTING ################################

#

diff --git a/src/dict.c b/src/dict.c

index d3cd47f..2206858 100644

--- a/src/dict.c

+++ b/src/dict.c

@@ -533,7 +533,10 @@ static int dictGenericDelete(dict *d, const void *key, int nofree)

if (server.persistent) {

oid.off = (uint64_t)he - (uint64_t)server.pm_pool;

oid.pool_uuid_lo = server.pool_uuid_lo;

+ pmemobj_tx_begin(server.pm_pool, NULL, TX_LOCK_NONE);

pmemobj_tx_free(oid);

+ pmemobj_tx_commit();

+ pmemobj_tx_end();

} else {

zfree(he);

}

@@ -581,7 +584,10 @@ int _dictClear(dict *d, dictht *ht, void(callback)(void *)) {

if (server.persistent) {

oid.off = (uint64_t)he - (uint64_t)server.pm_pool;

oid.pool_uuid_lo = server.pool_uuid_lo;

+ pmemobj_tx_begin(server.pm_pool, NULL, TX_LOCK_NONE);

pmemobj_tx_free(oid);

+ pmemobj_tx_commit();

+ pmemobj_tx_end();

} else {

zfree(he);

}

diff --git a/src/object.c b/src/object.c

index 8ccc974..0a19944 100644

--- a/src/object.c

+++ b/src/object.c

@@ -429,7 +429,10 @@ void decrRefCountPM(robj *o) {

if (server.persistent) {

oid.off = (uint64_t)o - (uint64_t)server.pm_pool;

oid.pool_uuid_lo = server.pool_uuid_lo;

+ pmemobj_tx_begin(server.pm_pool, NULL, TX_LOCK_NONE);

pmemobj_tx_free(oid);

+ pmemobj_tx_commit();

+ pmemobj_tx_end();

} else {

zfree(o);

}

diff --git a/src/sds.c b/src/sds.c

index ac3ec53..a056de9 100644

--- a/src/sds.c

+++ b/src/sds.c

@@ -249,7 +249,10 @@ void sdsfreePM(sds s) {

if (server.persistent) {

oid.off = (uint64_t)((char*)s-sdsHdrSize(s[-1])) - (uint64_t)server.pm_pool;

oid.pool_uuid_lo = server.pool_uuid_lo;

+ pmemobj_tx_begin(server.pm_pool, NULL, TX_LOCK_NONE);

pmemobj_tx_free(oid);

+ pmemobj_tx_commit();

+ pmemobj_tx_end();

} else {

s_free((char*)s-sdsHdrSize(s[-1]));

}

To unsubscribe from this group and all its topics, send an email to pmem+unsubscribe@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/d5028ee1-9646-480b-96c0-0fea7afb3bb8%40googlegroups.com.

Andrea Pellegrini

unread,

Oct 5, 2016, 11:03:41 AM10/5/16

to Krzysztof Czurylo, pmem, Hu Wan

Hi Krzysztof,

Were you able to reproduce the issue I am experiencing?

Thanks,

-Andrea

To unsubscribe from this group and all its topics, send an email to pmem+unsubscribe@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/e6f6b743-212a-4eea-942e-73cf639e6a92%40googlegroups.com.

Krzysztof Czurylo

unread,

Oct 5, 2016, 11:18:26 AM10/5/16

to pmem, krzyszto...@intel.com, wan...@gmail.com

Hi Andrea,

Sorry, I forgot to mention that.

Yes, I was able to reproduce it with redis-cli. It's enough to create an new K/V using SET and then delete it with DEL, like this:

> SET xxx 123

> DEL xxx

K.

Andrea Pellegrini

unread,

Oct 5, 2016, 11:33:50 AM10/5/16

to Krzysztof Czurylo, pmem, Hu Wan

Ok, cool.

Please let me know if there is anything I can help with on this.

Thanks,

-Andrea

To unsubscribe from this group and all its topics, send an email to pmem+unsubscribe@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/7eb54233-a7b5-475f-99bf-9f8965ca1a7a%40googlegroups.com.

Krzysztof Czurylo

unread,

Oct 6, 2016, 3:09:09 AM10/6/16

to pmem, krzyszto...@intel.com, wan...@gmail.com

Well, adding the tx begin/end in processCommand() helps for DEL, but not for HDEL.

Basically, this is because our prototype PM-aware Redis is missing support for any Redis data types other than string. So, LIST, SET, HSET, etc. are simply not supported.

It's probably not a big deal to add the support for all the data types, but it's definitely not a 5 min. task.

K.

Andrea Pellegrini

unread,

Oct 6, 2016, 11:15:08 AM10/6/16

to pmem, krzyszto...@intel.com, wan...@gmail.com

Hi Krzysztof,

Got it.

How do you guys benchmark pmem performance on some non-trivial benchmark? I was hoping that redis would be a good application to do that, but it seems that it's not quite ready.

I might be available to make the changes required to actually benchmark this workload (i.e. with memtier), do you have any suggestions/directions on what changes you think might be required to make this happen?

Thanks,

-Andrea

Andrea Pellegrini

unread,

Oct 12, 2016, 2:03:43 PM10/12/16

to pmem, Krzysztof Czurylo, Hu Wan

Hi Krzysztof,

Do you have any inputs on this?

Thanks,

-Andrea

To unsubscribe from this group and all its topics, send an email to pmem+unsubscribe@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/ed80bd2b-97a3-43b2-aecf-e87a5415ce3d%40googlegroups.com.

Krzysztof Czurylo

unread,

Oct 13, 2016, 3:14:53 PM10/13/16

to pmem, krzyszto...@intel.com, wan...@gmail.com

Hi Andrea,

I was pondering on what would be the best approach to implement PMEM-support for all the Redis types. For me, duplicating all the functions, like it is done in the latest prototype, doesn't seem to be the best option.

Basically, We need to identify all the calls to zmalloc/zrealloc/.../zfree that are related to key/value allocations and replace them with the calls to pmemobj allocator routines.

However, this is not that easy, as these functions are used i.e. in createObject() and dictCreate(), and that ones can be used both to allocate some server runtime data, as well as user data objects.

Originally (in the first prototype of PMEM-Redis), we added additional flag to almost each and every function to indicate whether the allocation should be done from PMEM or not, depending on the configuration and current context. I don't like it much, but maybe this is the best what we can do.

I'm also thinking about adding an attribute to each Redis data structure that would indicate whether this struct is a part of a data object or the runtime state. Based on that, we could make a decision whether to use zmalloc/zfree or pmemobj_tx_alloc/free to allocate/free the memory it resides in. Also, since this flag would be a part of an object, there would be no need to pass it down as an additional argument to each and every function on the call stack - it must be specified only when the object is created. So, "only" for createObject/dictCreate/zslCreate/ziplistNew/.../sdsnewlen, but not for their "free" counterparts.

K.

Andrea Pellegrini

unread,

Nov 2, 2016, 6:55:44 PM11/2/16

to Krzysztof Czurylo, pmem

Hi Krzysztof,

I spent some time thinking about how we could handle this.

I also tried to measure how many objects are created/destroyed during a simple run, and it seems that Redis does create quite a few just to manage the commands, logs, and so forth -- I was maybe thinking that if these were few enough I could just put everything in pmem and call it good.

I agree with you, we need to manage the memory allocation/free differently depending on whether we are working on objects in pmem or in dram, possibly without replicating a ton of code.

It seems that only some dictionaries should be placed in pmem, and your code already defines them:

for (j = 0; j < server.dbnum; j++) {

#ifdef USE_NVML

if (server.persistent)

server.db[j].dict = dictCreate(&dbDictTypePM,NULL);

else

#endif

server.db[j].dict = dictCreate(&dbDictType,NULL);

server.db[j].expires = dictCreate(&keyptrDictType,NULL);

server.db[j].blocking_keys = dictCreate(&keylistDictType,NULL);

server.db[j].ready_keys = dictCreate(&setDictType,NULL);

server.db[j].watched_keys = dictCreate(&keylistDictType,NULL);

...

}

server.pubsub_channels = dictCreate(&keylistDictType,NULL);

It's the first time I poke at this code in detail, and it seems that it makes sense to add a flag to the dictionary that will be stored in the pmem, in such a way that all the robj generated for that dictionary could go to the pmem instead of on dram.

Additionally, we probably want to add a flag to the redis object (robj) to indicate whether it is stored into pmem or in dram.

Once we carry that information around, we should be able to invoke the right function when we need to invoke malloc/free. That said, there are still functions we will need to change.

For instance, the code that creates a new object (such as a list, shown below) is invoked with no argument, hence we will likely need to add a flag to tell us whether this is a pmem object or not -- with the assumptions above we would need it to populate correctly robj.

robj *createQuicklistObject(void) {

quicklist *l = quicklistCreate();

robj *o = createObject(OBJ_LIST,l);

o->encoding = OBJ_ENCODING_QUICKLIST;

return o;

}

Sounds like you folks played with this for some time... Any thoughts? Do you see any issues in taking this approach?

Thanks,

-Andrea

To unsubscribe from this group and all its topics, send an email to pmem+unsubscribe@googlegroups.com.

To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/d493a2b4-f453-411d-9ef6-56bc2c880c8c%40googlegroups.com.

Reply all

Reply to author

Forward