redis is not responding

[Jonathan Leibiusky]

Hi! I have a redis server in production that just stopped responding. Even restarting doesn't help, since after some time it will stop answering.
I can't send an info command because it won't return anything. The last entry in the output is:
[18270] 24 Jan 08:41:21 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

The configuration of this redis is:

daemonize yes

databases 16

pidfile /tmp/redis.pid
port 6379
timeout 300
loglevel notice

logfile /tmp/redis.out

rdbcompression yes

dbfilename /tmp/dump.rdb

dir /tmp

slave-serve-stale-data yes

appendonly yes

appendfsync everysec

no-appendfsync-on-rewrite no

glueoutputbuf yes
hash-max-zipmap-entries 64
hash-max-zipmap-value 512
activerehashing yes

vm-enabled yes
vm-swap-file /tmp/redis.swap

vm-max-memory 12884901888
#vm-page-size 4096
#vm-pages 65536000

vm-page-size 64
vm-pages 4194304000
vm-max-threads 16


Hope this is enough, if you think there is more to check just let me know.

Thanks!


Jonathan

[Salvatore Sanfilippo]

Hello, please can you restart with loglevel warning and check what the
latest line is?

Also please attack a debugger launching gdb, then:

attach <pid of your process>
bt

then send us the output of "bt" (back trace).

Also please report what Redis version is.

Thanks!
Salvatore

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
>

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Jonathan Leibiusky]

Thanks, the log file remains the same with warning loglevel. Although there are some other messages that I'm pasting here:

[18825] 24 Jan 10:25:26 # SIGTERM received, scheduling shutting down...
[19551] 24 Jan 10:30:01 # Deprecated configuration directive: "glueoutputbuf"
[19552] 24 Jan 10:30:01 * Using '/data/redis/master1/redis.swap' as swap file
[19552] 24 Jan 10:30:01 * Allocating 268435456000 bytes of swap file
[19552] 24 Jan 10:30:01 * Swap file allocated with success
[19552] 24 Jan 10:30:01 * Server started, Redis version 2.1.11
[19552] 24 Jan 10:30:01 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[19552] 24 Jan 10:44:06 # SIGTERM received, scheduling shutting down...
[19948] 24 Jan 10:45:50 # Deprecated configuration directive: "glueoutputbuf"
[19949] 24 Jan 10:45:50 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

I attached to the process and the bt is the following:

#0  0x0000003b7dac62c0 in __write_nocancel () from /lib64/libc.so.6
#1  0x0000003b7da6bab3 in _IO_new_file_write () from /lib64/libc.so.6
#2  0x0000003b7da6b9c6 in _IO_new_do_write () from /lib64/libc.so.6
#3  0x0000003b7da6bff7 in _IO_new_file_sync () from /lib64/libc.so.6
#4  0x0000003b7da611a0 in fflush () from /lib64/libc.so.6
#5  0x00000000004276e1 in vmWriteObjectOnSwap (o=0x223525578, page=<value optimized out>) at vm.c:254
#6  0x0000000000427a9d in vmSwapObjectBlocking (val=0x223525578) at vm.c:273
#7  0x0000000000427ced in vmSwapOneObject (usethreads=0) at vm.c:512
#8  0x00000000004261a6 in loadAppendOnlyFile (filename=<value optimized out>) at aof.c:297
#9  0x000000000040f6c5 in main (argc=<value optimized out>, argv=0x7fff19952ad8) at redis.c:1555

At thos stage redis is not responsive, not to PING or INFO or any other command. From the bt it seems like it has some trouble loading the AOF file, am I right?

[Salvatore Sanfilippo]

Apparently what happens is that Redis is trying to load the append
only file, but this takes a lot of time with VM.
Please it is possible to have multiple outputs of "bt" in order to
confirm Redis is not blocked in the fflush() call?

I think that eventually it will start responding again but this may
take a long time.

For this reasons we are moving away from Virtual Memory in the next
version of Redis.

Cheers,
Salvatore

[Jonathan Leibiusky]

Ok, 2 more bts, I deattached and reattached after a couple of minutes.

#0  0x0000003b7dac62c0 in __write_nocancel () from /lib64/libc.so.6
#1  0x0000003b7da6bab3 in _IO_new_file_write () from /lib64/libc.so.6
#2  0x0000003b7da6b9c6 in _IO_new_do_write () from /lib64/libc.so.6
#3  0x0000003b7da6bff7 in _IO_new_file_sync () from /lib64/libc.so.6
#4  0x0000003b7da611a0 in fflush () from /lib64/libc.so.6

#5  0x00000000004276e1 in vmWriteObjectOnSwap (o=0x11a7d1658, page=<value optimized out>) at vm.c:254
#6  0x0000000000427a9d in vmSwapObjectBlocking (val=0x11a7d1658) at vm.c:273

#7  0x0000000000427ced in vmSwapOneObject (usethreads=0) at vm.c:512
#8  0x00000000004261a6 in loadAppendOnlyFile (filename=<value optimized out>) at aof.c:297
#9  0x000000000040f6c5 in main (argc=<value optimized out>, argv=0x7fff19952ad8) at redis.c:1555

#0  0x0000003b7dac62c0 in __write_nocancel () from /lib64/libc.so.6
#1  0x0000003b7da6bab3 in _IO_new_file_write () from /lib64/libc.so.6
#2  0x0000003b7da6b9c6 in _IO_new_do_write () from /lib64/libc.so.6
#3  0x0000003b7da6bff7 in _IO_new_file_sync () from /lib64/libc.so.6
#4  0x0000003b7da611a0 in fflush () from /lib64/libc.so.6

#5  0x00000000004276e1 in vmWriteObjectOnSwap (o=0x1f75f6e88, page=<value optimized out>) at vm.c:254
#6  0x0000000000427a9d in vmSwapObjectBlocking (val=0x1f75f6e88) at vm.c:273

#7  0x0000000000427ced in vmSwapOneObject (usethreads=0) at vm.c:512
#8  0x00000000004261a6 in loadAppendOnlyFile (filename=<value optimized out>) at aof.c:297
#9  0x000000000040f6c5 in main (argc=<value optimized out>, argv=0x7fff19952ad8) at redis.c:1555

I think I might download diskstore branch and give it a try, what do you think?

[Salvatore Sanfilippo]

Ok I can confirm this. The VM is (very slowly) loading the dataset into memory.

On Mon, Jan 24, 2011 at 4:15 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:

> I think I might download diskstore branch and give it a try, what do you
> think?

What kind of data do you have inside your instance?
Possibly you can disable VM and use 2.2 features that will be able to
load the whole dataset into memory.

The problem with diskstore is that you need to first convert your
redis.db dump into diskstore format, currently it's not possible.

Other questions are, how many keys do you have? What are the data
inside every key? Average length?

Thanks

Cheers,
Salvatore

[Jonathan Leibiusky]

On Mon, Jan 24, 2011 at 12:20 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:

Ok I can confirm this. The VM is (very slowly) loading the dataset into memory.

On Mon, Jan 24, 2011 at 4:15 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:

> I think I might download diskstore branch and give it a try, what do you
> think?

What kind of data do you have inside your instance?

I use redis here to store lots of sorted sets as indexes, so later I can intersect and union to make searches. A kind of free text search search engine.
 

Possibly you can disable VM and use 2.2 features that will be able to
load the whole dataset into memory.

Actually the reason I'm using VM right now is that the whole index will be around 500 GB, I'm sharding this across 5 servers, each need to handle around 100 GB, I have 16 GB RAM server, 12 GB assigned to redis. So I need VM to handle this.
 

The problem with diskstore is that you need to first convert your
redis.db dump into diskstore format, currently it's not possible.

I wouldn't mind reindexing my 500 GB of data to use diskstore. At this stage I am very happy to try different solution in production (still don't have lots of traffic).
 

Other questions are, how many keys do you have? What are the data
inside every key? Average length?

I have around 12764381 keys per instance (x5 instances)
99% of the keys are sorted sets, where I store numbers (the document ID) and as score I use a timestamp (need to bring back results ordered by date) and it is very hard for me to say how big are the sorted sets, can't think of a good way to check that.

Let me know if there is more information I can provide. And thanks for your help!
 

Thanks

Cheers,
Salvatore

--

Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Salvatore Sanfilippo]

On Mon, Jan 24, 2011 at 4:55 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:

> Let me know if there is more information I can provide. And thanks for your
> help!

Ok you are right, I guess... you know much better your data :)
Diskstore is the way to go for your problem... please try reindexing
in diskstore, I think this is the kind of application where it can
work very well and we can get some awesome hint about how it is
working.

So please:

- Use the "unstable" branch on github
- Configure it with the best values of cache-max-memory and
cache-flush-delay you can find. That is, you can start with a max
memory value that is 60% of the RAM in your system in order to be
conservative, and with a flush delay that is as large as you can, for
instane 30 seconds would be cool, or even more if you can live with
such a delay for writes.
- Reload the data inside diskstore. Bulk-writes will be slow of course
as the cache will not be of big use, but indeed the flush delay will
help *a lot* about this.

Then read experience should be great as the most used indexes will
likely stay in RAM.

Also restart is zero-time.

I really look forward to see how this will work for you but I'm
optimist about it.
I'll provide all the help you need as for me this is a great chance to
learn about diskstore strengths and limits in this early stage of
development.

Thank you

[Jonathan Leibiusky]

Done. Downloaded unstable branch and used the following configuration (maybe I have something wrong):

daemonize yes

databases 16

pidfile /data/redis/master1/redis.pid
port 6379
timeout 300
loglevel warning

logfile /data/redis/master1/redis.out

rdbcompression yes

dbfilename /data/redis/master1/dump.rdb

dir /data/redis/master1/

slave-serve-stale-data yes

appendonly yes

appendfsync everysec

no-appendfsync-on-rewrite no

hash-max-zipmap-entries 64
hash-max-zipmap-value 512
activerehashing yes

diskstore-enabled yes
diskstore-path /data/redis/master1/redis.ds
cache-max-memory 7000000000
cache-flush-delay 30


when loading redis I get the following segfault (it is worth to mention that /data/redis/master1 is a filer not local to the machine where I am running redis):

[21144] 24 Jan 12:21:43 # diskstore error opening /data/redis/master1/redis.ds/c8/f4/0_c8f4e512d94acffa861db02832b89826180b083a-1295886103-25429208: No such file or directory
[21144] 24 Jan 12:21:43 # ------------------------------------------------
[21144] 24 Jan 12:21:43 # !!! Software Failure. Press left mouse button to continue
[21144] 24 Jan 12:21:43 # Guru Meditation: "Unrecoverable diskstore error. Exiting." #diskstore.c:201
[21144] 24 Jan 12:21:43 # (forcing SIGSEGV in order to print the stack trace)
[21144] 24 Jan 12:21:43 # ------------------------------------------------
[21144] 24 Jan 12:21:43 # ======= Ooops! Redis 2.3.0 got signal: -11- =======
[21144] 24 Jan 12:21:43 # # Server
redis_version:2.3.0
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:21144
tcp_port:6379
uptime_in_seconds:40
uptime_in_days:0
lru_clock:1662338

# Clients
connected_clients:27
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:1155976
used_memory_human:1.10M
used_memory_rss:2146304
mem_fragmentation_ratio:1.86
use_tcmalloc:0

# Allocstats
allocation_stats:2=49,6=1,8=62,9=582,10=171,11=71,12=38,13=204,14=121,15=126,16=13355,17=311,18=20,19=41,20=82,21=166,22=4,23=61,24=2907,25=2,26=2,27=3,28=3,29=2,30=4,31=8,32=760,33=121,34=2,35=91,36=2,37=34,38=1,39=83,40=283,41=7,42=1,46=49,47=2,48=540,49=3,50=3,51=16,52=22,53=58,54=64,55=1,56=28,58=27,59=22,60=36,61=34,62=69,63=83,64=7,65=6,66=28,67=7,70=7,71=22,72=4,75=39,82=1,85=6,88=286,127=35,128=7,133=11,137=49,139=52,141=7,143=1,145=3,147=11,149=21,151=27,153=18,155=27,159=7,161=6,169=7,171=7,173=4,175=2,179=13,183=7,185=3,187=4,189=10,193=7,195=1
[21144] 24 Jan 12:21:43 # /department/redisindex/redis2.2.0/2.2.0-rc1-109/src/redis-server(_redisPanic+0x79) [0x4294e9]
[21144] 24 Jan 12:21:43 # /department/redisindex/redis2.2.0/2.2.0-rc1-109/src/redis-server(_redisPanic+0x79) [0x4294e9]
[21144] 24 Jan 12:21:43 # /department/redisindex/redis2.2.0/2.2.0-rc1-109/src/redis-server(dsSet+0x167) [0x42c497]
[21144] 24 Jan 12:21:43 # /department/redisindex/redis2.2.0/2.2.0-rc1-109/src/redis-server(IOThreadEntryPoint+0x188) [0x427cb8]
[21144] 24 Jan 12:21:43 # /lib64/libpthread.so.0 [0x3b7e60673d]
[21144] 24 Jan 12:21:43 # /lib64/libc.so.6(clone+0x6d) [0x3b7dad3f6d]
[21184] 24 Jan 12:24:31 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

[Salvatore Sanfilippo]

Hello, unfortunately you need some kind of POSIX compatible
filesystem, that is, ext3 or any other local filesystem. Not sure NFS
follows POSIX filesystem behavior.

But the error appears to be in this case different, like if Redis was
interrupted while creating the initial diskstore structure, and then
restarted. Deleting the whole diskstore with rm -rf and restarting
Redis should be enough. Wait the creation of the on disk structure
before stopping it.

Starting from now it's possible to terminate Redis in bad ways without
problems with the diskstore.
But the first creation needs to build 65536 directories.

I'm here to help with any further problem.

Cheers,
Salvatore

[Jonathan Leibiusky]

Changed the conf to use local filesystem just to check but it keep failing with the same segfault.
Deleted the folder to make sure redis is creating it from scratch when starting.

My conf:

daemonize yes

databases 16

pidfile /data/redis/master1/redis.pid
port 6379
timeout 300
loglevel warning

logfile /data/redis/master1/redis.out

rdbcompression yes

dbfilename /data/redis/master1/dump.rdb

dir /data/redis/master1/

slave-serve-stale-data yes

hash-max-zipmap-entries 64
hash-max-zipmap-value 512
activerehashing yes

diskstore-enabled yes

diskstore-path /tmp/redis.ds
cache-max-memory 7000000000
cache-flush-delay 1

The segfault is:
[22916] 24 Jan 14:29:50 # diskstore error opening
/tmp/redis.ds/d6/59/0_d659c10e27d52b00987b65e85d99bce5480adcae-1295893790-14689512:

No such file or directory

[22916] 24 Jan 14:29:50 # ------------------------------------------------
[22916] 24 Jan 14:29:50 # !!! Software Failure. Press left mouse button
to continue
[22916] 24 Jan 14:29:50 # Guru Meditation: "Unrecoverable diskstore
error. Exiting." #diskstore.c:201
[22916] 24 Jan 14:29:50 # (forcing SIGSEGV in order to print the stack
trace)
[22916] 24 Jan 14:29:50 # ------------------------------------------------
[22916] 24 Jan 14:29:50 # ======= Ooops! Redis 2.3.0 got signal: -11-
=======
[22916] 24 Jan 14:29:50 # # Server

redis_version:2.3.0
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll

process_id:22916
tcp_port:6379
uptime_in_seconds:21
uptime_in_days:0
lru_clock:1663107

# Clients
connected_clients:0

client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory

used_memory:796328
used_memory_human:777.27K
used_memory_rss:1548288
mem_fragmentation_ratio:1.94
use_tcmalloc:0

# Allocstats
allocation_stats:6=1,8=20,9=47,10=13,11=37,12=6,13=38,14=24,15=62,16=10115,17=9,18=6,19=15,20=6,21=3,22=5,23=27,24=131,25=2,26=2,27=2,28=3,29=3,30=2,31=1,32=31,34=2,35=1,36=2,37=3,38=1,39=11,40=1,47=2,48=23,49=1,56=1,58=1,59=1,64=18,69=1,71=1,88=117,128=16,>=256=27

# Persistence
loading:0
aof_enabled:0
changes_since_last_save:1
bgsave_in_progress:0
last_save_time:1295893769
bgrewriteaof_in_progress:0

# Diskstore
ds_enabled:1
cache_max_memory:7000000000
cache_blocked_clients:0

# Stats
total_connections_received:3
total_commands_processe
[22916] 24 Jan 14:29:50 #
/department/redisindex/redis/redis-server(_redisPanic+0x79) [0x4294e9]
[22916] 24 Jan 14:29:50 #
/department/redisindex/redis/redis-server(_redisPanic+0x79) [0x4294e9]
[22916] 24 Jan 14:29:50 #
/department/redisindex/redis/redis-server(dsSet+0x167) [0x42c497]
[22916] 24 Jan 14:29:50 #
/department/redisindex/redis/redis-server(IOThreadEntryPoint+0x188)
[0x427cb8]
[22916] 24 Jan 14:29:50 # /lib64/libpthread.so.0 [0x3b7e60673d]
[22916] 24 Jan 14:29:50 # /lib64/libc.so.6(clone+0x6d) [0x3b7dad3f6d]


Let me know if you want me to check something else.

[Jonathan Leibiusky]

It seems like it is not creating the folder structure in redis.ds
Am I missing something?

[Salvatore Sanfilippo]

On Mon, Jan 24, 2011 at 7:38 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:
> It seems like it is not creating the folder structure in redis.ds
> Am I missing something?

Very strange, please can you check if you actually have folder 'd6/59' ?

Thanks,

[Jonathan Leibiusky]

didn't have those folders. the strange thing is that I deleted it once again, changed loglevel to notice and restarted and now it created the folders (so maybe I did something wrong before).

now I have a different problem. just for testing I set the cache-flush-delay to 10 seconds.
when the server is up I:
SET foo bar

wait 2 minutes, and I:

SHUTDOWN

after restarting the key is not there. so it seems like it is not storing it.

do I have to explicitly SAVE?

[Salvatore Sanfilippo]

On Mon, Jan 24, 2011 at 8:13 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:
> didn't have those folders. the strange thing is that I deleted it once
> again, changed loglevel to notice and restarted and now it created the
> folders (so maybe I did something wrong before).
>
> now I have a different problem. just for testing I set the cache-flush-delay
> to 10 seconds.
> when the server is up I:
> SET foo bar
>
> wait 2 minutes, and I:
>
> SHUTDOWN
>
> after restarting the key is not there. so it seems like it is not storing
> it.
>
> do I have to explicitly SAVE?

There is no need for explicit save, so I wonder what's happening.

Example, if I use this config:

diskstore-enabled yes
diskstore-path redis.ds
# cache-max-memory 10mb
cache-max-memory 5mb
cache-flush-delay 10
# loglevel notice
loglevel debug

And I perform the "SET foo bar" operation I see in the log:

[18939] 24 Jan 20:16:38 . Scheduling key foo for saving
[18939] 24 Jan 20:16:39 - DB 0: 1 keys (0 volatile) in 4 slots HT.
[18939] 24 Jan 20:16:39 - 1 clients connected (0 slaves), 936016 bytes in use
[18939] 24 Jan 20:16:44 - DB 0: 1 keys (0 volatile) in 4 slots HT.
[18939] 24 Jan 20:16:44 - 1 clients connected (0 slaves), 936016 bytes in use
[18939] 24 Jan 20:16:48 . Creating IO save Job for key foo
[18939] 24 Jan 20:16:48 . Queued IO Job 0x1010001c0 type 1 about key 'foo'

[18939] 24 Jan 20:16:48 . 1 IO jobs to process
[18939] 24 Jan 20:16:48 . Thread 4301594624: new job type save:
0x1010001c0 about key 'foo'
[18939] 24 Jan 20:16:48 . Thread 4301594624 completed the job:
0x1010001c0 (key foo)
[18939] 24 Jan 20:16:48 . Processing I/O completed job
[18939] 24 Jan 20:16:48 . COMPLETED Job type save, key: foo

From the scheduling at 16:38 after 10 seconds the key is saved, at 16:48.

What do you see instead?
It is possible that next time you started Redis without specifying the
config file so it started in-memory?

[Jonathan Leibiusky]

Ok... it seems like I found the bug (although not sure it is a bug).

When you restart the server the keys are not loaded. So if you run "KEYS" or "INFO" command, it won't find anything unless you load a command explicitly, by doing GET foo

So if you want to reproduce this do on an empty diskstore redis instance:

SET foo bar

SHUTDOWN
restart

INFO --> you'll see there are no keys in the keyspace
DBSIZE -> you'll get 0
RANDOMKEY -> nil
etc...

GET foo -> you'll get value "bar"

DBSIZE -> you'll get 1

and so on

:)

On Mon, Jan 24, 2011 at 4:18 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:

On Mon, Jan 24, 2011 at 8:13 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:
> didn't have those folders. the strange thing is that I deleted it once
> again, changed loglevel to notice and restarted and now it created the
> folders (so maybe I did something wrong before).
>
> now I have a different problem. just for testing I set the cache-flush-delay
> to 10 seconds.
> when the server is up I:
> SET foo bar
>
> wait 2 minutes, and I:
>
> SHUTDOWN
>
> after restarting the key is not there. so it seems like it is not storing
> it.
>

> do I have to explicitly SAVE?,

[Jonathan Leibiusky]

And to add more general information on diskstore. Creating the initial structure over NFS takes around 10 minutes. And it seems like NFS is POSIX compliant because it worked :)

So the only bug found so far is the one I just reported in my previous e-mail. Let me know what do you think and if you want me to open an issue. Meanwhile I think I can store the whole index in diskstore and report back :) (I'm not using any command that work only on keys)

Jonathan

[Salvatore Sanfilippo]

On Mon, Jan 24, 2011 at 8:33 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:
> Ok... it seems like I found the bug (although not sure it is a bug).
>
> When you restart the server the keys are not loaded. So if you run "KEYS" or
> "INFO" command, it won't find anything unless you load a command explicitly,
> by doing GET foo

This is a known issue indeed, basically KEYS is not implemented as a
diskstore operation so will just show entries currently on the
diskstore cache.

The same happens to RADNOMKEY and DBSIZE. All the other commands
should work as expected.
While KEYS will be implemented as diskstore operation, I'm not sure
about DBSIZE and RANDOMKEY, those may just return an error '-ERR not
supported in diskstore mode", or otherwise be implemented with a very
slow implementation.

Another interesting this is that you can BGSAVE while using diskstore,
in order to obtain single-file backups.
But currently it's hard to restore this backups if the database is
bigger than RAM.
We'll soon have a .rdb -> diskstore conversion utility that can do the trick.

Diskstore databases can be hot-copied while the database is running
without problems.

Cheers,

[Salvatore Sanfilippo]

On Mon, Jan 24, 2011 at 8:55 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:
> And to add more general information on diskstore. Creating the initial
> structure over NFS takes around 10 minutes. And it seems like NFS is POSIX
> compliant because it worked :)
>
> So the only bug found so far is the one I just reported in my previous
> e-mail. Let me know what do you think and if you want me to open an issue.
> Meanwhile I think I can store the whole index in diskstore and report back
> :) (I'm not using any command that work only on keys)

Awesome! Thank you, this is very appreciated and I'm eager to know hot
it will work :)
In Italy now it's 9 pm so in the next hours I may not respond, but
please write anything and tomorrow morning I'll carefully reply to all
your emails.

Thanks!

[Jonathan Leibiusky]

Storing data seems to be extremely slow with diskstore. Not sure why.
I tried setting the cache time to 360 secs thinking it would give
better results but without luck. Is it because it is creating a file
per key? Not 100% sure, but it seems like a slow operation.
Btw it is worth to mention I am using local HD for this test.

[Salvatore Sanfilippo]

On Mon, Jan 24, 2011 at 10:26 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:
> Storing data seems to be extremely slow with diskstore. Not sure why.
> I tried setting the cache time to 360 secs thinking it would give
> better results but without luck. Is it because it is creating a file
> per key? Not 100% sure, but it seems like a slow operation.
> Btw it is worth to mention I am using local HD for this test.

Most likely the problem is that you are not creating sets one after
the other, but are writing at random in many sorted sets at the same
time. So basically a given key can't be taken in memory for 360
seconds, because there are too many other sets that are changing at
the same time so Redis needs to flush data on disk.

After the first bulk load, write speed should start to be better as
you will stress probably a smaller percentage of the working set.

You can obtain huge performance increases if you try to model your
program to re-create the dataset writing the first key, then the
second, and so forth, sequentially, but this is not always possible...

The work on the b-tree will provide a better write speed, the
filesystem is reliable but very slow indeed...

I wonder if you have some number about commands per second that you
are able to obtain so far.

Cheers,
Salvatore

[Jonathan Leibiusky]

On Mon, Jan 24, 2011 at 6:31 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:

On Mon, Jan 24, 2011 at 10:26 PM, Jonathan Leibiusky <iona...@gmail.com> wrote:
> Storing data seems to be extremely slow with diskstore. Not sure why.
> I tried setting the cache time to 360 secs thinking it would give
> better results but without luck. Is it because it is creating a file
> per key? Not 100% sure, but it seems like a slow operation.
> Btw it is worth to mention I am using local HD for this test.

Most likely the problem is that you are not creating sets one after
the other, but are writing at random in many sorted sets at the same
time. So basically a given key can't be taken in memory for 360
seconds, because there are too many other sets that are changing at
the same time so Redis needs to flush data on disk.

In this test I am indexing the same item (but with a different ID) several times. So the sorted sets are created with the first item I index (19 ZSETS), after that I'm just adding to those sets and creating a list and 2 hashes per item.
So with every indexed item I'm changing 19 ZSETS and creating 1 list and 2 hashes.
 

After the first bulk load, write speed should start to be better as
you will stress probably a smaller percentage of the working set.

Still testing but in my tests it just gets worse. As I index more and more items (always the same item as mentioned above), the response time increases, and after a single command takes more than 2 seconds my socket just throws a timeout.
 

You can obtain huge performance increases if you try to model your
program to re-create the dataset writing the first key, then the
second, and so forth, sequentially, but this is not always possible...

Didn't understand. What do you mean by creating keys sequentially?
 

The work on the b-tree will provide a better write speed, the
filesystem is reliable but very slow indeed...

I wonder if you have some number about commands per second that you
are able to obtain so far.

Just benchmarking GETs and SETs I get up to 9000 req/s with diskstore.

I will try to make more tests after sleeping for a couple of hours :)