Want to use Redis? Got more data than memory? Use NDS!

[Matt Palmer]

Hi everyone,

I've developed an experimental modification to Redis to store all keys in a
fast dbm-style format (using KyotoCabinet), and then using memory
effectively as a cache of the recently-used keys. I'm calling it "NDS", for
the "Naive Data Store", because basically I did the simplest thing that
could possibly work, as a proof of concept, without thinking too much about
how to do things "optimally". The result is surprisingly workable, and is
already in production for a customer of $DAYJOB.

The way it works is to flush modified keys to disk via a forked child
(similar to how RDB dumps and AOF rewrites work, to keep critical path
performance high), and then lookup keys in this datastore if they're not
present in memory.

The benefits of this are twofold: you can keep memory usage under control by
setting maxmemory without permanently losing data when keys get evicted.
Or, if that doesn't get you excited, you can just let memory usage run
rampant, and gain the benefits of more granular persistence than RDB, and
less maintenance and disk space overhead than AOF.

Also, by fetching data from disk as-needed, rather than having load all data
into memory before being able to do anything, Redis is available to service
client requests immediately on startup. Alternately, Redis' regular performance
characteristics can be maintained by loading all data into memory in the
background on startup (if performance is more important than RAM usage for
your use case), while still being able to service requests immediately.
It's like some sort of magic!

Rather than spend a long time describing the hows, whys, etc, I'll just
refer you to the post that got written on the company blog with all the
details:

http://www.anchor.com.au/blog/2013/04/redis-rethought-exciting-extremes-with-larger-than-memory-datasets/

At this stage, I'm interested in whether this is of use to other people, and
whether it works for you. If it *doesn't* work for you, please provide
patches. If nobody else finds it useful, I won't bother to keep the github
repo up-to-date with bugfixes.

- Matt

[Pierre Chapuis]

This looks really cool, I have to find the time to try it.

I already knew Redis and Tokyo Cabinet mix well for having written https://github.com/Moodstocks/redisk with a colleague some time ago. But it was a toy, whereas this thing looks like it could actually be used.

The KEYS issue is annoying though. I would prefer it to be slow and work correctly. KEYS is more a debugging command than something you should use in real code anyway so performance doesn't matter so much.

[Matt Palmer]

On Mon, Apr 15, 2013 at 01:52:32AM -0700, Pierre Chapuis wrote:
> The KEYS issue is annoying though. I would prefer it to be slow and work
> correctly. KEYS is more a debugging command than something you should use
> in real code anyway so performance doesn't matter so much.

The blog post is actually out of date on that point (I should get it fixed
up tomorrow); KEYS now works (along with RDB dumps and a host of other
things that involve grovelling through the entire keyspace, including the
keys on disk). With the exception of the expiry stuff, I don't know of
anything that NDS doesn't support now.

- Matt

[yunfan]

i have wrote a blog about redis like service in a vm architecture
but i dont have an english version

On Mon, Apr 15, 2013 at 01:52:32AM -0700, Pierre Chapuis wrote:

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Abhishek Rai]

Hi Matt,

Redis-NDS seems really cool and I have started testing it.
I am sharing you the first observation I saw observed about Redis-NDS. I'm sharing you the top command output. The swap space taken by redis server is 443 GB and the available swap is only 31 GB.
Can you explain what's actually happening ?


Mem:   3931056k total,  2576660k used,  1354396k free,   153396k buffers
Swap: 32406772k total,   190240k used, 32216532k free,   270472k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND                                                                                                 
10461 root      15   0  444g 1.0g  972 S  0.0 27.1   0:12.90 443g redis-server

[Javier Guerra Giraldez]

On Tue, Nov 12, 2013 at 5:03 AM, Abhishek Rai <shriabh...@gmail.com> wrote:
> The swap space taken by redis server is 443 GB and the available swap is
> only 31 GB.
> Can you explain what's actually happening ?

the database file is mmap()'ed, so that counts as 'swap space'

--
Javier

[Matt Palmer]

Hi Abhishek,

On Tue, Nov 12, 2013 at 02:03:31AM -0800, Abhishek Rai wrote:
> Redis-NDS seems really cool and I have started testing it.
> I am sharing you the first observation I saw observed about Redis-NDS. I'm
> sharing you the top command output. The swap space taken by redis server is
> 443 GB and the available swap is only 31 GB.
> Can you explain what's actually happening ?

What's happening is that top is lying to you. Whoever added that
never-to-be-sufficiently-damned "SWAP" column to top needs re-education,
because that's not swap.

As Javier mentioned, LMDB (which NDS uses for it's on-disk storage) mmap()s
the entirity of the data file (plus a bunch of extra space) into memory.
This ballons out the virtual memory of the process (the VIRT column) quite
considerably.

Where top is going wrong is in assuming that SWAP is simply VIRT - RES, when
it isn't. VIRT is RES + SWAP + MMAP + a few other things besides. It's
something that really bugs me about top.

- Matt

[Abhishek Rai]

Hi Matt,

Thanks for the explanation.
I'm facing a difficulty with Redis-NDS. I set the maxmemory in redis.conf file to 8 GB. My aim was to fille Redis with 8 GB data. I checked the info about redis through redis-cli. I'm sharing you the output of the info command here -> http://pastebin.com/zHywejcN
On executing the info command it displays information about db0 only. Executing info command for the second time it displayed information about all the 16 db's from db0 to db15. I executed the info command many times and it displays the information alternatively in this manner, one time it shows info about db0 and next time it shows info for db0 to db15. Why is this happening ?

Second difficulty I faced is that it takes infinite time to execute commands through redis-cli. I executed the command ' KEYS "*" ' and I have not received its output yet. I executed RANDOMKEY command and still no output. I quit the redis-cli and started it again. Still I was not able to execute the commands. What could be the problem ? I think redis-nds is not able to read the data ?

This is some information I'm sharing with you. Maybe it can help you to understand the problem better..

Page faults:
 MINFL  MAJFL
8853105 2175920

Size of data.mdb file created is 332 MB.

Top command status :
305 is the redis server and 8252 is the child process.

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND                                                                                                 

  305 root      18   0  451g 2.8g 6864 D  2.0 75.3  39:46.34 448g redis-server                                                                                             
 8252 root      18   0  451g 2.4g 6860 D  1.0 63.6   1:11.27 449g redis-server

----

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  SWAP COMMAND                                                                                                 

  305 root      18   0  451g 2.4g 6072 D  1.7 64.0  40:33.51 449g redis-server

-----
  305 root      18   0  451g 3.5g  24m D  0.3 93.9  41:30.97 448g redis-server

You can contact me at shriabh...@gmail.com for more discussion about the problems.

[Abhishek Rai]

Hi Matt,

This is in response to the previous reply I posted. I communicating the information in a point wise manner so you can get it easily.

Before Crashing the redis server : -
keyspace - db0:keys=66932,expires=0,avg_ttl=0,nds=94502

Page faults
MINFL     MAJFL
9016448  2394371

Redis uptime in seconds : 72593 

Time taken for the 'randomkey' command to output the result = 83 minutes.
2nd time when i executed 'randomkey' command , it took 1.63 seconds, 3rd time it took 1.82 seconds....

Then I executed 'keys "*" ' command. It took 16 minutes to display the output. And when next time I executed it took 5.5 seconds..

These commands took very much time when they are executed for the first time.

After Crashing and restarting the redis server :-

keyspace - db0:keys=0,expires=0,avg_ttl=0,nds=94502

Page faults  MINFL  MAJFL  61324      0

Executed 'randomkey' command. Output = (nil)
Again executed 'randomkey' command. Output = (nil)

Executed ' keys "*" ' command. All the 94502 keys were displayed on the screen in 3.62 seconds.
Again executed 'keys "*"' command. Result was same as above.

Executed 'randomkey' command. Output = (nil)

Executed 'lrange' command. Output was on the screen in seconds.
I executed 'lrange' command 7 times and then check the info. Number of keys in db0 were 7.
Page faults after executing 'lrange' command :-

MINFL  MAJFL 73993     7

keyspace - db0:keys=7,expires=0,avg_ttl=0,nds=94502


Again executed 'randomkey' command. And a key was displayed as a output.
I executed 'randomkey' many times and the output was immidiately displayed on the screen. But it displayed one of the 7 keys which i read through 'lrange'


My question is why on restarting the redis server, the dataset was not loaded into the memory ? Because of this 'randomkey' command was not displaying any key on execution.
I would like to mention that preload option was not enabled in the redis.conf file.

I set the maxmemory to 8 GB in the redis.conf file but it was able to store 18.46 GB of data. However it didn't used used_memory_peak_human more than 8 GB.
So setting the maxmemory doesn't impact the amount of data redis can store ?

What is the best settings that can be configured in redis.conf file to store max 32 GB of data with NDS enabled ?    RAM of my system is 4 GB.
 
Is there any dependency on the swap space ?

Does any other process other than redis-server runs in the background when NDS is enabled ?

Where is the data stored on the disk ?

[Abhishek Rai]

Hi Matt,

I want to use Redis-NDS in my production system. I'm testing it in my environment.
Questions are arising in my mind about the working of redis-nds.
I would like to know wether it will be convinient for you to discuss the problems here or some where else in private?

Thanks.

[Matt Palmer]

I'm happy to discuss things in public. I'd prefer it if you didn't Cc me on
list mail, though; it confuses things terribly.

- Matt

[Abhishek Rai]

Hi Matt,

I carried out some tests on Redis-NDS in my environment.
My system RAM is 4 GB, Swap space - 31 GB, Redis is set to maxmeory of 23 GB, maxmemory-policy allkeys-lru, maxmemory-samples 15.

I have following points on which I would like you explain about :

i)             Preload setting of Redis.

ii)            Mechanism behind considering the keys as the hot data after restarting.

iii)          How is it able to backup 8 GB data into a 167 MB file ? What compression method is used here? What’s the mechanism behind this ? Is it also storing data somewhere else also?

iv)           ‘NDS Flush’ command is not working. Why ?

v)            MDB file creation is done by redis-nds (redis-server) or any other process do this job ?

vi)           How is it able to accept more data greater than the maxmemory of Redis ? (I set maxmemory to 8 GB and it accepted data around 18.5 GB. It could have accepted more that if I didn’t stopped inserting data into redis).

vii)         Limitations of Redis-NDS.

Thanks,
Abhishek

[Matt Palmer]

On Fri, Nov 15, 2013 at 08:12:12AM -0800, Abhishek Rai wrote:
> I carried out some tests on Redis-NDS in my environment.
> My system RAM is 4 GB, Swap space - 31 GB, Redis is set to maxmeory of 23
> GB, maxmemory-policy allkeys-lru, maxmemory-samples 15.
>
> I have following points on which I would like you explain about :
>
> i) Preload setting of Redis.

It would help me (and all future users of NDS) if you could explain exactly
what questions you have about preloading that aren't answered by the
(admittedly brief) notes in README.nds. That way I can explain it in there
for the benefit of everyone.

> ii) Mechanism behind considering the keys as the hot data after
> restarting.

It works entirely based on which keys are requested. NDS makes no judgment
call on which keys may be "hot" on restart; it simply loads keys as they're
requested, and doesn't expire any keys until maxmemory is hit.

> iii) How is it able to backup 8 GB data into a 167 MB file ? What
> compression method is used here? What’s the mechanism behind this ? Is it
> also storing data somewhere else also?

All data is stored in the data.mdb file. It serializes and compresses the
values using the same format and systems as RDB.

> iv) ‘NDS Flush’ command is not working. Why ?

I don't know. What are the symptoms you're seeing?

> v) MDB file creation is done by redis-nds (redis-server) or any
> other process do this job ?

It's done by the LMDB library, if it is deemed necessary.

> vi) How is it able to accept more data greater than the maxmemory
> of Redis ? (I set maxmemory to 8 GB and it accepted data around 18.5 GB. It
> could have accepted more that if I didn’t stopped inserting data into
> redis).

What do you mean by "accepted data"? The whole point of NDS is to allow you
to store more data than is allowed by memory, so a simple reading of your
question doesn't make any sense to me, and I'd prefer not to speculate as to
your true meaning.

> vii) Limitations of Redis-NDS.

Apart from "it isn't as widely used (and hence tested) as regular Redis, and
so it is likely to contain all sorts of horrible, data-eating bugs", all the
non-obvious limitations of NDS are described towards the bottom of
README.nds.

- Matt

--
"Mr... Baggins. It seems you have been leading... two lives. In one... you are
an ordinary hobbit... you organize your uncle's diary... give generous gifts to
children. In the other... you are the Ring Bearer... charged to carry the One
Ring to Mount Doom. One life has... a future, Mr... Baggins, and the other..."

[Abhishek Rai]

Hi Matt,

i) I started redis server with preload settings enabled. The number of keys in db0 and nds were same when I executed the 'info' command. I tried to read the keys using 'lrange' command and the output I got was (nil). I tried to read many keys like this and I didn't get any output.
When I started redis server with preload setting disabled, the number of keys present in db0 is 0 and nds contained the total number of keys. I used 'lrange' command and it was successfully returned the output (value of the key).
In README.nds it is written that when preload option is enabled NDS will load all the keys off the disk at startup. My question is why I'm unable to read the data when preload is enabled ?

ii) ' KEYS * ' takes time to display all the keys. Normal redis executes ' KEYS * ' command faster.
When the number of keys displayed same in db0 and nds, are keys actually loaded into the memory as db0 shows number of keys equal to nds? (RAM is 4 GB, data is 10 GB)

iii) I think I have found a bug in Redis-NDS. I am discussing you the test case which reported a bug. Redis is running with preload setting disabled.
I inserted 10 GB data (about 50,000 keys). I crashed and restarted the server. When I executed the ' Debug Object ' command (This commands shows the idle time of the key), it resulted in a error "KEY NOT FOUND".
When I executed 'lrange' command on that key, it returned the value of that key successfully. Again I executed the 'Debug Object' command on the same key, it successfully returned the idle time of the key.
The summary is that whenever key is requested for reading or writing operation, that key is brought into the memory. But when commands like ' Debug object ' is executed, nds is unable to bring that key into the memroy and show the output.
This shouldn't happen. All the commands should execute successfully. 'Debug Object' should execute successfully.
You can try this test case. Insert some keys in the server. Crash and restart the server. Run 'Debug object' command. You will get the same error.
This is the first command which failed to execute successfully. It is possible that many more commands will not execute successfully.when the server is restarted.
I hope this problem is solved as soon as possible.


- Abhishek.

[Matt Palmer]

On Mon, Nov 18, 2013 at 04:37:26AM -0800, Abhishek Rai wrote:
> i) I started redis server with preload settings enabled. The number of keys
> in db0 and nds were same when I executed the 'info' command. I tried to
> read the keys using 'lrange' command and the output I got was (nil). I
> tried to read many keys like this and I didn't get any output.
> When I started redis server with preload setting disabled, the number of
> keys present in db0 is 0 and nds contained the total number of keys. I used
> 'lrange' command and it was successfully returned the output (value of the
> key).
> In README.nds it is written that when preload option is enabled NDS will
> load all the keys off the disk at startup. My question is why I'm unable to
> read the data when preload is enabled ?

I'm not sure. It sounds like a bug.

> ii) ' KEYS * ' takes time to display all the keys. Normal redis executes '
> KEYS * ' command faster.
> When the number of keys displayed same in db0 and nds, are keys actually
> loaded into the memory as db0 shows number of keys equal to nds? (RAM is 4
> GB, data is 10 GB)

Yep, KEYS has gone from being slow to being "OMFG will this thing ever
finish" slow, because we're trawling the entire on-disk database in order to
get the full list of keys. The reason why all the keys get loaded is
because Redis needs to check the expiry time on every key (which is stored
on disk), and it's easiest just to load all the keys into memory as you go.

Given that KEYS, even in Redis au-naturale, is a debug-only command, and
should not be used in any situation where you actually care about
performance and response time, I have no intention of spending time
improving this behaviour, but if you were feeling frisky it wouldn't be an
impossible job to have Redis check the expiry time without actually loading
the key into the main store (just discard it from memory once you've checked
that they key hasn't expired).

> iii) I think I have found a bug in Redis-NDS.

This is my not surprised face ----> :-|

<grin>

> I am discussing you the test
> case which reported a bug. Redis is running with preload setting disabled.
> I inserted 10 GB data (about 50,000 keys). I crashed and restarted the
> server. When I executed the ' Debug Object ' command (This commands shows
> the idle time of the key), it resulted in a error "KEY NOT FOUND".
> When I executed 'lrange' command on that key, it returned the value of that
> key successfully. Again I executed the 'Debug Object' command on the same
> key, it successfully returned the idle time of the key.
> The summary is that whenever key is requested for reading or writing
> operation, that key is brought into the memory. But when commands like '
> Debug object ' is executed, nds is unable to bring that key into the memroy
> and show the output.
> This shouldn't happen. All the commands should execute successfully. 'Debug
> Object' should execute successfully.

Yep, it wouldn't surprise me -- I don't tend to use the DEBUG commands, so I
wouldn't have noticed it.

<clickety click>

Yep, there it is. DEBUG OBJECT is doing an end-run around the usual key
loading routines (presumably to avoid LRU updates) and so I didn't catch it.
Give me until about 3pm your time, and the update should be available in the
Github repo.

> This is the first command which failed to execute successfully. It is
> possible that many more commands will not execute successfully.when the
> server is restarted.

Possibly. If so, please let me know.

> I hope this problem is solved as soon as possible.

Should be solved by 3pm+pull/build time. <grin>

- Matt

[Abhishek Rai]

Hi Matt,

I faced a problem. I set maxmemory to 140 GB and 'save' option as 5 1. Swap space availabe in the system is 140 GB. I inserted 724450 keys (about 140 GB data) into redis. Out of 724450 keys displayed against db0, only 349378 keys were there in nds. All keys were not flushed into the disk. What could be the reason? All the swap space was also consumed.
Before crashing the redis server - db0:keys=724450,expires=0,avg_ttl=0,nds=349378
After restarting the redis server - db0:keys=0,expires=0,avg_ttl=0,nds=349378

'Save' option is used for Snapshotting mode. What is the use of this option in Redis-NDS? 
"I use `5 1 1 5` by default, because I like the symmetry". Save option requires two parameters. How do you use 5 1 1 5, four parameters for 'save' option?

I also observed that a child process of redis is created when I was monitoring processes using top command. What is the job of this child process?

I want redis to hold 140 GB of data. Can you suggest a good configuration setting ('save' option, maxmemory) for redis so that all the keys are flushed into the disk ?

Thanks,
Abhishek.

[Matt Palmer]

On Tue, Nov 26, 2013 at 03:55:54AM -0800, Abhishek Rai wrote:
> I faced a problem. I set maxmemory to 140 GB and 'save' option as 5 1. Swap
> space availabe in the system is 140 GB. I inserted 724450 keys (about 140
> GB data) into redis. Out of 724450 keys displayed against db0, only 349378
> keys were there in nds. All keys were not flushed into the disk. What could
> be the reason? All the swap space was also consumed.

Almost certainly it'll be because the flushing child didn't manage to get
all the keys written to disk before it fell over -- it probably saved that
349k keys to disk while you were writing them, and then couldn't fork to
write the rest because it ran out of memory.

> 'Save' option is used for Snapshotting mode. What is the use of this option
> in Redis-NDS?

It is used to determine when NDS writes modified keys to disk. The logic is
identical to how RDB dumps work, except that instead of taking an RDB dump
of *all* keys, NDS only writes out those keys which have changed.

> "I use `5 1 1 5` by default, because I like the symmetry". Save option
> requires two parameters. How do you use 5 1 1 5, four parameters for 'save'
> option?

You can actually give save any even number of parameters. In fact, if
you're using `config set` to change the `save` option, you *have* to put
them all in one line. Redis just splits it into pairs of numbers
representing "change count" and "timeout" when configuring itself.

> I also observed that a child process of redis is created when I was
> monitoring processes using top command. What is the job of this child
> process?

That is the process that actually writes out the changed keys to disk. The
"master" process, which handles all the client requests, doesn't do any
writes itself -- that would cause too much of a performance hit. Instead,
NDS keeps a list of changed keys, and when Redis decides that enough changes
have been made (using the `save` option), it forks a child to take that list
of changed keys and write them to disk, while allowing the master to
continue handling client requests.

> I want redis to hold 140 GB of data. Can you suggest a good configuration
> setting ('save' option, maxmemory) for redis so that all the keys are
> flushed into the disk ?

Set `save 1 1`, `maxmemory` to 2/3 of the available system memory, and
`nds-watermark` to available system memory. "Available" means "whatever
isn't allocated to other things on the system", so if you're also running a
database and a webserver on there, it'll be significantly less than what
`free`(1) will report.

NDS can, in theory, store *any* quantity of data, more-or-less regardless of
`maxmemory`. That setting *only* controls how much of the data is cached in
*memory*; how much data ends up on disk is dependent only on how much you
put in. There is no setting in NDS to limit how much diskspace is consumed.

- Matt

--
CH3_ _ _ _ _ _ _ _ _ _ _
CH3_X_X_X_X_X_X_X_X_X_X_>
<_X_X_X_X_X_X_X_X_X_> 1,2-dimethylchickenwire
<_X_X_X_X_X_X_X_X_X_> -- Michael McConnell, ASR

[Abhishek Rai]

Hi Matt,

Thanks for the suggestion. Redis-NDS worked fine with the settings you recommended.

What is nds-watermark? I want to know about this new setting. There is no description given in README.nds about this setting.

I inserted about 140 GB data into Redis-NDS. The size of MDB file created is 3.7 GB. After inserting 140 GB data I deleted all the data but the size of the mdb file was still 3.7 GB. Why the size of MDB file did not reduced?
Is MDB file an ever growing file?

Does NDS uses 'rdbcompression' and 'rdbchecksum' settings ?

I would also like to know about these parameters which are displayed in redis info.
nds_keycache:
nds_cache_hits:
nds_cache_misses:
nds_cache_hit_rate:
nds_usec:
nds_dirty_keys:
nds_flushing_keys:
nds_flush_success:
nds_flush_failure:

[Matt Palmer]

[Please don't override the Reply-To header and Cc me directly; it screws up
my mail filtering]

On Thu, Nov 28, 2013 at 02:56:48AM -0800, Abhishek Rai wrote:
> What is nds-watermark? I want to know about this new setting. There is no
> description given in README.nds about this setting.

I've written up some additional documentation in README.nds, describing all
of the available config parameters and INFO variables, along with the
beginnings of a "Theory of Operation" section, which should explain more
about how it all works.

> I inserted about 140 GB data into Redis-NDS. The size of MDB file created
> is 3.7 GB. After inserting 140 GB data I deleted all the data but the size
> of the mdb file was still 3.7 GB. Why the size of MDB file did not reduced?
> Is MDB file an ever growing file?

LightningDB, like pretty much every disk-based data storage program in existence,
doesn't ever release disk space once it's got it. Instead, it just marks
chunks of data as "available" to be used again. You'll probably find that
if you stuff that data back in again, the file won't grow much (or perhaps
even at all).

I'm pretty impressed that you managed to put 140GB of data into 3.7GB of
disk space. I've seen impressive reductions before, but that might be a
first.

> Does NDS uses 'rdbcompression' and 'rdbchecksum' settings ?

It doesn't respect those settings, but it does compress and checksum data
before writing it to disk.

> I would also like to know about these parameters which are displayed in
> redis info.

I've documented all those parameters now; please let me know if the
explanations don't cover what you're after.

- Matt

[Abhishek Rai]

Hi Matt,

I am getting some error in Redis - NDS. Please tell me about these errors:

I'm sharing you the redis logs :

[22082] 30 Jan 17:24:46.135 - 16 clients connected (0 slaves), 1153320 bytes in use
[22082] 30 Jan 17:24:51.251 - DB 2: 1 keys (0 volatile) in 4 slots HT.
[22082] 30 Jan 17:24:51.251 - DB 3: 1 keys (0 volatile) in 4 slots HT.
[22082] 30 Jan 17:24:51.251 - 16 clients connected (0 slaves), 1153320 bytes in use
[22082] 30 Jan 17:24:51.601 # mdb_get(CF_HASH) failed: Invalid argument
[22082] 30 Jan 17:24:51.601 # mdb_get(CF_HASH) failed: Invalid argument
[22082] 30 Jan 17:24:56.363 - DB 2: 1 keys (0 volatile) in 4 slots HT.
[22082] 30 Jan 17:24:56.363 - DB 3: 1 keys (0 volatile) in 4 slots HT.
[22082] 30 Jan 17:24:56.363 - 16 clients connected (0 slaves), 1153320 bytes in use


[21582] 30 Jan 17:23:27.993 # mdb_env_open() failed: No such file or directory
[21582] 30 Jan 17:23:28.018 * 1 changes in 1 seconds. Saving...
[21784] 30 Jan 17:23:28.023 # mdb_env_open() failed: No such file or directory
[21582] 30 Jan 17:23:28.126 * NDS background save completed.  exitcode=1, bysignal=0
[21582] 30 Jan 17:23:28.226 * 1 changes in 1 seconds. Saving...
[21785] 30 Jan 17:23:28.231 # mdb_env_open() failed: No such file or directory
[21582] 30 Jan 17:23:28.333 * NDS background save completed.  exitcode=1, bysignal=0
[21582] 30 Jan 17:23:28.440 * 1 changes in 1 seconds. Saving...
[21786] 30 Jan 17:23:28.445 # mdb_env_open() failed: No such file or directory
[21582] 30 Jan 17:23:28.550 * NDS background save completed.  exitcode=1, bysignal=0

Thanks,
Abhishek

[Simone Scarduzio]

Dear Matt, 

I can reliably reproduce a crash just sending SET and GET of ascii keys containing non-ascii values and observing the flow with redis-cli MONITOR.

Find here the bug report (I can't find the github issues section in your project, so I mail you here):

http://paste.ubuntu.com/6850118/

[Simone Scarduzio]

Rapid update: seems like it's the monitor command itself crashing it. As of now, I could load half a gig of keys without crashing it.

[pedigree]

I've just installed it and slaved it to my 1.2gb master and it runs greats.  I tried to get it to preload and get a segfault the moment I issue "NDB PRELOAD" in the cli

Program received signal SIGSEGV, Segmentation fault.
_redisAssert (estr=0x4aa940 "retval == REDIS_OK", file=0x4b0eb2 "nds.c", line=644) at debug.c:397
397         *((char*)-1) = 'x';
(gdb)
(gdb) bt
#0  _redisAssert (estr=0x4aa940 "retval == REDIS_OK", file=0x4b0eb2 "nds.c", line=644) at debug.c:397
#1  0x000000000044fac8 in preloadWalker (key=0x7ffff6cd0160, data=<optimized out>) at nds.c:644
#2  preloadWalker (data=0x7ffff6c59100, key=0x7ffff6cd0160) at nds.c:637
#3  0x000000000044f20d in walkNDS (db=<optimized out>, walkerCallback=0x44fa40 <preloadWalker>, data=0x7ffff6c59100, interrupt_rate=1000) at nds.c:604
#4  0x000000000044f38b in preloadNDS () at nds.c:658
#5  0x0000000000450984 in ndsCommand (c=0x7ffff6d63000) at nds.c:1103
#6  0x000000000041b02f in call (c=0x7ffff6d63000, flags=7) at redis.c:1713
#7  0x000000000041d62d in processCommand (c=0x7ffff6d63000) at redis.c:1889
#8  0x0000000000425dbf in processInputBuffer (c=0x7ffff6d63000) at networking.c:1017
#9  0x0000000000425ed3 in readQueryFromClient (el=<optimized out>, fd=9, privdata=0x7ffff6d63000, mask=<optimized out>) at networking.c:1080
#10 0x0000000000416cc5 in aeProcessEvents (eventLoop=0x7ffff6c58150, flags=3) at ae.c:382
#11 0x0000000000416f9b in aeMain (eventLoop=0x7ffff6c58150) at ae.c:425
#12 0x0000000000415dc1 in main (argc=<optimized out>, argv=<optimized out>) at redis.c:3015

[pedigree]

Cough, NDS PRELOAD

[Paul L]

I've gone over the code and my C++ / Redis coding skills just arent up to it, so its still segfaulting .  damn :(

[Simone Scarduzio]

I couldn't either. Apparently there's no plan to merge this or do anything similar in the Redis project.

What I ended up doing is moving to ARDB.

It's based on LevelDB, so it has C++ dependencies (sorry Salvatore, I know you'd rather die!).

I've been an early adopter of ARDB. Together with the main developer, we spent a considerable amount of work in making it more stable. As a result, I'm now able to run it in production. At the moment, thanks to ARDB, my company is saving several thousands of dollars per month.

Compared to Redis it's a slightly more CPU hungry, but no big deal, since it's multithreaded (Redis is not). Also, predicting the memory usage requires a bit of fine tuning of the LevelDB parameters.

But yeah, give it a try and let me know :)

[Raidon]

Hello Matt,

Redis NDS is using more RAM than it's specified maxmemory.

redis 127.0.0.1:6379> config get maxmemory
1) "maxmemory"
2) "67108864"

output of info command:

# Memory
used_memory:67028088
used_memory_human:63.92M
used_memory_rss:474562560
used_memory_peak:810703048
used_memory_peak_human:773.15M
used_memory_lua:31744
mem_fragmentation_ratio:7.08
mem_allocator:libc

Output of top command:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8596 root 20 0 49.7g 452m 1024 S 0.0 11.8 5:11.58 redis-server

Why Redis is using more RAM than it's maxmemory? It should not use RAM greater than its specified maxmemory.

[Paul L]

ve reported a couple of bugs to ARDB in the past but will look at again now that someone is using it in production.  I like the tables feature that it uses.  If it had interval sets, then I would even consider not using redis at all.

I've tried compiling it so that I could use the lightningdb (lmdb engine) but it wasnt working, I'll need to revisit now :)

[Paul L]

It looks like the hyperloglog code broke the compile :(

On Tuesday, April 8, 2014 8:15:48 PM UTC+1, Simone Scarduzio wrote:

[Simone Scarduzio]

Some trivial (not for me) C++ compiler reason still prevents ARDB to build in OSX, so make sure you’re building in linux. Or more hopefully, fix it! :)

https://github.com/yinqiwen/ardb/issues/25

-Simone

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/ui4hEVpDxS4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Matt Palmer]

On Fri, Apr 25, 2014 at 01:12:27AM -0700, Raidon wrote:
> Hello Matt,
>
> Redis NDS is using more RAM than it's specified maxmemory.
>
> redis 127.0.0.1:6379> config get maxmemory
> 1) "maxmemory"
> 2) "67108864"
>
> output of info command:
>
> # Memory
> used_memory:67028088

Hmm... 67028088 is less than 67108864, so I'm not sure what the issue is
you're referring to. If it's that the peak usage is up high:

> used_memory_peak:810703048
> used_memory_peak_human:773.15M

That's because there are certain circumstances (such as high change rates)
in which the background worker can't write out changes quickly enough, and
so those changes are cached in memory (rather than being discarded or
causing writes to fail). In more recent versions of NDS, I've added a new
"watermark" feature which *does* cause writes to fail if it would cause
memory usage to exceed max memory, so if you really need to run under tight
memory constraints, I'd look into using that. However, before that, I'd
try:

> mem_fragmentation_ratio:7.08
> mem_allocator:libc

Changing this. That fragmentation ratio is pretty nasty. I find jemalloc
to work much better.

[Abhishek Rai]

I have not found any updates in README.nds file.
Still no idea about the INFO variables.

On Saturday, November 30, 2013 2:55:57 PM UTC+5:30, Matt Palmer wrote:

[Please don't override the Reply-To header and Cc me directly; it screws up
my mail filtering]

On Thu, Nov 28, 2013 at 02:56:48AM -0800, Abhishek Rai wrote:
> What is nds-watermark? I want to know about this new setting. There is no
> description given in README.nds about this setting.

I've written up some additional documentation in README.nds, describing all
of the available config parameters and INFO variables, along with the
beginnings of a "Theory of Operation" section, which should explain more
about how it all works.