Re: MISCONF Redis is configured to save RDB snapshots... With no OOM. What is making SIGKILL?

[Colin Vipurs]

Quick thing - have you got enough disk space to perform the save?

If this is for unit testing, do you even need persistence turned on?

On Thu, Jan 10, 2013 at 4:32 PM, Konstantin Lutovich <luto...@gmail.com> wrote:
> Hello!
>
> I have a development server with 20 concurrently running Redis instances
> with default save configuration.
> These instances are used for unit test executions. Each instance has at most
> 20_000 keys at once, which are not "heavy" and those keys changes quite
> often as each test clear and populate its data.
>
> Quite rarely Redis fails with error:
> "MISCONF Redis is configured to save RDB snapshots, but is currently not
> able to persist on disk. Commands that may modify the data set are disabled.
> Please check Redis logs for details about the error."
>
> Forked for persistence process is killed with signal 9.
> I can see this in Redis logs: "Background saving terminated by signal 9" and
> in acct process monitoring: "redis-server F X root [...]".
>
> Server has 24gb of RAM and as a rule no more than 5gb is used, so no
> out-of-memory is possible. And there is 10gb swap partition. Monitoring(with
> Ganglia) is turned on and it confirms no OOM.
>
> I've searched the whole /var/log/ but did not find anything useful there.
> /proc/sys/vm/overcommit_memory is set to 1
> /proc/sys/vm/overcommit_ratio is set to 50
> Using Ubuntu 12.04.1 LTS
>
> Any suggestions why this can happen or where to look for the answer would be
> highly appreciated!
>
> Thanks in advance!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/rcDQe7N2HJsJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.



--
Maybe she awoke to see the roommate's boyfriend swinging from the
chandelier wearing a boar's head.

Something which you, I, and everyone else would call "Tuesday", of course.

[Konstantin Lutovich]

Yes, disk space is ok.

Actually I don't need persistence on a unit tests machine, so save can be simply commented out or stop-writes-on-bgsave-error could be set to no.

But what if the same happens on production? This is my main concern.

[Colin Vipurs]

Path to the rdb file is writeable? I've only ever seen this problem
occur because Redis can't actually write the RDB file or runs out of
memory while trying to do it.

> https://groups.google.com/d/msg/redis-db/-/sDlBTnhCCeIJ.

[Konstantin Lutovich]

Yes path and privileges are fine. Such failures are not permanent. 

Here is a part of log:

[1779] 10 Jan 16:04:32.822 * Background saving started by pid 4728

[1779] 10 Jan 16:04:33.172 * DB saved on disk

[1779] 10 Jan 16:04:33.172 # Background saving terminated by signal 9

I assume this means that file was written but still the whole saving was killed.

Next saving is successful:

[1779] 10 Jan 16:05:34.007 * 10000 changes in 60 seconds. Saving...

[1779] 10 Jan 16:05:34.008 * Background saving started by pid 4784

[4784] 10 Jan 16:05:34.139 * DB saved on disk

[4784] 10 Jan 16:05:34.140 * RDB: 0 MB of memory used by copy-on-write

[1779] 10 Jan 16:05:34.149 * Background saving terminated with success

[Jokea]

Check dmesg output and /var/log/messages to see why it was killed.


Regards,
Jokea

[Josiah Carlson]

If your process is not running as a daemon, and the controlling terminal dies (due to disconnection), Redis will die too. You should daemonize Redis, use screen, or use nohup.

If Redis is not able to persist, and you have enough disk/memory, then you have a permissions issue. Check the user that Redis is running under, and check the permissions on the path/file that Redis is attempting to write to. You probably ran into the issue because Redis is not being daemonized automatically through its configuration or via an init.d with start-stop-daemon, both of which would usually be run under a 'redis' or 'root' user on Linux.

 - Josiah

On Wed, Mar 20, 2013 at 1:12 PM, yati sagade <yati....@gmail.com> wrote:

I have the exact same problem, except that sometimes the server itself dies silently(not running as a daemon) and most of the time, the Python Redis client throws an exception wrapping MISCONF

redis.exceptions.ResponseError: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

I have ample disk space and memory and am running Redis 2.6.11

On Thursday, 10 January 2013 22:02:00 UTC+5:30, Konstantin Lutovich wrote:

Hello!

I have a development server with 20 concurrently running Redis instances with default save configuration. 

These instances are used for unit test executions. Each instance has at most 20_000 keys at once, which are not "heavy" and those keys changes quite often as each test clear and populate its data.

Quite rarely Redis fails with error:

"MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error."

Forked for persistence process is killed with signal 9. 

I can see this in Redis logs: "Background saving terminated by signal 9" and in acct process monitoring: "redis-server      F  X root [...]".

Server has 24gb of RAM and as a rule no more than 5gb is used, so no out-of-memory is possible. And there is 10gb swap partition. Monitoring(with Ganglia) is turned on and it confirms no OOM.

I've searched the whole /var/log/ but did not find anything useful there.
/proc/sys/vm/overcommit_memory is set to 1
/proc/sys/vm/overcommit_ratio is set to 50
Using Ubuntu 12.04.1 LTS

Any suggestions why this can happen or where to look for the answer would be highly appreciated!

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.

To post to this group, send email to redi...@googlegroups.com.

Visit this group at http://groups.google.com/group/redis-db?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Josiah Carlson]

John,

What evidence are you operating under that would suggest that the OP is having an issue with multiple Redis servers running at the same time, and that he would need to kill one (or more) of them?

Or are you trolling?

 - Josiah

On Mon, Jun 10, 2013 at 10:13 PM, John Santhosh <extre...@gmail.com> wrote:

This Error Occurs because there are more than one instances of redis serer running.
You Will Have To Kill Them Manually and restart server.

ps -auwxxx|grep redis              //To get instances of redis running

sudo kill -9 892                       //To Kill Process 892