temp file buildup

[Jason Priebe]

I just noticed a large number of temp*rdb files piling up in my redis
data directory.

At first, I thought it was because I was starting/stopping redis
incorrectly.

But I cleaned the files up, started redis, and let it run for the past
24 hours, and I still have tons of files:

...
-rw-r--r-- 1 redis redis 173650391 Apr 23 12:12
temp-1240503045.1804289383.rdb
-rw-r--r-- 1 redis redis 21027335 Apr 23 12:07
temp-1240502847.1804289383.rdb
-rw-r--r-- 1 redis redis 21072362 Apr 23 12:04
temp-1240502646.1804289383.rdb
-rw-r--r-- 1 redis redis 78972905 Apr 23 12:00
temp-1240502402.1804289383.rdb
-rw-r--r-- 1 redis redis 20949982 Apr 23 11:56
temp-1240502189.1804289383.rdb
-rw-r--r-- 1 redis redis 78805861 Apr 23 11:53
temp-1240501948.1804289383.rdb
-rw-r--r-- 1 redis redis 20910080 Apr 23 11:48
temp-1240501727.1804289383.rdb
-rw-r--r-- 1 redis redis 323742088 Apr 23 11:45 dump.rdb
...

I wonder -- could this have something to do with redis taking too long
to write the data to disk?

Example: redis is configured to write to disk after 300 seconds if
more than 10 keys changed. What if it takes 7 minutes to write to
disk? Or worse, what if I'm getting updates at a rate of 10,000 per
minute, and it takes 3 minutes to write?

Does another process start up, and there's some sort of contention,
and a file gets abandoned?

It is now 17:35, and the timestamp on dump.rdb is 11:45. It seems
that for the past 6 hours, redis has been unable to write to dump.rdb,
although it has a lot of these temp files.

[Jason Priebe]

Salvatore:

Do you have any thoughts on this one? Our redis server has not saved
its data for 5 days, so I'm a bit worried. It's writing a lot of temp
files, though.

-Jason

[Salvatore Sanfilippo]

Hello Jason!

this looks like a problem with permissions. Redis saves dbs creating a
temp file and then renaming this temp file into the db filename. What
happens is that for some reason the moving does not happen, because
for example there is a dump.rdb file that can't be overwritten. Is
there some info in the log files maybe?

Cheers,
Salvatore

>
> -Jason
>
>
> >
>



--
Salvatore 'antirez' Sanfilippo
http://invece.org

[Jason Priebe]

Here are the most recent log entries:

- 10000 changes in 60 seconds. Saving...
- Background saving started by pid 20153
- Background saving terminated with success
- 10000 changes in 60 seconds. Saving...
- Background saving started by pid 20319
- Background saving terminated with success
- 10000 changes in 60 seconds. Saving...
- Background saving started by pid 20449
- Background saving terminated with success

The redis-server process is running as user 'redis'. Its data
directory is
/var/lib/redis. Here are the permissions on that directory:

drwxrwx--- 2 redis redis 28672 Apr 29 07:10 /var/lib/redis

The dump file permissions are this:

-rw-r--r-- 1 redis redis 323742088 Apr 23 11:45 dump.rdb

redis reports that it's using about 320MB of memory, but the temp
files are only about 40MB.

Have a look at the most recent files:

-rw-r--r-- 1 redis redis 35669641 Apr 29 07:11
temp-1241003416.1804289383.rdb
drwxrwx--- 2 redis redis 28672 Apr 29 07:10 .
-rw-r--r-- 1 redis redis 35669641 Apr 29 07:09
temp-1241003308.1804289383.rdb
-rw-r--r-- 1 redis redis 40427520 Apr 29 07:07
temp-1241003191.1804289383.rdb
-rw-r--r-- 1 redis redis 29245440 Apr 29 07:05
temp-1241003091.1804289383.rdb
-rw-r--r-- 1 redis redis 29245440 Apr 29 07:03
temp-1241002993.1804289383.rdb
-rw-r--r-- 1 redis redis 29245440 Apr 29 07:02
temp-1241002892.1804289383.rdb
-rw-r--r-- 1 redis redis 17993728 Apr 29 07:00
temp-1241002808.1804289383.rdb

At first I thought that the background saves were having trouble
keeping up with the
rate of updates. Now I'm not so sure.

Right now, our redis server is idle. I issued the "save" command via
redis-cli. I got
a 28MB temp file, but no dump.rdb. Then I noticed that redis-server
was dead. I guess
redis-server crashed during the save. I suspect that all the other
temp* files I'm seeing
are due to the background processes crashing.

When I issued the "save" command, nothing showed up in the redis log,
BTW.


On Apr 28, 5:06 pm, Salvatore Sanfilippo <anti...@gmail.com> wrote:

[Jason Priebe]

Nevermind -- I see that 0.093 fixes a bug in the save. This is
probably what I'm
running into.

-Jason

[Salvatore Sanfilippo]

Yes I wonder if it's the same problem. Please if you test it ping me
about this issue.

Thanks