Crash on save with "Unknown encoding type"

33 views
Skip to first unread message

Manton Reece

unread,
Jan 3, 2020, 7:13:54 PM1/3/20
to Redis DB
Hi everyone,

I have a Redis server (version 3.2.6) that recently started crashing whenever the RDB is saved. I've temporarily disabled the automatic save to keep the server running, but all bgsave and bgrewriteaof commands fail. Unfortunately AOF is not enabled.

The memory used is 10 GB on a 48 GB Linode instance. The last successful RDB is about 3 GB on disk. When saving now, it writes a temp file up to about 2 GB before crashing. Running bgrewriteaof also creates an incomplete file before crashing.

Here is the full logging output of the crash: https://gist.github.com/manton/9632251a886c5cc152a65eaf6bc34063

I know 3.2.6 is old. I would be fine running a blocking save instead of bgsave but I'm concerned that will only crash the main process, losing all of the changes since the last RDB.

My guess is that there is either corruption with some of the newer keys, or bad memory in the forked process. There is no slave but I also assume that setting one up would crash while doing a full sync.

Any tips for recovering from this problem without losing any data currently in memory? Thanks!

— Manton

Greg Andrews

unread,
Jan 3, 2020, 10:31:08 PM1/3/20
to Redis DB

My suggestion would be to write a client that iterates over all the keys, reading from your v3.2.6 Redis process and writing into another Redis process.  Then the other process can save to RDB for recovery purposes.

Manton Reece

unread,
Jan 6, 2020, 5:20:30 PM1/6/20
to Redis DB
Thanks Greg! I was thinking something like that as well… I could iterate over all the "important" keys first that I really don't want to lose, and then the rest.

Based on what I've described about the forked process crashing on every save, does anyone think it's likely that the main process will also crash just when iterating over the keys? With this "unknown encoding type" error, could a simple GET (or similar) crash or will it just return an error or nil value?

— Manton

Greg Andrews

unread,
Jan 6, 2020, 11:13:49 PM1/6/20
to Redis DB
That's a good question.  I tend to feel that the running Redis server process won't crash, especially if you iterate over your important keys first.  The reason I feel it won't be a problem for you is that your Redis server process isn't crashing when your clients do their normal reads from it.  That suggests either the trouble key(s) do not crash Redis on your normal client GET commands, or the trouble key(s) are not ones your clients are reading anymore.

I could be wrong, but that's my logic.

孟江莱

unread,
Jul 19, 2022, 12:02:17 PM7/19/22
to Redis DB
Hello Manton,
I just got the same problem as you did, and I don't know what kind of key or value it was. Do you have the samples about the encoding that cause the crash? 
what you had worried about happened  in my instance,  the main process crashed when I scan the error encoding key. Now I am trying to find out the bug key or value. 
If you know about the key or encoding, that will be very helpful. 
Thanks~
—mengjianglai
Reply all
Reply to author
Forward
0 new messages