Redis `dir` path changing.

[Virendra Negi]

Following question was posted on SO.

We are facing an issue with redis, where the 'dir' path for the redis is getting set without any notice.

Resulting in the following error (while writing to redis).

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Information we have to support this.

- Redis Config file

A look at our redis.conf (located at /etc/redis/redis.conf) hint us that the dir path value is...

# The working directory.
#
# The DB will be written inside this directory, with the filename specified
# above using the 'dbfilename' configuration directive.
#
# The Append Only File will also be created inside this directory.
#
# Note that you must specify a directory here, not a file name.
dir /var/lib/redis

- Redis INFO Result.

{"redis_version"=>"2.8.17", "redis_git_sha1"=>"00000000", "redis_git_dirty"=>"0", "redis_build_id"=>"e2de0295da7f0ee1", "redis_mode"=>"standalone", "os"=>"Linux 3.16.0-4-amd64 x86_64", "arch_bits"=>"64", "multiplexing_api"=>"epoll", "gcc_version"=>"4.9.2", "process_id"=>"6860", "run_id"=>"4ecd0ff19798a88793e7a3ce281f398ba9aa2f5a", "tcp_port"=>"6379", "uptime_in_seconds"=>"99525", "uptime_in_days"=>"1", "hz"=>"10", "lru_clock"=>"11741717", "config_file"=>"/etc/redis/redis.conf", "connected_clients"=>"2", "client_longest_output_list"=>"0", "client_biggest_input_buf"=>"0", "blocked_clients"=>"0", "used_memory"=>"784840", "used_memory_human"=>"766.45K", "used_memory_rss"=>"3624960", "used_memory_peak"=>"784840", "used_memory_peak_human"=>"766.45K", "used_memory_lua"=>"33792", "mem_fragmentation_ratio"=>"4.62", "mem_allocator"=>"jemalloc-3.6.0", "loading"=>"0", "rdb_changes_since_last_save"=>"0", "rdb_bgsave_in_progress"=>"0", "rdb_last_save_time"=>"1454582288", "rdb_last_bgsave_status"=>"ok", "rdb_last_bgsave_time_sec"=>"0", "rdb_current_bgsave_time_sec"=>"-1", "aof_enabled"=>"0", "aof_rewrite_in_progress"=>"0", "aof_rewrite_scheduled"=>"0", "aof_last_rewrite_time_sec"=>"-1", "aof_current_rewrite_time_sec"=>"-1", "aof_last_bgrewrite_status"=>"ok", "aof_last_write_status"=>"ok", "total_connections_received"=>"12", "total_commands_processed"=>"4651", "instantaneous_ops_per_sec"=>"0", "rejected_connections"=>"0", "sync_full"=>"0", "sync_partial_ok"=>"0", "sync_partial_err"=>"0", "expired_keys"=>"0", "evicted_keys"=>"0", "keyspace_hits"=>"0", "keyspace_misses"=>"0", "pubsub_channels"=>"0", "pubsub_patterns"=>"0", "latest_fork_usec"=>"206", "role"=>"master", "connected_slaves"=>"0", "master_repl_offset"=>"0", "repl_backlog_active"=>"0", "repl_backlog_size"=>"1048576", "repl_backlog_first_byte_offset"=>"0", "repl_backlog_histlen"=>"0", "used_cpu_sys"=>"57.78", "used_cpu_user"=>"25.71", "used_cpu_sys_children"=>"0.00", "used_cpu_user_children"=>"0.00", "db0"=>"keys=2851,expires=0,avg_ttl=0"}

Key point

 redis.info['config_file'] 

 => /etc/redis/redis.conf

Process id of Redis

 redis.info['process_id']

 => 6860

- Redis Config

And then I inspected redis config and...

redis.config "get" ,"dir"

=> ["dir", "/etc/ssh"]
## ideally should be /var/lib/redis

Things, I did to fix the above error.

redis.config "set" ,"dir","/var/lib/redis"
=> "OK" 

redis.config "get", "dir"
=> ["dir", "/var/lib/redis"]

This sort of fix it (the above issue). But for temporary purpose only because.....

After couple of hours I noticed this.

redis.config "get" ,"dir"
=> ["dir", "/etc/ssh"]

Inspecting the redis config again

- Redis Info

{"redis_version"=>"2.8.17", "redis_git_sha1"=>"00000000", "redis_git_dirty"=>"0", "redis_build_id"=>"e2de0295da7f0ee1", "redis_mode"=>"standalone", "os"=>"Linux 3.16.0-4-amd64 x86_64", "arch_bits"=>"64", "multiplexing_api"=>"epoll", "gcc_version"=>"4.9.2", "process_id"=>"6860", "run_id"=>"4ecd0ff19798a88793e7a3ce281f398ba9aa2f5a", "tcp_port"=>"6379", "uptime_in_seconds"=>"109032", "uptime_in_days"=>"1", "hz"=>"10", "lru_clock"=>"11751224", "config_file"=>"/etc/redis/redis.conf", "connected_clients"=>"2", "client_longest_output_list"=>"0", "client_biggest_input_buf"=>"0", "blocked_clients"=>"0", "used_memory"=>"788984", "used_memory_human"=>"770.49K", "used_memory_rss"=>"3629056", "used_memory_peak"=>"808952", "used_memory_peak_human"=>"789.99K", "used_memory_lua"=>"33792", "mem_fragmentation_ratio"=>"4.60", "mem_allocator"=>"jemalloc-3.6.0", "loading"=>"0", "rdb_changes_since_last_save"=>"82", "rdb_bgsave_in_progress"=>"0", "rdb_last_save_time"=>"1454591521", "rdb_last_bgsave_status"=>"ok", "rdb_last_bgsave_time_sec"=>"0", "rdb_current_bgsave_time_sec"=>"-1", "aof_enabled"=>"0", "aof_rewrite_in_progress"=>"0", "aof_rewrite_scheduled"=>"0", "aof_last_rewrite_time_sec"=>"-1", "aof_current_rewrite_time_sec"=>"-1", "aof_last_bgrewrite_status"=>"ok", "aof_last_write_status"=>"ok", "total_connections_received"=>"14", "total_commands_processed"=>"6966", "instantaneous_ops_per_sec"=>"0", "rejected_connections"=>"0", "sync_full"=>"0", "sync_partial_ok"=>"0", "sync_partial_err"=>"0", "expired_keys"=>"0", "evicted_keys"=>"0", "keyspace_hits"=>"0", "keyspace_misses"=>"0", "pubsub_channels"=>"0", "pubsub_patterns"=>"0", "latest_fork_usec"=>"135", "role"=>"master", "connected_slaves"=>"0", "master_repl_offset"=>"0", "repl_backlog_active"=>"0", "repl_backlog_size"=>"1048576", "repl_backlog_first_byte_offset"=>"0", "repl_backlog_histlen"=>"0", "used_cpu_sys"=>"62.96", "used_cpu_user"=>"27.97", "used_cpu_sys_children"=>"0.00", "used_cpu_user_children"=>"0.00", "db0"=>"keys=2902,expires=0,avg_ttl=0"}

Key point.

 redis.info["process_id"]
 => 6860

 redis.info['config_file']
 => /etc/redis/redis.conf

So,

1. Redis Server was never restarted. (process_id says a lot about this).

2. Redis is picking the correct config file i.e. /etc/redis/redis.conf.

Question.

    So, If point 1, and 2 are true what is causing redis to set the config dir value to /etc/sshafter some hours? ( I have reset the config 5-6 times since yesterday.)

Note:

    Also restarted the server (yesterday) in a hope that the issue would be resolved permanently. But with no luck.

Note:

    Responding to @Keith answer

   Yes! the server is perfectly behind a restrictive firewall.

 Thought?

[Greg Andrews]

You asked this 3 days ago here on the mailing list and it was answered.  See this link:  https://groups.google.com/forum/#!topic/redis-db/OmJ50YTJuyU

To reiterate the answer:

Your Redis is exposed to someone you should not trust.  Probably to the whole world.  Someone is trying to use your exposed Redis to gain root access to the server.

They are connecting to your Redis and issuing configuration commands that will make it save its database to the root user's SSH authorized_keys file (on Linux servers this is usually /root/.ssh/authorized_keys).  Then they are writing a key to your Redis's database that contains their valid ssh public key.  Then they give the BGSAVE command to your Redis.  If your Redis is running as the root user, then the attacker's actions will make Redis save their SSH key to your root account, and they will try to log into your server as root.

This attack was described in this blog post:  http://antirez.com/news/96

The remedy is clear:

• Determine if your Redis successfully saved the attacker's ssh key to the root authorized_keys file.  If so, and your ssh daemon allows root logins, then you should consider the server to be cracked and restore the server from backups or re-create it from scratch.
• Prevent the world from connecting to your Redis.  Lock access down so only the machines you trust can access Redis.
• It would be far better if you would alter the start/stop scripts to run your Redis as a user different from root.

  -Greg

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Greg Andrews]

Yes, I see your statement that your Redis is behind a restrictive firewall.  I am confident that you will find someone malicious is getting around that firewall.  Either that firewall doesn't block access from the outside world in the way you believe it does, or the attack is coming from a point inside that firewall because the attacker found a way around it.

The Redis code discussed on this list does not unexpectedly change its 'dir' and 'dbfilename' or 'appendfilename' settings on its own.  It changes these when commanded to change them by a client.  The client can be through a remote TCP connection, or through a local socket connection.

  -Greg

On Sun, Feb 7, 2016 at 9:23 PM, Virendra Negi <vire...@idyllic-software.com> wrote:

--

[The Real Bill]

Fact: someone is kicking with your Redis instance. This is undeniable. It is a known attack pattern. I would thus conclude that factually speaking your Redis server is provably not "perfectly behind a restrictive firewall" that works.

Just let the ego go and realize your Redis server is factually exposed and compromised. Repeatedly. Stop the Redis instance, configure it correctly including a password, then start it back up. Then you will realize the correctness of what you've been told and can move back on to better things to do with your time than chase red herrings.

"Ok you say it is impossible but there it is so now what?" To paraphrase a Doctor.

[Anton Pavlov]

I'm witnessing a somewhat similar pattern. My redis was complaining about same persistence issue. In  .conf file I have "dir /opt/redis/var/db" and yet when it starts up it shows 
 127.0.0.1:6379> config get dir 
 1) "dir"
 2) "/"
All this is inside a very restricted dev space and I find chances that someone attacked it in the milliseconds between starting up redis and issuing "config get dir" to not be the most plausible explanation. 

All other settings from config file seem to load fine. 

[Greg Andrews]

Did the redis-server process you queried on 127.0.0.1 read the same config file you checked?  When I have a process that couldn't have been connected to and its config changed via commands, yet it has a different config than the config file, it has always been because I viewed a different config file than the process used.  Or I have two redis-server processes and I compared the config file for the other process, thinking it was for the same process.

If there were a bug in the code, there would be a lot of traffic here, on forums like StackOverflow/StackExchange, and in Twitter about it.