LinkedIn Redis API -- Part 1 -- Problem statement and Redis server configuration / policies

229 views

Skip to first unread message

Mike Svoboda

unread,

Apr 3, 2013, 12:54:21 PM4/3/13

to help-c...@googlegroups.com

Problem Statement

We have several use cases at LinkedIn where data across tens of thousands of machines needs to be centrally collected. Configuration management products (Chef, Puppet, Cfengine) are all good at taking a golden image and making thousands of machines identical, but they do not have a good way of centrally collecting arbitrary data.

Cfengine's own Enterprise product allows for central collection of "variable" data, or "class" data, but you can not collect arbitrary files off of the filesystem. If you want to ask questions like, "what machines have a connection to host X in netstat", using existing Cfengine infrastructure, you would need to write a policy which sets a class and reports on that class, and then push the policy.

We want flexibility where anyone in the company could ask these questions and get instantaneous results. You shouldn't have to be a Cfengine administrator to figure out the above question.

First generation of our central collection of data depended upon RSYNC. The client would rsync files from /var/cfengine/outgoing to the Cfengine policy server; the policy server would then RSYNC to two central machines.
- This does not scale. At 60 files per machine, x 10k machines = 600,000 files. At a 5 minute update interval, RSYNC became completely random small I/O – disks were pegged at 100% utilization performing near zero throughput.

We need an API that we can extract data out from the local core, datacenter, or globally from all datacenters. It would be nice if we could query this data from anywhere. What types of questions are we trying to answer?

What machine has experienced hardware failure?
Where do I have something NFS mounted?
What sudoers groups are active on which machines?
Where does Mike Svoboda have an account installed on machines?
What version of XXX package is installed on every machine?
What do the runit scripts for various services look like?
What version of RedHat Linux is running? CentOS? RHEL5? RHEL6.1? RHEL6.4?
What account profile is installed?
How many CPUs / Cores / Memory / Disks / Any hardware profiling you could want, across all sites? Globally across all datacenter?
What version of /usr/local/linkedin is installed on all machines, in all datacenter?
What machines have a TCP connection to machine X?
How many JVMs do I have running on every machine?

If a file exists on the filesystem that could possibly contain useful information, its candidate for inclusion into the Redis Caches that can be crawled via Sysops API. This data can be returned from tens of thousands of machines in seconds. Automation can be built on top of this to leverage instant answers to any problem you could imagine.

Redis Server

We decided to implement an in-memory key-value store called Redis. http://redis.io/
Redis is similar to memcache, but offers additional features that we found useful out of the box.
- Complex data structures (lists, dictionaries, etc.)
- Messaging system (Think activemq).
- Persistent caches, which are saved to disk. Shutting down the server does not lose data. On startup, caches are read from disk.

Redis Server Configuration

Cfengine policy which administrates Redis on the MPS.

mps_redis_servers.cf

msvoboda-mn:generic_cf-agent_policies msvoboda$ cat mps_redis_servers.cf

bundle agent mps_redis_servers

{

vars:

master_policy_servers::

"redis_passwd" string => "redis:x:2147483264:2147483264:Redis User:/export/content/redis:/sbin/nologin";

"redis_group" string => "redis:x:2147483264:";

"redis_shadow" string => "redis:!!:::::::";

files:

master_policy_servers::

"/etc/passwd"

handle => "mps_redis_etc_passwd",

edit_line => append_if_no_line("$(redis_passwd)"),

classes => if_repaired("etc_passwd_modified");

master_policy_servers::

"/etc/group"

handle => "mps_redis_etc_group",

edit_line => append_if_no_line("$(redis_group)"),

classes => if_repaired("etc_group_modified");

master_policy_servers::

"/etc/shadow"

handle => "mps_redis_etc_shadow",

edit_line => append_if_no_line("$(redis_shadow)"),

classes => if_repaired("etc_shadow_modified");

master_policy_servers::

"/etc/init.d/redis"

handle => "mps_setup_etc_initd_redis",

perms => mog("0755","root","root"),

copy_from => backup_cp_md5_compare("/var/cfengine/inputs/config-linux/mps_redis_servers/redis"),

classes => if_repaired("redis_server_modified");

master_policy_servers::

"/etc/redis.conf"

handle => "mps_etc_redis_conf",

perms => mog("0644","root","root"),

copy_from => backup_cp_md5_compare("/var/cfengine/inputs/config-linux/mps_redis_servers/redis.conf"),

classes => if_repaired("redis_server_modified");

master_policy_servers::

"/export/content/redis/."

handle => "mps_export_content_redis",

create => "true",

perms => mog("0755","redis","redis"),

classes => if_repaired("redis_server_modified");

master_policy_servers::

"/export/content/redis/data/."

handle => "mps_export_content_redis_data",

create => "true",

perms => mog("0755","redis","redis"),

classes => if_repaired("redis_server_modified");

master_policy_servers::

"/export/content/redis/logs/."

handle => "mps_export_content_redis_logs",

create => "true",

perms => mog("0755","redis","redis"),

classes => if_repaired("redis_server_modified");

master_policy_servers::

"/etc/logrotate.d/redis"

handle => "mps_etc_logrotate_d_redis",

perms => mog("0644","root","root"),

copy_from => backup_cp_md5_compare("/var/cfengine/inputs/config-linux/mps_redis_servers/logrotate-redis"),

classes => if_repaired("redis_server_modified");

master_policy_servers::

"/etc/cron.hourly/redis"

handle => "mps_etc_cron_hourly_redis",

perms => mog("0755","root","root"),

copy_from => backup_cp_md5_compare("/var/cfengine/inputs/config-linux/mps_redis_servers/cron_hourly_redis"),

classes => if_repaired("redis_server_modified");

commands:

redis_server_modified::

"/etc/init.d/redis restart",

comment => "mps_redis_servers bounce redis";

services:

master_policy_servers::

"redis"

handle => "mps_verify_redis_service",

service_policy => "start";

packages:

master_policy_servers::

"LINK-redis"

package_policy => "addupdate",

package_method => yum,

package_select => "==",

package_architectures => { "x86_64", },

package_version => "2.6.10-1.el6",

classes => if_repaired("redis_server_modified");

reports:

etc_passwd_modified::

"cf3: The redis user was inserted into /etc/passwd on $(sys.host)";

etc_group_modified::

"cf3: The redis user was inserted into /etc/group on $(sys.host)";

etc_shadow_modified::

"cf3: The redis user was inserted into /etc/shadow on $(sys.host)";

redis_server_modified::

"cf3: The redis server was modified on $(sys.host). Executing a restart of redis.";

}

standard_services.cf

This has Cfengine actually monitoring the redis process, and will fire it up if it finds it offline. See the "services" promise above.

msvoboda-mn:generic_cf-agent_policies msvoboda$ cat standard_services.cf

bundle agent standard_services(service, state)

{

vars:

linux::

"startcommand[redis]" string => "/etc/init.d/redis start";

"stopcommand[redis]" string => "/etc/init.d/redis stop";

"processname[redis]" string => "/usr/sbin/redis-server";

classes:

"start" expression => strcmp("start","$(state)");

"stop" expression => strcmp("stop","$(state)");

processes:

start::

".*$(processname[$(service)]).*"

comment => "Verify that the service appears in the process table",

restart_class => "restart_$(service)";

stop::

".*$(processname[$(service)]).*"

comment => "Verify that the service does not appear in the process table",

process_stop => "$(stopcommand[$(service)])",

signals => { "term", "kill"};

commands:

"$(startcommand[$(service)])"

comment => "Execute command to restart the $(service) service",

ifvarclass => "restart_$(service)";

}

/etc/redis.conf

only showing the relevant parts. Notice the 5gb max memory size, and that the working directory is stored under /export/content/redis/(logs|data)

[msvoboda@esv4-cfe-test ~]$ grep -v "#" /etc/redis.conf | grep -v ^$

daemonize yes

pidfile /var/run/redis/redis.pid

port 6379

bind 0.0.0.0

timeout 5

loglevel verbose

logfile /export/content/redis/logs/redis.log

databases 16

save 900 1

save 300 10

save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilename dump.rdb

dir /export/content/redis/data

maxclients 10000

maxmemory 5g

maxmemory-policy volatile-lru

maxmemory-samples 3

appendonly no

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

lua-time-limit 5000

slowlog-log-slower-than 10000

slowlog-max-len 128

hash-max-ziplist-entries 512

hash-max-ziplist-value 64

list-max-ziplist-entries 512

list-max-ziplist-value 64

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 64

activerehashing yes

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb 64mb 60

client-output-buffer-limit pubsub 32mb 8mb 60

Log rotation

/etc/logrotate.d/redis

[msvoboda@esv4-cfe-test ~]$ cat /etc/logrotate.d/redis

/export/content/redis/logs/redis.log {

daily

rotate 5

copytruncate

compress

notifempty

missingok

}

/etc/cron.hourly/redis

[msvoboda@esv4-cfe-test ~]$ cat /etc/cron.hourly/redis

/usr/sbin/logrotate -f /etc/logrotate.d/redis -v > /dev/null 2>&1

Marco Marongiu

unread,

Apr 5, 2013, 8:49:00 AM4/5/13

to help-c...@googlegroups.com

Interesting stuff Mike! I read this first part, and planning to read the
other 6th. Good job!

-- M

Reply all

Reply to author

Forward

0 new messages