LinkedIn Redis API -- Part 1 -- Problem statement and Redis server configuration / policies

220 views
Skip to first unread message

Mike Svoboda

unread,
Apr 3, 2013, 12:54:21 PM4/3/13
to help-c...@googlegroups.com

Problem Statement

  • We have several use cases at LinkedIn where data across tens of thousands of machines needs to be centrally collected. Configuration management products (Chef, Puppet, Cfengine) are all good at taking a golden image and making thousands of machines identical, but they do not have a good way of centrally collecting arbitrary data.
  • Cfengine's own Enterprise product allows for central collection of "variable" data, or "class" data, but you can not collect arbitrary files off of the filesystem.  If you want to ask questions like, "what machines have a connection to host X in netstat", using existing Cfengine infrastructure, you would need to write a policy which sets a class and reports on that class, and then push the policy.
    • We want flexibility where anyone in the company could ask these questions and get instantaneous results.   You shouldn't have to be a Cfengine administrator to figure out the above question.
  • First generation of our central collection of data depended upon RSYNC. The client would rsync files from /var/cfengine/outgoing to the Cfengine policy server; the policy server would then RSYNC to two central machines.
    • This does not scale. At 60 files per machine, x 10k machines = 600,000 files. At a 5 minute update interval, RSYNC became completely random small I/O – disks were pegged at 100% utilization performing near zero throughput.
  • We need an API that we can extract data out from the local core, datacenter, or globally from all datacenters. It would be nice if we could query this data from anywhere. What types of questions are we trying to answer?
  1. What machine has experienced hardware failure?
  2. Where do I have something NFS mounted?
  3. What sudoers groups are active on which machines?
  4. Where does Mike Svoboda have an account installed on machines?
  5. What version of XXX package is installed on every machine?
  6. What do the runit scripts for various services look like?
  7. What version of RedHat Linux is running? CentOS? RHEL5? RHEL6.1? RHEL6.4?
  8. What account profile is installed?
  9. How many CPUs / Cores / Memory / Disks / Any hardware profiling you could want, across all sites? Globally across all datacenter?
  10. What version of /usr/local/linkedin is installed on all machines, in all datacenter?
  11. What machines have a TCP connection to machine X?
  12. How many JVMs do I have running on every machine?
  • If a file exists on the filesystem that could possibly contain useful information, its candidate for inclusion into the Redis Caches that can be crawled via Sysops API.  This data can be returned from tens of thousands of machines in seconds.  Automation can be built on top of this to leverage instant answers to any problem you could imagine. 

Redis Server

  • We decided to implement an in-memory key-value store called Redis. http://redis.io/
  • Redis is similar to memcache, but offers additional features that we found useful out of the box.
    • Complex data structures (lists, dictionaries, etc.)
    • Messaging system (Think activemq).
    • Persistent caches, which are saved to disk. Shutting down the server does not lose data. On startup, caches are read from disk.

Redis Server Configuration

Cfengine policy which administrates Redis on the MPS.

mps_redis_servers.cf

msvoboda-mn:generic_cf-agent_policies msvoboda$ cat mps_redis_servers.cf
bundle agent mps_redis_servers
{
vars:
    master_policy_servers::
        "redis_passwd"            string  =>       "redis:x:2147483264:2147483264:Redis User:/export/content/redis:/sbin/nologin";
        "redis_group"         string  =>       "redis:x:2147483264:";
        "redis_shadow"            string  =>       "redis:!!:::::::";
 
files:
    master_policy_servers::
        "/etc/passwd"
            handle              =>       "mps_redis_etc_passwd",
            edit_line           =>       append_if_no_line("$(redis_passwd)"),
            classes             =>       if_repaired("etc_passwd_modified");
 
    master_policy_servers::
        "/etc/group"
            handle              =>       "mps_redis_etc_group",
            edit_line           =>       append_if_no_line("$(redis_group)"),
            classes             =>       if_repaired("etc_group_modified");
 
    master_policy_servers::
        "/etc/shadow"
            handle              =>       "mps_redis_etc_shadow",
            edit_line           =>       append_if_no_line("$(redis_shadow)"),
            classes             =>       if_repaired("etc_shadow_modified");
 
    master_policy_servers::
        "/etc/init.d/redis"
            handle              =>       "mps_setup_etc_initd_redis",
            perms               =>       mog("0755","root","root"),
            copy_from           =>       backup_cp_md5_compare("/var/cfengine/inputs/config-linux/mps_redis_servers/redis"),
            classes             =>       if_repaired("redis_server_modified");
 
    master_policy_servers::
        "/etc/redis.conf"
            handle              =>       "mps_etc_redis_conf",
            perms               =>       mog("0644","root","root"),
            copy_from           =>       backup_cp_md5_compare("/var/cfengine/inputs/config-linux/mps_redis_servers/redis.conf"),
            classes             =>       if_repaired("redis_server_modified");
 
    master_policy_servers::
        "/export/content/redis/."
            handle              =>       "mps_export_content_redis",
            create              =>       "true",
            perms               =>       mog("0755","redis","redis"),
            classes             =>       if_repaired("redis_server_modified");
 
    master_policy_servers::
        "/export/content/redis/data/."   
            handle              =>       "mps_export_content_redis_data",
            create              =>       "true",
            perms               =>       mog("0755","redis","redis"),
            classes             =>       if_repaired("redis_server_modified");
 
    master_policy_servers::
        "/export/content/redis/logs/."   
            handle              =>       "mps_export_content_redis_logs",
            create              =>       "true",
            perms               =>       mog("0755","redis","redis"),
            classes             =>       if_repaired("redis_server_modified");
 
    master_policy_servers::
        "/etc/logrotate.d/redis"
            handle              =>       "mps_etc_logrotate_d_redis",
            perms               =>       mog("0644","root","root"),
            copy_from           =>       backup_cp_md5_compare("/var/cfengine/inputs/config-linux/mps_redis_servers/logrotate-redis"),
            classes             =>       if_repaired("redis_server_modified");
 
    master_policy_servers::
        "/etc/cron.hourly/redis"
            handle              =>       "mps_etc_cron_hourly_redis",
            perms               =>       mog("0755","root","root"),
            copy_from           =>       backup_cp_md5_compare("/var/cfengine/inputs/config-linux/mps_redis_servers/cron_hourly_redis"),
            classes             =>       if_repaired("redis_server_modified");
 
commands:
    redis_server_modified::
        "/etc/init.d/redis restart",
            comment             =>       "mps_redis_servers bounce redis";
 
services:
        master_policy_servers::
                "redis"
                        handle                          =>      "mps_verify_redis_service",
                        service_policy                  =>      "start";
 
packages:
    master_policy_servers::
        "LINK-redis"
            package_policy          =>       "addupdate",
            package_method          =>       yum,
            package_select          =>       "==",
            package_architectures       =>       { "x86_64", },
            package_version         =>       "2.6.10-1.el6",
            classes             =>       if_repaired("redis_server_modified");
 
reports:
    etc_passwd_modified::
        "cf3:  The redis user was inserted into /etc/passwd on $(sys.host)";
 
    etc_group_modified::
        "cf3:  The redis user was inserted into /etc/group on $(sys.host)";
 
    etc_shadow_modified::
        "cf3:  The redis user was inserted into /etc/shadow on $(sys.host)";
 
    redis_server_modified::
        "cf3:  The redis server was modified on $(sys.host).  Executing a restart of redis.";
}

standard_services.cf

  • This has Cfengine actually monitoring the redis process, and will fire it up if it finds it offline. See the "services" promise above.
msvoboda-mn:generic_cf-agent_policies msvoboda$ cat standard_services.cf
bundle agent standard_services(service, state)
{
vars:
    linux::
        "startcommand[redis]"     string  =>   "/etc/init.d/redis start";
        "stopcommand[redis]"      string  =>   "/etc/init.d/redis stop";
        "processname[redis]"      string  =>   "/usr/sbin/redis-server";
 
classes:
    "start" expression => strcmp("start","$(state)");
    "stop" expression => strcmp("stop","$(state)");
 
processes:
    start::
            ".*$(processname[$(service)]).*"
                    comment => "Verify that the service appears in the process table",
                restart_class => "restart_$(service)";
    stop::
            ".*$(processname[$(service)]).*"
                    comment => "Verify that the service does not appear in the process table",
                process_stop => "$(stopcommand[$(service)])",
                    signals => { "term", "kill"};
commands:
   "$(startcommand[$(service)])"
         comment => "Execute command to restart the $(service) service",
         ifvarclass => "restart_$(service)";
}

/etc/redis.conf

  • only showing the relevant parts. Notice the 5gb max memory size, and that the working directory is stored under /export/content/redis/(logs|data)

    [msvoboda@esv4-cfe-test ~]$ grep -v "#" /etc/redis.conf | grep -v ^$
    daemonize yes
    pidfile /var/run/redis/redis.pid
    port 6379
    bind 0.0.0.0
    timeout 5
    loglevel verbose
    logfile /export/content/redis/logs/redis.log
    databases 16
    save 900 1
    save 300 10
    save 60 10000
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /export/content/redis/data
    maxclients 10000
    maxmemory 5g
    maxmemory-policy volatile-lru
    maxmemory-samples 3
    appendonly no
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    lua-time-limit 5000
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-entries 512
    list-max-ziplist-value 64
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit slave 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60

Log rotation

  • /etc/logrotate.d/redis
[msvoboda@esv4-cfe-test ~]$ cat /etc/logrotate.d/redis
 
/export/content/redis/logs/redis.log {
    daily
    rotate 5
    copytruncate
    compress
    notifempty
    missingok
}
  • /etc/cron.hourly/redis
[msvoboda@esv4-cfe-test ~]$ cat /etc/cron.hourly/redis
 
/usr/sbin/logrotate -f /etc/logrotate.d/redis -v > /dev/null 2>&1

Marco Marongiu

unread,
Apr 5, 2013, 8:49:00 AM4/5/13
to help-c...@googlegroups.com
Interesting stuff Mike! I read this first part, and planning to read the
other 6th. Good job!

-- M
Reply all
Reply to author
Forward
0 new messages