Redis having issues as message broker between beat clients and logstash

[Andrew Freitas]

Hi,

Versions:

redis version 3.2.12

filebeat/packetbeat version 7.5.1

logstash version 7.5.1

For Beat clients

Dec 03 09:52:20 *beatclient_host* filebeat[20968]: 2020-12-03T09:52:19.996Z ERROR pipeline/output.go:100 Failed to connect to redis(tcp://*redis_host*:6379): dial
tcp *redis_ip*:6379: i/o timeout

For logstash

Dec 03 10:43:39 *logstash_host* logstash[671]: [2020-12-03T10:43:39,060][WARN ][logstash.inputs.redis ][CCP] Redis connection problem {:exception=>#<Redis::CannotConnectError: Error connecting to Redis on *redis_host*:6379 (Redis::TimeoutError)>}

The server running redis has 32gb system memory, with a cap on redis mem to 26gb

127.0.0.1:6379> CONFIG GET maxmemory
1) "maxmemory"
2) "25769803776"

Redis doesn't start up with any errors and it isn't particulary at a lack for system resources. Is there anything else I can check to see why redis is not allowing connections to it?

[Greg Andrews]

Wow Redis server v3.2.12 is pretty old.  June of 2018.  Did you prefer the older version rather than current versions for a specific reason?

The default configuration in  most of the out-of-the-box installations of Redis server is to listen on the localhost IP address, also known as 127.0.0.1.  This is a safety feature that allows software on the same machine to connect to the Redis server, but prevents software on other machines on the network from connecting to the Redis server.  The sample log lines from filebeat and logstash suggest they're on other machines, while your redis-cli is on the same machine.

To allow other machines to connect, the default configuration would need to be changed.  One starting point would be https://redis.io/topics/config

The middle of the page https://redis.io/topics/security gives the configuration file parameter that controls which IP address the Redis server listens on.  Also, the example configuration file supplied with the source code has explanation lines for the various parameters.  I don't know if you have that example file, since it's not clear how yours was installed on the machine.  (e.g., from a package?  compiled from source code and copied to its folders/directories on the disk?)

  -Greg

[Andrew Freitas]

Hi Greg,

No particular preference for using v3.2.12 lol, its just the latest version of redis we have stored in our pulp server at my place of work. Is there a particular newer version that you would recommend that brings many changes? As I understand, my current version of redis is single threaded, is this still the case?

Yes, I am aware of this default network setting and have commented it out so that redis carries out default behaviour of listening for every available network interface on server, see below

(redis service)

[root@*redis_host* ~]# systemctl status redis
● redis.service - Redis persistent key-value database
Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/redis.service.d
└─limit.conf
Active: active (running) since Wed 2020-12-02 14:43:43 UTC; 1 day 18h ago
Process: 26173 ExecStop=/usr/libexec/redis-shutdown (code=killed, signal=TERM)
Main PID: 26339 (redis-server)
CGroup: /system.slice/redis.service
└─26339 /usr/bin/redis-server *:6379

(network section of redis conf)

[root@*redis_host* ~]# cat /etc/redis.conf | grep bind
# By default, if no "bind" configuration directive is specified, Redis listens
# the "bind" configuration directive, followed by one or more IP addresses.
# bind 192.168.1.100 10.0.0.1
# bind 127.0.0.1 ::1
# internet, binding to all the interfaces is dangerous and will expose the
# following bind directive, that will force Redis to listen only into
#bind 127.0.0.1

Redis was installed via yum. And yes, beats/redis/logstash are all running on separate machines.

Andrew

On Friday, 4 December 2020 at 06:49:40 UTC hva...@gmail.com wrote:

Wow Redis server v3.2.12 is pretty old.  June of 2018.  Did you prefer the older version rather than current versions for a specific reason?

The default configuration in  most of the out-of-the-box installations of Redis server is to listen on the localhost IP address, also known as 127.0.0.1.  This is a safety feature that allows software on the same machine to connect to the Redis server, but prevents software on other machines on the network from connecting to the Redis server.  The sample log lines from filebeat and logstash suggest they're on other machines, while your redis-cli is on the same machine.

To allow other machines to connect, the default configuration would need to be changed.  One starting point would be https://redis.io/topics/config

The middle of the page https://redis.io/topics/security gives the configuration file parameter that controls which IP address the Redis server listens on.  Also, the example configuration file supplied with the source code has explanation lines for the various parameters.  I don't know if you have that example file, since it's not clear how yours was installed on the machine.  (e.g., from a package?  compiled from source code and copied to its folders/directories on the disk?)

  -Greg

[Greg Andrews]

Hi Andrew,

After seeing the description of your configuration, I compiled my copy of 3.2.12 and used the default config file with the `bind` line commented out.  I launched redis-server with the argument pointing to my modified config file, and verified with 'netstat -lnt' that it was listening to all addresses on port 6379.  Then I connected using redis-cli just as you did, on the same machine, using redis-cli's default host of 127.0.0.1.  And it worked well, just as it did for you.  'ping' and other commands returned happy responses.

Then I did a partial simulation of connecting from another machine.  I connected using redis-cli on the same machine, but I used the -h option to specify my machine's LAN ip address instead of 127.0.0.1.  Here's what happened:

redis-3.2.12$ src/redis-cli -h 192.168.1.103
192.168.1.103:6379> ping
(error) DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.

I was going to test from another machine on my network, but the error message is clear about what would happen.

Do you think your Redis server configuration could also be in 'protected mode'?

It looks like specifying `bind 0.0.0.0` will do the equivalent listening on all addresses without invoking protected mode.  Or give the 'bind' parameter the specific address of your Redis machine.  Or change the 'protected-mode' line in the config file and restart the process.

Hope this helps,

  -Greg

[Andrew Freitas]

Hi Greg,

The issue has now been resolved, turns out someone had manually enabled firewalld service which was blocking all traffic - as we have our infra hosted on Azure, security rules are set at subnet level rather than server. Thank you for your time.

I did, however, have another issue that I wanted to raise and my real problem. I can constantly see that redis input > redis output, this means that we're never able to clear redis queue and a substantial backlog always develops - meaning that data to elasticsearch always has a delay to it, varying from 1hr to a day (not ideal).

I have the following config on logstash, increasing pipeline.* values increases system resource usage but not necessarily performance

```

- pipeline.id: CCP
  path.config: "/etc/logstash/conf.d/CCP_logstash.conf"
  pipeline.workers: 200
  pipeline.batch.size: 1000

```

Threads correspond to connections logstash makes to redis server to poll data from (default batch_count: 125)

```

  redis {
host => " *redis_1* "
port => "6379"
data_type => "list"
key => "filebeat-ccp"
password => " *** "
threads => 12
}
redis {
host => " *redis_1* "
port => "6379"
data_type => "list"
key => "packetbeat-ccp"
password => " *** "
threads => 4
}
redis {
host => " *redis_2* "
port => "6379"
data_type => "list"
key => "filebeat-ccp"
password => " *** "
threads => 12
}
redis {
host => " *redis_2* "
port => "6379"
data_type => "list"
key => "packetbeat-ccp"
password => " *** "
threads => 4
}
redis {
host => " *redis_3* "
port => "6379"
data_type => "list"
key => "filebeat-ccp"
password => " *** "
threads => 12
}
redis {
host => " *redis_3* "
port => "6379"
data_type => "list"
key => "packetbeat-ccp"
password => " *** "
threads => 4
}

```

Was wondering if there was an optimal way to set up logstash to poll data from redis, particularly around batch_count, batch_size and threads I feel is where you could most add value to as docs around these 3 together are lacking from my searches on the internet.

Let me know if you can answer this here or want me to start a new conversation?

Andrew