Filebeat lumberjack protocol error

[Hafeezy2j]

Hello,

I'm following Setting up SSL for Filebeat and Logstash  to setup SSL between and filebear and logstash and I'm getting lumberjack protocol error on filebeat and routines:OPENSSL_internal:WRONG_VERSION_NUMBER  on logstash server. Without SSL, everything works fine..

Running following:
elasticsearch-6.1.3
logstash-6.1.3
kibana-6.1.3
Wazuh-manager 3.1.0
wazuh-api 3.1.0
filebeat 5.6.7


Distro: Linux  3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux



tail log of  /var/log/filebeat/filebeat.log

2018-02-05T13:20:57-05:00 ERR Connecting error publishing events (retrying): dial tcp 172.XX.XX.67:5000: getsockopt: connection refused
2018-02-05T13:21:07-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:21:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:21:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: write tcp 172.XX.XX.60:60754->172.XX.XX.67:5000: write: connection reset by peer
2018-02-05T13:21:58-05:00 INFO Error publishing events (retrying): write tcp 172.XX.XX.60:60754->172.XX.XX.67:5000: write: connection reset by peer
2018-02-05T13:22:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.write_bytes=285256 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:22:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:22:58-05:00 ERR Connecting error publishing events (retrying): dial tcp 172.XX.XX.67:5000: getsockopt: connection refused
2018-02-05T13:23:07-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:23:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:23:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: lumberjack protocol error
2018-02-05T13:23:58-05:00 INFO Error publishing events (retrying): lumberjack protocol error
2018-02-05T13:24:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=388290 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:24:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:24:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: lumberjack protocol error
2018-02-05T13:24:58-05:00 INFO Error publishing events (retrying): lumberjack protocol error
2018-02-05T13:25:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=387931 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:25:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:25:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: lumberjack protocol error
2018-02-05T13:25:58-05:00 INFO Error publishing events (retrying): lumberjack protocol error
2018-02-05T13:26:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=388315 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:26:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:26:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: lumberjack protocol error
2018-02-05T13:26:58-05:00 INFO Error publishing events (retrying): lumberjack protocol error
2018-02-05T13:27:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=387545 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:27:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:27:59-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: write tcp 172.XX.XX.60:60766->172.XX.XX.67:5000: write: connection reset by peer
2018-02-05T13:27:59-05:00 INFO Error publishing events (retrying): write tcp 172.XX.XX.60:60766->172.XX.XX.67:5000: write: connection reset by peer
2018-02-05T13:28:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.write_bytes=357656 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_but_not_acked_events=1743
 tail -f /var/log/filebeat/filebeat
2018-02-05T13:20:57-05:00 ERR Connecting error publishing events (retrying): dial tcp 172.XX.XX.67:5000: getsockopt: connection refused
2018-02-05T13:21:07-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:21:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:21:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: write tcp 172.XX.XX.60:60754->172.XX.XX.67:5000: write: connection reset by peer
2018-02-05T13:21:58-05:00 INFO Error publishing events (retrying): write tcp 172.XX.XX.60:60754->172.XX.XX.67:5000: write: connection reset by peer
2018-02-05T13:22:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.write_bytes=285256 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:22:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:22:58-05:00 ERR Connecting error publishing events (retrying): dial tcp 172.XX.XX.67:5000: getsockopt: connection refused
2018-02-05T13:23:07-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:23:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:23:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: lumberjack protocol error
2018-02-05T13:23:58-05:00 INFO Error publishing events (retrying): lumberjack protocol error
2018-02-05T13:24:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=388290 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:24:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:24:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: lumberjack protocol error
2018-02-05T13:24:58-05:00 INFO Error publishing events (retrying): lumberjack protocol error
2018-02-05T13:25:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=387931 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:25:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:25:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: lumberjack protocol error
2018-02-05T13:25:58-05:00 INFO Error publishing events (retrying): lumberjack protocol error
2018-02-05T13:26:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=388315 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:26:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:26:58-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: lumberjack protocol error
2018-02-05T13:26:58-05:00 INFO Error publishing events (retrying): lumberjack protocol error
2018-02-05T13:27:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=6 libbeat.logstash.publish.write_bytes=387545 libbeat.logstash.published_but_not_acked_events=1743
2018-02-05T13:27:37-05:00 INFO No non-zero metrics in the last 30s
2018-02-05T13:27:59-05:00 ERR Failed to publish events (host: 172.XX.XX.67:5000:10200), caused by: write tcp 172.XX.XX.60:60766->172.XX.XX.67:5000: write: connection reset by peer
2018-02-05T13:27:59-05:00 INFO Error publishing events (retrying): write tcp 172.XX.XX.60:60766->172.XX.XX.67:5000: write: connection reset by peer
2018-02-05T13:28:07-05:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.write_bytes=357656 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_but_not_acked_events=1743


tail log of /var/log/logstash/logstash-plain.log

[2018-02-05T13:23:58,383][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5000, remote: 172.XX.XX.60:60758] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2018-02-05T13:24:58,502][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5000, remote: 172.XX.XX.60:60760] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2018-02-05T13:25:58,691][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5000, remote: 172.XX.XX.60:60762] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2018-02-05T13:26:58,828][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5000, remote: 172.XX.XX.60:60764] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER

Thanks,
Hafeez

[javier...@wazuh.com]

Hi,

sorry for the late response.

I have been digging into this and unfortunately haven't found something useful...

Even so, it seems there may be some problem with filebeat-logstash connectivity? Make sure they can see each other.

Also, can you paste your filebeat and logstash configuration, maybe I can see something wrong in there.

Regards!