Filebeat not showing in beats dashboard

[bughatti]

All I have been playing with SO the last couple of days, first time user. I installed the ISO 16.04.4.3 and was able to get it up and running. I am currently only using it for the Bro syslog page and the Beats page. So far I have all my network equipment sending syslog data to it fine and I have 5 windows machines sending Winlogbeat data to it fine. The issue I am having is filebeat on linux Ubuntu machines. All my Ubuntu machines are 16.04.3 Desktop images running in ESXi 6.5. The Windows machines are a mix of different OS's and some are VMs and some are physical. I followed the install instructions for filebeat from SO links, I have validated that the service is running and that I can telnet to my SO server via 5044.

My filebeat config file
- type: log

# Change to true to enable this input configuration.
enabled: false

# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/auth.log
- /var/log/syslog
#- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*

output.logstash:
# The Logstash hosts
hosts: ["10.55.99.237:5044"]
bulk_max_size:1024

When I run sudo filebeat -e on the client I get

2018-08-07T12:15:37.572+0300 INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.2
2018-08-07T12:15:37.573+0300 INFO pipeline/module.go:81 Beat name: NetMon
2018-08-07T12:15:37.573+0300 INFO instance/beat.go:315 filebeat start running.
2018-08-07T12:15:37.573+0300 INFO registrar/registrar.go:117 Loading registrar data from /var/lib/filebeat/registry
2018-08-07T12:15:37.573+0300 INFO registrar/registrar.go:124 States Loaded from registrar: 0
2018-08-07T12:15:37.573+0300 WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-08-07T12:15:37.573+0300 INFO crawler/crawler.go:48 Loading Inputs: 1
2018-08-07T12:15:37.573+0300 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 0
2018-08-07T12:15:37.574+0300 INFO cfgfile/reload.go:122 Config reloader started
2018-08-07T12:15:37.574+0300 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-08-07T12:15:37.574+0300 INFO cfgfile/reload.go:214 Loading of config files completed.
2018-08-07T12:16:07.576+0300 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":4}},"total":{"ticks":10,"time":{"ms":20},"value":10},"user":{"ticks":10,"time":{"ms":16}}},"info":{"ephemeral_id":"73670594-2f1a-48da-ba68-eacc1a12ba40","uptime":{"ms":30011}},"memstats":{"gc_next":4473924,"memory_alloc":2987936,"memory_total":2987936,"rss":21241856}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0,"5":0.02,"norm":{"1":0,"15":0,"5":0.01}}}}}}

I did not read anywhere that I would need to do extra configuration steps in SO for filebeat when Winlogbeat works out of the box.

Any help would be greatly appreciated

[Wes Lambert]

You may want to try changing this to "true".

enabled: false 

Thanks,

Wes 

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[bughatti]

Copy, I did that and it is still not showing in the beats dashboard like the 5 windows servers I have. Is there a command I can check on the logstash server to validate it is picking up filebeat? Even if I go to the discover tab and select logstash-beat and put in the IP address, it is not picking it up.

[bughatti]

Am I required to setup a Logstash configuration pipeline in security onion to get it to read filebeat, where as winlogstash works out of the box?

[Wes Lambert]

You shouldn't have to set up Logstash config unless you want it specially parsed.

A couple things you can try:

-Enable debug logging on the Filebeat side to ensure messages are actually getting (a) picked up from the host on whic FB is running, and (b) sent to Logstash.

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html#configuration-logging

-Enable dead_letter_queue to see if messages are not getting processed:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Logstash#dead-letter-queue

-Check the Logstash log (/var/log/logstash/logstash.log) for clues

Thanks,

Wes

On Tue, Aug 7, 2018 at 10:55 AM bughatti <enochb...@gmail.com> wrote:

Am I required to setup a Logstash configuration pipeline in security onion to get it to read filebeat, where as winlogstash works out of the box?

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[bughatti]

Ok, so I went ahead and downloaded the newest version 5.1 I am now seeing the linux machines under Discover > logstash-beats. Under Dashboard > Host Hunting > Beats I see the Beats - Log Count increasing but I still do not see any data under Beats - Computer name. Since it is a new install I will give it some time to see if it populates correctly.

[bughatti]

So after a few hours of monitoring and slowly adding other machines in SO, it looks like it is an error with reading the data. I see all my filebeat linux machines under discover > logstash-beats but none of them show under Beats - Computer Names in the Beats dashboard. Oddly under discover, I see the hostname under the beat.name column so filebeat seems to be sending the data correctly but something in the Host Hunting > Beats dashboard is not able to interpret the host name correctly to show the linux machines under Computer Names.

[Wes Lambert]

I'll have to test this and see if I can replicate on my side.

Thanks,

Wes

On Wed, Aug 8, 2018 at 7:12 AM bughatti <enochb...@gmail.com> wrote:

So after a few hours of monitoring and slowly adding other machines in SO, it looks like it is an error with reading the data.  I see all my filebeat linux machines under discover > logstash-beats but none of them show under Beats - Computer Names in the Beats dashboard.  Oddly under discover, I see the hostname under the beat.name column so filebeat seems to be sending the data correctly but something in the Host Hunting > Beats dashboard is not able to interpret the host name correctly to show the linux machines under Computer Names.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[zal sag]

Hello all,
I have the very same question, posted under another thread opened last night. Wes, you probably remember.
filebeat scans /var/log/secure

On Kibana Under discover:
message:Oct 11 05:39:01 ubuntu1 CRON[15279]: pam_unix(cron:session): session closed for user root beat_host.name:ubuntu1 prospector.type:log tags:beat, beats_input_codec_plain_applied input.type:log logstash_time:0.001 offset:49,283 source:/var/log/secure @timestamp:October 11th 2018, 08:39:08.409 beat.name:ubuntu1 beat.version:6.4.2 beat.hostname:ubuntu1 @version:1 _id:279HYmYBFHGlbBizxSKk _type:doc _index:so:logstash-beats-2018.10.11 _score:

But no computer name (ubuntu1) under Dashboard/Beats "Beats - Computer Names"

Thank you,
Zali

[zal sag]

I had the same issue. The solution I found was to change computerame.keyword to beat.hostame. Not sure if correct...

[tc]

It looks like the fields from filebeat and winlogbeat are different. The computer_name, event_id etc. are specific to the winlogbeat event log records field. https://www.elastic.co/guide/en/beats/winlogbeat/current/exported-fields-eventlog.html
Also note the text in Winlogbeat for computer_name: "The name of the computer that generated the record. When using Windows event forwarding, this name can differ from the beat.hostname."

Filebeat is using a different set of fields, for example https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-log.html

So I guess there has to be a different set of logstash-filters in SO for filebeat from non-Windows machines.

/TC