Logstash errors

Mat

unread,

Mar 9, 2021, 11:29:22 AM3/9/21

to security-onion

Hi,

We are having issues after updating SO 16.04 and not too sure how to fix it. I think the template needs re-doing but cant remember how to carry that out. Squert is working but kibana dashboards wont load. Sudo so-status shows no errors.

Here are the logs from:

Logstash:

[2021-03-09T16:23:15,412][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-ossec-2021.03.09", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x1188aa39>], :response=>{"index"=>{"_index"=>"logstash-ossec-2021.03.09", "_type"=>"_doc", "_id"=>"shnMF3gBUfPOixOHtmOo", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [data.data] of type [text] in document with id 'shnMF3gBUfPOixOHtmOo'. Preview of field's value: '{checksum=b2f879d81028d35546e6431b3d707df92b2326d9, end=/sbin, id=1615306994, begin=/bin}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:224"}}}}}

Elasticsearch:

[2021-03-09T08:43:41,239][DEPRECATION][org.elasticsearch.deprecation.action.bulk.BulkRequestParser] [types removal] Specifying types in bulk requests is deprecated.

[2021-03-09T08:43:41,414][DEPRECATION][org.elasticsearch.deprecation.action.bulk.BulkRequestParser] [types removal] Specifying types in bulk requests is deprecated.

[2021-03-09T08:43:41,422][DEPRECATION][org.elasticsearch.deprecation.action.bulk.BulkRequestParser] [types removal] Specifying types in bulk requests is deprecated.

[2021-03-09T08:43:41,437][DEPRECATION][org.elasticsearch.deprecation.action.bulk.BulkRequestParser] [types removal] Specifying types in bulk requests is deprecated.

Hoping that you can help thanks,

Shane Mullins

unread,

Mar 9, 2021, 11:50:41 AM3/9/21

to securit...@googlegroups.com

We are having the same issue.

--
Please keep in mind that Security Onion 16.04 reaches End Of Life soon!
https://blog.securityonion.net/2020/10/6-month-eol-notice-for-security-onion.html
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/e2ef6df3-1a17-476c-8335-970a2e3d8f30n%40googlegroups.com.

Doug Burks

unread,

Mar 10, 2021, 4:45:58 PM3/10/21

to securit...@googlegroups.com

We haven't seen this issue on any of our 16.04 installs.

Did you have any custom templates?

Were you sending any custom data?

Did you have any Wazuh agents? If so, are they the same version as the Wazuh server on your Security Onion box?

If all else fails, you may just want to upgrade to Security Onion 2 as Security Onion 16.04 reaches End Of Life next month.

On Tue, Mar 9, 2021 at 11:29 AM Mat <jobem...@gmail.com> wrote:

--

Please keep in mind that Security Onion 16.04 reaches End Of Life soon!
https://blog.securityonion.net/2020/10/6-month-eol-notice-for-security-onion.html
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/e2ef6df3-1a17-476c-8335-970a2e3d8f30n%40googlegroups.com.

--

Doug Burks
Founder and CEO
Security Onion Solutions, LLC

Mat

unread,

Mar 11, 2021, 12:21:40 PM3/11/21

to security-onion

Hi Doug,

It has the following templates and they have not been changed:
- ".kibana-event-log-7.9.3-template",
- ".logstash-management",
- ".management-beats",
- ".ml-anomalies-",
- ".ml-config",
- ".ml-inference-000002",
- ".ml-inference-000003",
- ".ml-meta",
- ".ml-notifications-000001",
- ".ml-state",
- ".ml-stats",
- ".monitoring-alerts-7",
- ".monitoring-beats",
- ".monitoring-es",
- ".monitoring-kibana",
- ".monitoring-logstash",
- ".slm-history",
- ".transform-internal-005",
- ".transform-notifications-000002",
- ".triggered_watches",
- ".watch-history-11",
- ".watches",
- "ilm-history",
- "kibana",
- "logstash",
- "logstash-beats",
- "logstash-ossec"
There is no custom data
The environment is all wazuh agents and syslog with the agents running 3.9.3 i believe.
Has all the latest updates including docker

Im still looking into this, will update with more info

Mat

unread,

Mar 11, 2021, 12:27:55 PM3/11/21

to security-onion

**More info**

Sostat output: the elastic instance is on yellow and i cant request any information from it like curl localhost:9200/_cat/indices and get this message:

The following error was encountered while trying to retrieve the URL: <a href="http://localhost:9200/_cat/indices">http://localhost:9200/_cat/indices</a>

Connection to ::1 failed.

</blockquote>

The system returned: (111) Connection refused

The remote host or network may be down. Please try the request again.

=========================================================================

Last update

=========================================================================

Commandline: apt install ./docker-ce-rootless-extras_20.10.5~3-0~ubuntu-xenial_amd64.deb

Requested-By: viper (1000)

Upgrade: docker-ce-rootless-extras:amd64 (5:20.10.4~3-0~ubuntu-xenial, 5:20.10.5~3-0~ubuntu-xenial)

End-Date: 2021-03-11 16:41:32

Start-Date: 2021-03-11 16:41:54

Commandline: apt-get -y dist-upgrade

Requested-By: viper (1000)

Upgrade: python2.7-dev:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), git-man:amd64 (1:2.7.4-0ubuntu1.9, 1:2.7.4-0ubuntu1.10), python2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), libpython2.7:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), python2.7:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), git:amd64 (1:2.7.4-0ubuntu1.9, 1:2.7.4-0ubuntu1.10), libpython2.7-dev:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), securityonion-suricata:amd64 (5.0.5-1ubuntu1securityonion2, 5.0.6-1ubuntu1securityonion1), libglib2.0-bin:amd64 (2.48.2-0ubuntu4.6, 2.48.2-0ubuntu4.7), wpasupplicant:amd64 (2.4-0ubuntu6.7, 2.4-0ubuntu6.8), libglib2.0-data:amd64 (2.48.2-0ubuntu4.6, 2.48.2-0ubuntu4.7), libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), libglib2.0-0:amd64 (2.48.2-0ubuntu4.6, 2.48.2-0ubuntu4.7)

End-Date: 2021-03-11 16:42:11

=========================================================================

Elasticsearch

=========================================================================

Elasticsearch is running.

Cluster Name: "cg-mstr-vipertd"

Cluster Status: "yellow"

Total Nodes: 1

Failed Nodes: 0

Total Indices: 294

Total Shards: 314

Total Documents: 1968562317

Total Size: 1992169MB

Free Memory: 2%

Total Number of Events: 725903763

Avg. Event Size (In Bytes): 1234

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS

268383a7756d so-elasticsearch 5.40% 5.518GiB / 15.64GiB 35.27% 265MB / 463MB 7.63GB / 425MB 133

=========================================================================

Logstash

=========================================================================

Logstash is running.

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS

39388c0eb03b so-logstash 10.49% 1.125GiB / 15.64GiB 7.19% 140MB / 257MB 97.8MB / 11.3MB 150

Logstash Queue Stats:

Queue Type: memory

Queue settings can be modified in /etc/logstash/logstash.yml.

Event Summary (since restart):

Events In: 112218

Events Out: 112218

=========================================================================

Kibana

=========================================================================

Kibana is running.

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS

77c8b5235967 so-kibana 0.94% 212.4MiB / 15.64GiB 1.33% 32.1MB / 9.63MB 60.1MB / 0B 12

=========================================================================

ElastAlert

=========================================================================

ElastAlert is running.

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS

3e1b11307a10 so-elastalert 0.02% 71.96MiB / 15.64GiB 0.45% 229kB / 305kB 27.1MB / 12.3kB 13

=========================================================================

Curator

=========================================================================

Curator is running.

CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS

ff983f3b8089 so-curator 0.00% 14.73MiB / 15.64GiB 0.09% 425MB / 931kB 14.8MB / 0B 1

=========================================================================

syslog-ng stats

=========================================================================

SourceName;SourceId;SourceInstance;State;Type;Number

destination;d_syslog;;a;processed;333

destination;d_console_all;;a;processed;16

dst.tcp;d_logstash#0;tcp,127.0.0.1:6050;a;dropped;62106

dst.tcp;d_logstash#0;tcp,127.0.0.1:6050;a;processed;240580

destination;d_cron;;a;processed;232

destination;d_error;;a;processed;15

center;;queued;a;processed;241832

destination;d_auth;;a;processed;530

destination;d_daemon;;a;processed;101

global;payload_reallocs;;a;processed;13

destination;d_xconsole;;a;processed;16

destination;d_debug;;a;processed;9

destination;d_logstash;;a;processed;240580

WARNING! syslog-ng reports drops!

dst.tcp;d_logstash#0;tcp,127.0.0.1:6050;a;dropped;62106

=========================================================================

Version Information

=========================================================================

Ubuntu 16.04.7 LTS

securityonion-sostat 20120722-0ubuntu0securityonion148

Doug Burks

unread,

Mar 12, 2021, 11:54:00 AM3/12/21

to securit...@googlegroups.com

A fully updated Security Onion 16.04 installation should be running Wazuh 3.9.5 so you'll want to make sure that your Wazuh agents are at that version as well:

https://docs.securityonion.net/en/16.04/upgrade.html#wazuh

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/b1997918-60bb-4289-9b1f-c5f10adbd210n%40googlegroups.com.

Doug Burks

unread,

Mar 12, 2021, 11:57:28 AM3/12/21

to securit...@googlegroups.com

If the elastic instance is yellow, then you'll want to check the elasticsearch logs for any additional clues:

https://docs.securityonion.net/en/16.04/elasticsearch.html#logs

As a reminder, Security Onion 16.04 reaches End Of Life next month, so it may be a better use of your time to go ahead and move to Security Onion 2.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/235d751d-e37e-410c-a48e-727211b7ab37n%40googlegroups.com.

Reply all

Reply to author

Forward