Logstash errors

69 views
Skip to first unread message

Mat

unread,
Mar 9, 2021, 11:29:22 AMMar 9
to security-onion
Hi,

We are having issues after updating SO 16.04 and not too sure how to fix it. I think the template needs re-doing but cant remember how to carry that out. Squert is working but kibana dashboards wont load. Sudo so-status shows no errors.

Here are the logs from:

Logstash:

[2021-03-09T16:23:15,412][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-ossec-2021.03.09", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x1188aa39>], :response=>{"index"=>{"_index"=>"logstash-ossec-2021.03.09", "_type"=>"_doc", "_id"=>"shnMF3gBUfPOixOHtmOo", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [data.data] of type [text] in document with id 'shnMF3gBUfPOixOHtmOo'. Preview of field's value: '{checksum=b2f879d81028d35546e6431b3d707df92b2326d9, end=/sbin, id=1615306994, begin=/bin}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:224"}}}}}

Elasticsearch:
[2021-03-09T08:43:41,239][DEPRECATION][org.elasticsearch.deprecation.action.bulk.BulkRequestParser] [types removal] Specifying types in bulk requests is deprecated.
[2021-03-09T08:43:41,414][DEPRECATION][org.elasticsearch.deprecation.action.bulk.BulkRequestParser] [types removal] Specifying types in bulk requests is deprecated.
[2021-03-09T08:43:41,422][DEPRECATION][org.elasticsearch.deprecation.action.bulk.BulkRequestParser] [types removal] Specifying types in bulk requests is deprecated.
[2021-03-09T08:43:41,437][DEPRECATION][org.elasticsearch.deprecation.action.bulk.BulkRequestParser] [types removal] Specifying types in bulk requests is deprecated.


Hoping that you can help thanks,

Shane Mullins

unread,
Mar 9, 2021, 11:50:41 AMMar 9
to securit...@googlegroups.com
We are having the same issue. 

--
Please keep in mind that Security Onion 16.04 reaches End Of Life soon!
https://blog.securityonion.net/2020/10/6-month-eol-notice-for-security-onion.html
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/e2ef6df3-1a17-476c-8335-970a2e3d8f30n%40googlegroups.com.

Doug Burks

unread,
Mar 10, 2021, 4:45:58 PMMar 10
to securit...@googlegroups.com
We haven't seen this issue on any of our 16.04 installs.

Did you have any custom templates?

Were you sending any custom data?

Did you have any Wazuh agents?  If so, are they the same version as the Wazuh server on your Security Onion box?

If all else fails, you may just want to upgrade to Security Onion 2 as Security Onion 16.04 reaches End Of Life next month.

On Tue, Mar 9, 2021 at 11:29 AM Mat <jobem...@gmail.com> wrote:
--
Please keep in mind that Security Onion 16.04 reaches End Of Life soon!
https://blog.securityonion.net/2020/10/6-month-eol-notice-for-security-onion.html
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/e2ef6df3-1a17-476c-8335-970a2e3d8f30n%40googlegroups.com.


--
Doug Burks
Founder and CEO
Security Onion Solutions, LLC

Mat

unread,
Mar 11, 2021, 12:21:40 PMMar 11
to security-onion
Hi Doug,

  • It has the following templates and they have not been changed:
    •  ".kibana-event-log-7.9.3-template",
    •   ".logstash-management",
    •   ".management-beats",
    •   ".ml-anomalies-",
    •   ".ml-config",
    •   ".ml-inference-000002",
    •   ".ml-inference-000003",
    •   ".ml-meta",
    •   ".ml-notifications-000001",
    •   ".ml-state",
    •   ".ml-stats",
    •   ".monitoring-alerts-7",
    •   ".monitoring-beats",
    •   ".monitoring-es",
    •   ".monitoring-kibana",
    •   ".monitoring-logstash",
    •   ".slm-history",
    •   ".transform-internal-005",
    •   ".transform-notifications-000002",
    •   ".triggered_watches",
    •   ".watch-history-11",
    •   ".watches",
    •   "ilm-history",
    •   "kibana",
    •   "logstash",
    •   "logstash-beats",
    •   "logstash-ossec"
  • There is no custom data
  • The environment is all wazuh agents and syslog with the agents running 3.9.3 i believe.
  • Has all the latest updates including docker
Im still looking into this, will update with more info

Mat

unread,
Mar 11, 2021, 12:27:55 PMMar 11
to security-onion
**More info**

Sostat output: the elastic instance is on yellow and i cant request any information from it like curl localhost:9200/_cat/indices and get this message:
The following error was encountered while trying to retrieve the URL: <a href="http://localhost:9200/_cat/indices">http://localhost:9200/_cat/indices</a></p>

<blockquote id="error">
<p><b>Connection to ::1 failed.</b></p>
</blockquote>

<p id="sysmsg">The system returned: <i>(111) Connection refused</i></p>

<p>The remote host or network may be down. Please try the request again.</p>

=========================================================================
Last update
=========================================================================
Commandline: apt install ./docker-ce-rootless-extras_20.10.5~3-0~ubuntu-xenial_amd64.deb
Requested-By: viper (1000)
Upgrade: docker-ce-rootless-extras:amd64 (5:20.10.4~3-0~ubuntu-xenial, 5:20.10.5~3-0~ubuntu-xenial)
End-Date: 2021-03-11  16:41:32

Start-Date: 2021-03-11  16:41:54
Commandline: apt-get -y dist-upgrade
Requested-By: viper (1000)
Upgrade: python2.7-dev:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), git-man:amd64 (1:2.7.4-0ubuntu1.9, 1:2.7.4-0ubuntu1.10), python2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), libpython2.7:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), python2.7:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), git:amd64 (1:2.7.4-0ubuntu1.9, 1:2.7.4-0ubuntu1.10), libpython2.7-dev:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), securityonion-suricata:amd64 (5.0.5-1ubuntu1securityonion2, 5.0.6-1ubuntu1securityonion1), libglib2.0-bin:amd64 (2.48.2-0ubuntu4.6, 2.48.2-0ubuntu4.7), wpasupplicant:amd64 (2.4-0ubuntu6.7, 2.4-0ubuntu6.8), libglib2.0-data:amd64 (2.48.2-0ubuntu4.6, 2.48.2-0ubuntu4.7), libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.16, 2.7.12-1ubuntu0~16.04.18), libglib2.0-0:amd64 (2.48.2-0ubuntu4.6, 2.48.2-0ubuntu4.7)
End-Date: 2021-03-11  16:42:11


=========================================================================
Elasticsearch
=========================================================================

Elasticsearch is running.

Cluster Name: "cg-mstr-vipertd"
Cluster Status: "yellow"
Total Nodes: 1
Failed Nodes: 0
Total Indices: 294
Total Shards: 314
Total Documents: 1968562317
Total Size: 1992169MB
Free Memory: 2%
Total Number of Events: 725903763
Avg. Event Size (In Bytes): 1234

CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O        PIDS
268383a7756d   so-elasticsearch   5.40%     5.518GiB / 15.64GiB   35.27%    265MB / 463MB   7.63GB / 425MB   133


=========================================================================
Logstash
=========================================================================

Logstash is running.

CONTAINER ID   NAME          CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O         PIDS
39388c0eb03b   so-logstash   10.49%    1.125GiB / 15.64GiB   7.19%     140MB / 257MB   97.8MB / 11.3MB   150

Logstash Queue Stats:

Queue Type: memory
Queue settings can be modified in /etc/logstash/logstash.yml.

Event Summary (since restart):

Events In: 112218
Events Out: 112218



=========================================================================
Kibana
=========================================================================

Kibana is running.

CONTAINER ID   NAME        CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O     PIDS
77c8b5235967   so-kibana   0.94%     212.4MiB / 15.64GiB   1.33%     32.1MB / 9.63MB   60.1MB / 0B   12


=========================================================================
ElastAlert
=========================================================================

ElastAlert is running.

CONTAINER ID   NAME            CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O         PIDS
3e1b11307a10   so-elastalert   0.02%     71.96MiB / 15.64GiB   0.45%     229kB / 305kB   27.1MB / 12.3kB   13


=========================================================================
Curator
=========================================================================

Curator is running.

CONTAINER ID   NAME         CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
ff983f3b8089   so-curator   0.00%     14.73MiB / 15.64GiB   0.09%     425MB / 931kB   14.8MB / 0B   1


=========================================================================
syslog-ng stats
=========================================================================

SourceName;SourceId;SourceInstance;State;Type;Number
destination;d_syslog;;a;processed;333
destination;d_console_all;;a;processed;16
dst.tcp;d_logstash#0;tcp,127.0.0.1:6050;a;dropped;62106
dst.tcp;d_logstash#0;tcp,127.0.0.1:6050;a;processed;240580
destination;d_cron;;a;processed;232
destination;d_error;;a;processed;15
center;;queued;a;processed;241832
destination;d_auth;;a;processed;530
destination;d_daemon;;a;processed;101
global;payload_reallocs;;a;processed;13
destination;d_xconsole;;a;processed;16
destination;d_debug;;a;processed;9
destination;d_logstash;;a;processed;240580

WARNING! syslog-ng reports drops!
dst.tcp;d_logstash#0;tcp,127.0.0.1:6050;a;dropped;62106


=========================================================================
Version Information
=========================================================================

Ubuntu 16.04.7 LTS
securityonion-sostat 20120722-0ubuntu0securityonion148

Doug Burks

unread,
Mar 12, 2021, 11:54:00 AMMar 12
to securit...@googlegroups.com
A fully updated Security Onion 16.04 installation should be running Wazuh 3.9.5 so you'll want to make sure that your Wazuh agents are at that version as well:

Doug Burks

unread,
Mar 12, 2021, 11:57:28 AMMar 12
to securit...@googlegroups.com
If the elastic instance is yellow, then you'll want to check the elasticsearch logs for any additional clues:

As a reminder, Security Onion 16.04 reaches End Of Life next month, so it may be a better use of your time to go ahead and move to Security Onion 2.

Reply all
Reply to author
Forward
0 new messages