Filebeat issue on all-in-one deployment

[Draenoel Leonard]

Hello,

It seems like my filebeat is having issues communicating with Elasticsearch.
when i perform "filebeat test output" i get the following:

filebeat test output
elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...
    security... WARN server's certificate chain verification is disabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: 400 Bad Request: {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [_license], must not start with '_'.","index_uuid":"_na_","index":"_license"}],"type":"invalid_index_name_exception","reason":"Invalid index name [_license], must not start with '_'.","index_uuid":"_na_","index":"_license"},"status":400}

only error in my filebeat log:
2022-02-10T17:23:38.549-0600    WARN    [cfgwarn]       template/config.go:88   DEPRECATED: Please migrate your JSON templates from legacy template format to composable index template. Will be removed in version: 8.0.0
2022-02-10T17:23:38.550-0600    WARN    [cfgwarn]       tlscommon/config.go:100 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0

my filebeat version is
filebeat version 7.16.3 (amd64), libbeat 7.16.3

output from curl localhost:9200

{
  "name" : "node-1",
  "cluster_name" : "wazuh-cluster",
  "cluster_uuid" : "baftwac9Qk-25fxCDwjngA",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "rpm",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

everything seems to be working fine, except that I cannot see any recent security events or fim events.  I'm still receiving emails about alerts. and active response is still working fine.

One of the things that Stand out to me is the WARN for the TLS security being disabled.. I know when it was on when I first deployed

Any assistance would be appreciated.

[Camila Salome Romero]

Hi! I hope you are well!

Seeing this log, I would say you have different versions for Filebeat and Elasticsearch.

The Elastic Stack should share the same version.

Could you tell me which version of Elasticsearch you have installed?

• To know about Elasticsearch:

curl -XGET https://localhost:9200 -u admin:admin -k

(where admin:admin should be your credentials) or as an alternative, you can try:

/usr/share/elasticsearch/bin/elasticsearch --version

•  For Filebeat: 

filebeat version

Regards, Camila!

[Draenoel Leonard]

Thank you for your time! You were correct.

Elasticstack:
Version: 7.10.2, Build: oss/rpm/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z, JVM: 15.0.1

Filebeat:
filebeat version 7.16.3 (amd64), libbeat 7.16.3 [d420ccdaf201e32a524632b5da729522e50257ae built 2022-01-07 00:36:57 +0000 UTC]

I held on to my .yml and reinstalled the proper version of filebeat.

current versions are now:

Elasticstack:
Version: 7.10.2, Build: oss/rpm/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z, JVM: 15.0.1

Filebeat:

filebeat version 7.10.2 (amd64), libbeat 7.10.2 [aacf9ecd9c494aa0908f61fbca82c906b16562a8 built 2021-01-12 23:11:24 +0000 UTC]

When I perform "filebeat test output" i receive this error message now.

filebeat test output
elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...

    security: server's certificate chain verification is enabled

    handshake... OK
    TLS version: TLSv1.3
    dial up... OK

  talk to server... ERROR Connection marked as failed because the onConnect callback failed: Filebeat requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of Filebeat.

I was looking at the unattended installation.sh file to download filebeat, I'm not sure what the issue is now?

Thanks again

[Draenoel Leonard]

Actually, I just found the way to install the oss version of filebeats. seems I no longer get an error message when testing output.

does it take some time to populate the dashboard?

Thank you.

[Draenoel Leonard]

I am now unable to see any alerts on the page.  not sure if this error gives any insight

Feb 11 10:45:29 Wazuh-Manager-SB kibana[1826]: {"type":"log","@timestamp":"2022-02-11T16:45:29Z","tags":["error","elasticsearch","data"],"pid":1826,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [14909], primary term [21]. current document has seqNo [14910] and primary term [21]"}

[Draenoel Leonard]

Update:

I am able to see new logs. however it is not showing old ones. is there a way to resolve this issue?

[Juan Carlos]

Hi,

I'm glad you were able to resolve the issue. I apologize for the slow reply but you may reinject the information for the logs that were not indexed by following this guide: https://wazuh.com/blog/recover-your-data-using-wazuh-alert-backups/

To summarize, Wazuh rotates and compresses the alerts logs daily, and with the help of a simple script you may gradually reinject these logs to a file which is in turn being read by Filebeat.

To avoid any conflict we use a dedicated file for this, for example /tmp/recovery.json and configure the Wazuh module's alert pipeline on filebeat to read it by adding that path to this file: /usr/share/filebeat/module/wazuh/alerts/manifest.yml

Then using the script attached we run the recovery:

nohup python recovery.py -eps 500 -min 2019-07-21T13:59:30 -max 2019-07-24T22:00:00 -o /tmp/recovery.json -log ./recovery.log -sz 2.5 &

Where you must specify the dates for which you are missing information on Elasticsearch.

Be careful to verify what the actual date of the events are and not the timezone shifted value provided by Kibana (which uses your browser's clock for convenience).

Let us know if you have any more questions.

Best Regards,

Juan C. Tello

[Draenoel Leonard]

Thank you for your reply.

I am able to get the recovery.py working and see that it's writing to the proper /tmp/ file.

however I am having an issue with the step following that.

it wants me to put:

filebeat.modules:

  - module: wazuh

    alerts:

      enabled: true

      input:

        paths:

          - /var/ossec/logs/alerts/alerts.json

          - /tmp/recovery.json

In the manifest.yml. am i to overwrite what's currently in there? or just add the above. it looks nothing like this entry.

What's in there currently in cd /usr/share/filebeat/module/wazuh/alerts/manifest.yml:

module_version: 0.1

var:
  - name: paths
    default:
      - /var/ossec/logs/alerts/alerts.json
    

   - name: index_prefix
    default: wazuh-alerts-4.x-

input: config/alerts.yml

ingest_pipeline: ingest/pipeline.json

Thank you!

[Juan Carlos]

Hi,

It's only necessary to add the -/tmp/recovery.json line after the alerts.json reference in the /usr/share/filebeat/module/wazuh/alerts/manifest.yml file.

The blog seems to have been poorly edited as it used to instruct to edit the /etc/filebeat/filebeat.yml file and although it was switched to the module manifest file it still shows the content of filebeat.yml.

Thanks for bringing this to our attention, I'll request internally for this to be fixed.

Best Regards,

Juan C. Tello

[Draenoel Leonard]

Thank you for the clarification.

Unfortunately it doesn't seem to be working. I have changed the /usr/share/filebeat/module/wazuh/alerts/manifest.yml to appear as follows:

module_version: 0.1

var:
  - name: paths
    default:
      - /var/ossec/logs/alerts/alerts.json

      - /tmp/recovery.json

  - name: index_prefix
    default: wazuh-alerts-4.x-

input: config/alerts.yml

ingest_pipeline: ingest/pipeline.json

files are being written to the recovery.json, but it still doesn't seem to be getting picked up through filebeat and placed back in history. is there a way I can check and verify? I'm currently looking back in wazuh and it's saying there's nothing in the timeline for the files that have been placed in the /tmp/recovery.json

I'm probably missing something simple. more assistance will be appreciated.

[Juan Carlos Tello]

Hello Draenoel,

This can happen if the filebeat service is not restarted after the configuration change (systemctl restart filebeat), you may verify that filebeat is reading the file by executing lsof /tmp/recovery.json where you will see which processes have the file open, filebeat must be one of them.

Please let us know if you have any more questions.

Best Regards,

Juan C. Tello

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/07e8d1f7-b920-4771-8115-18fca7248847n%40googlegroups.com.