Memory locking changes

[Nuno Campos]

Hi all, 

Probably a basic question, but I am new in wazuh and elasticsearch.

When I use,

curl "http://localhost:9200/_nodes?filter_path=**.mlockall&pretty"

to check the memory locking changes, it gives bellow output:

curl: (52) Empty reply from server

Then I try to read log file, but I cant find the file: elasticsearch.log

tail -f /var/log/elasticsearch/elasticsearch.log

notes:

- Wazuh installation With all-in-one deployment;

- Unattended installation - 4.2

Anyone can help ? 

Thanks!

[Alfonso Ruiz-Bravo]

Hello ncampos!!

The request should work without problem, as well as you should have elasticsearch logs in the path you indicate:  /var/log/elasticsearch/...

Could you check if the elasticsearch service is running?

service elasticsearch status

Gives the impression that elasticsearch is down or stopped.

I look forward to hearing from you, have a nice day.

Best regards,

Alfonso Ruiz-Bravo

[Nuno Campos]

Hi Alfonso,

The files that appears in the path  /var/log/elasticsearch are:

-rw-r--r--  1 elasticsearch elasticsearch 457607 Nov  2 12:09 gc.log

-rw-r--r--  1 elasticsearch elasticsearch   2045 Oct 28 09:57 gc.log.00

-rw-r--r--  1 elasticsearch elasticsearch 223672 Oct 28 15:44 gc.log.01

-rw-r--r--  1 elasticsearch elasticsearch   2045 Oct 28 15:52 gc.log.02

-rw-r--r--  1 elasticsearch elasticsearch 264073 Oct 29 13:28 gc.log.03

-rw-r--r--  1 elasticsearch elasticsearch   2045 Oct 29 13:29 gc.log.04

-rw-r--r--  1 elasticsearch elasticsearch 123861 Oct 29 13:51 gc.log.05

-rw-r--r--  1 elasticsearch elasticsearch   2045 Oct 29 13:53 gc.log.06

-rw-r--r--  1 elasticsearch elasticsearch  91974 Oct 29 14:05 gc.log.07

-rw-r--r--  1 elasticsearch elasticsearch   2045 Oct 29 14:05 gc.log.08

-rw-r--r--  1 elasticsearch elasticsearch  16743 Oct 29 00:00 wazuh-cluster-2021-10-28-1.json.gz

-rw-r--r--  1 elasticsearch elasticsearch  15444 Oct 29 00:00 wazuh-cluster-2021-10-28-1.log.gz

-rw-r--r--  1 elasticsearch elasticsearch  24829 Oct 30 00:00 wazuh-cluster-2021-10-29-1.json.gz

-rw-r--r--  1 elasticsearch elasticsearch  21910 Oct 30 00:00 wazuh-cluster-2021-10-29-1.log.gz

-rw-r--r--  1 elasticsearch elasticsearch   4152 Oct 31 00:00 wazuh-cluster-2021-10-30-1.json.gz

-rw-r--r--  1 elasticsearch elasticsearch   3584 Oct 31 00:00 wazuh-cluster-2021-10-30-1.log.gz

-rw-r--r--  1 elasticsearch elasticsearch   4089 Nov  1 00:00 wazuh-cluster-2021-10-31-1.json.gz

-rw-r--r--  1 elasticsearch elasticsearch   3486 Nov  1 00:00 wazuh-cluster-2021-10-31-1.log.gz

-rw-r--r--  1 elasticsearch elasticsearch   4269 Nov  2 00:00 wazuh-cluster-2021-11-01-1.json.gz

-rw-r--r--  1 elasticsearch elasticsearch   3580 Nov  2 00:00 wazuh-cluster-2021-11-01-1.log.gz

-rw-r--r--  1 elasticsearch elasticsearch  26590 Nov  2 12:10 wazuh-cluster.log

-rw-r--r--  1 elasticsearch elasticsearch  17945 Nov  2 11:49 wazuh-cluster_deprecation.json

-rw-r--r--  1 elasticsearch elasticsearch 680844 Nov  2 12:12 wazuh-cluster_deprecation.log

-rw-r--r--  1 elasticsearch elasticsearch      0 Oct 28 09:57 wazuh-cluster_index_indexing_slowlog.json

-rw-r--r--  1 elasticsearch elasticsearch      0 Oct 28 09:57 wazuh-cluster_index_indexing_slowlog.log

-rw-r--r--  1 elasticsearch elasticsearch      0 Oct 28 09:57 wazuh-cluster_index_search_slowlog.json

-rw-r--r--  1 elasticsearch elasticsearch      0 Oct 28 09:57 wazuh-cluster_index_search_slowlog.log

-rw-r--r--  1 elasticsearch elasticsearch  71819 Nov  2 12:10 wazuh-cluster_server.json

I cant find the file: elasticsearch.log

The output of service elasticsearch status:

● elasticsearch.service - Elasticsearch

     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)

    Drop-In: /etc/systemd/system/elasticsearch.service.d

             └─elasticsearch.conf

     Active: active (running) since Fri 2021-10-29 14:05:29 UTC; 3 days ago

       Docs: https://www.elastic.co

   Main PID: 96855 (java)

      Tasks: 86 (limit: 19026)

     Memory: 9.0G

     CGroup: /system.slice/elasticsearch.service

             └─96855 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.>

Thanks a lot for your time!

Nuno Campos

[Alfonso Ruiz-Bravo]

Hi Nuno,

Well, you could try changing the log path and restarting the service.

elasticsearch.yml setting -> path.logs: /var/log/elasticsearch

https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html#_config_file_format

On the other hand, have you tried to run more curls to the Elasticsearch API? for example: 

curl -k -u admin "https://localhost:9200/_cluster/health?pretty"

Let's see with this command the status of the service once the changes have been applied.

 I look forward to hearing from you.

Regards,

Alfonso Ruiz-Bravo Cloud computing engineer The Open Source Security Platform

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/1zaIfItIBU4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/b0314668-9e1c-457e-9139-cca9605d3581n%40googlegroups.com.

[Nuno Campos]

Hi Alfonso,

1st- I made the changes at log path:

elasticsearch.yml setting -> path.logs: /var/log/elasticsearch/log

2- Create de the var directory -> /var/log/elasticsearch/log

chown elasticsearch:elasticsearch log/

3-  Restarting the service ->  systemctl restart elasticsearch and check the var directory

/var/log/elasticsearch/log

-rw-r--r-- 1 elasticsearch elasticsearch 37304 Nov  2 14:40 wazuh-cluster.log

-rw-r--r-- 1 elasticsearch elasticsearch  1582 Nov  2 14:35 wazuh-cluster_deprecation.json

-rw-r--r-- 1 elasticsearch elasticsearch  5664 Nov  2 14:44 wazuh-cluster_deprecation.log

-rw-r--r-- 1 elasticsearch elasticsearch     0 Nov  2 14:30 wazuh-cluster_index_indexing_slowlog.json

-rw-r--r-- 1 elasticsearch elasticsearch     0 Nov  2 14:30 wazuh-cluster_index_indexing_slowlog.log

-rw-r--r-- 1 elasticsearch elasticsearch     0 Nov  2 14:30 wazuh-cluster_index_search_slowlog.json

-rw-r--r-- 1 elasticsearch elasticsearch     0 Nov  2 14:30 wazuh-cluster_index_search_slowlog.log

-rw-r--r-- 1 elasticsearch elasticsearch 54239 Nov  2 14:40 wazuh-cluster_server.json

tail -f wazuh-cluster.log

[2021-11-02T14:34:03,390][INFO ][o.e.c.m.MetadataMappingService] [node-1] [security-auditlog-2021.11.02/YhDJh-w0Rli0AndeqLDw5w] update_mapping [_doc]

[2021-11-02T14:35:13,905][INFO ][c.a.o.j.s.JobSweeper     ] [node-1] Running full sweep

[2021-11-02T14:40:13,907][INFO ][c.a.o.j.s.JobSweeper     ] [node-1] Running full sweep

[2021-11-02T14:40:33,768][INFO ][c.a.o.s.p.PrivilegesEvaluator] [node-1] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[*], allIndices=[*], types=[*], originalRequested=[*], remoteIndices=[]] [Action [indices:admin/mappings/get]] [RolesChecked [wazuh_ui_admin, own_index, kibana_server]]

[2021-11-02T14:40:33,797][INFO ][c.a.o.s.p.PrivilegesEvaluator] [node-1] No permissions for [indices:admin/mappings/get]

[2021-11-02T14:40:33,908][INFO ][c.a.o.s.p.PrivilegesEvaluator] [node-1] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[], allIndices=[security-auditlog-2021.10.31, wazuh-alerts-4.x-2021.10.30, wazuh-alerts-4.x-2021.10.31, security-auditlog-2021.10.30, wazuh-alerts-4.x-2021.10.29, security-auditlog-2021.10.29, wazuh-statistics-2021.44w, wazuh-monitoring-2021.45w, wazuh-monitoring-2021.44w, wazuh-statistics-2021.43w, security-auditlog-2021.10.28, security-auditlog-2021.11.02, wazuh-alerts-4.x-2021.10.28, wazuh-statistics-2021.45w, wazuh-alerts-4.x-2021.11.02, wazuh-alerts-4.x-2021.11.01, security-auditlog-2021.11.01], types=[*], originalRequested=[.ent-search-*, .app-search-*, *magento2*, *magento*, *shopify*, *wordpress*, *drupal*, *joomla*, *search*, *sharepoint*, *squarespace*, *sitecore*, *weebly*, *acquia*, filebeat-*, metricbeat-*, apm-*, functionbeat-*, heartbeat-*, logstash-*, fluentd*, telegraf*, prometheusbeat*, fluentbit*, *nginx*, *apache*, endgame-*, logs-endpoint.*, metrics-endpoint.*, .siem-signals-*, auditbeat-*, winlogbeat-*, packetbeat-*, *tomcat*, *artifactory*, *aruba*, *barracuda*, *bluecoat*, arcsight-*, *checkpoint*, *cisco*, *citrix*, *cyberark*, *cylance*, *fireeye*, *fortinet*, *infoblox*, *kaspersky*, *mcafee*, *paloaltonetworks*, pan-*, pan_*, pan.*, rsa.*, rsa-*, rsa_*, snort-*, logstash-snort*, *sonicwall*, *sophos*, squid-*, squid_*, squid.*, *symantec*, *tippingpoint*, *trendmicro*, *tripwire*, *zscaler*, *zeek*, *sigma_doc*, ecs-corelight*, *suricata*, *wazuh*, *meow*, *-*-*], remoteIndices=[]] [Action [indices:monitor/stats]] [RolesChecked [wazuh_ui_admin, own_index, kibana_server]]

[2021-11-02T14:40:33,909][INFO ][c.a.o.s.p.PrivilegesEvaluator] [node-1] No permissions for [indices:monitor/stats]

[2021-11-02T14:45:00,743][INFO ][o.e.c.m.MetadataUpdateSettingsService] [node-1] updating number_of_replicas to [0] for indices [wazuh-monitoring-2021.45w]

[2021-11-02T14:45:13,907][INFO ][c.a.o.j.s.JobSweeper     ] [node-1] Running full sweep

The file wazuh-cluster.log is the same of elasticsearch.log?

It will be better to rollback changing the log path? -> elasticsearch.yml setting -> path.logs: /var/log/elasticsearch

4- The output of curl -k -u admin "https://localhost:9200/_cluster/health?pretty"

{

  "cluster_name" : "wazuh-cluster",

  "status" : "yellow",

  "timed_out" : false,

  "number_of_nodes" : 1,

  "number_of_data_nodes" : 1,

  "active_primary_shards" : 34,

  "active_shards" : 34,

  "relocating_shards" : 0,

  "initializing_shards" : 0,

  "unassigned_shards" : 6,

  "delayed_unassigned_shards" : 0,

  "number_of_pending_tasks" : 0,

  "number_of_in_flight_fetch" : 0,

  "task_max_waiting_in_queue_millis" : 0,

  "active_shards_percent_as_number" : 85.0

}

Regards,

Nuno Campos

[Nuno Campos]

Correction:

2- Create de the log directory -> /var/log/elasticsearch/log

chown elasticsearch:elasticsearch log/

3-  Restarting the service ->  systemctl restart elasticsearch and check the log directory

Sorry Alfonso ;)

[Alfonso Ruiz-Bravo]

Hi Nuno,

The logs may be in the wazuh-cluster.log file.

It looks like you have Elasticsearch running, could you retest the memory request? could you provide us with the whole output? can you launch it with the -v flag to get more information.

Regards,

Alfonso Ruiz-Bravo Cloud computing engineer The Open Source Security Platform

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f66b9d10-b68d-4fcc-acd5-89cc6329b96dn%40googlegroups.com.

[Nuno Campos]

Hello Alfonso,

Sure, please check the snippets below:

*   Trying 127.0.0.1:9200...

* TCP_NODELAY set

* Connected to localhost (127.0.0.1) port 9200 (#0)

> GET /_nodes?filter_path=**.mlockall&pretty HTTP/1.1

> Host: localhost:9200

> User-Agent: curl/7.68.0

> Accept: */*

> 

* Empty reply from server

* Connection #0 to host localhost left intact

curl: (52) Empty reply from server

Thanks.
Nuno Campos

[Alfonso Ruiz-Bravo]

Hi Nuno,

could you try with HTTPS instead HTTP?

curl "https://localhost:9200/_nodes?filter_path=**.mlockall&pretty"

Regards,

Alfonso Ruiz-Bravo Cloud computing engineer The Open Source Security Platform

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9fc8f1f3-25c3-4317-8451-791668517dddn%40googlegroups.com.

[Nuno Campos]

Hello Alfonso,

Please check the snippets below:

 Trying 127.0.0.1:9200...

* TCP_NODELAY set

* Connected to localhost (127.0.0.1) port 9200 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

*   CAfile: /etc/ssl/certs/ca-certificates.crt

  CApath: /etc/ssl/certs

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* TLSv1.3 (IN), TLS handshake, Server hello (2):

* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

* TLSv1.3 (IN), TLS handshake, Request CERT (13):

* TLSv1.3 (IN), TLS handshake, Certificate (11):

* TLSv1.3 (OUT), TLS alert, unknown CA (560):

* SSL certificate problem: unable to get local issuer certificate

* Closing connection 0

curl: (60) SSL certificate problem: unable to get local issuer certificate

More details here: https://curl.haxx.se/docs/sslcerts.html

Regards.

NC

[Alfonso Ruiz-Bravo]

Hi Nuno,

could you try this?

curl -k -u admin "https://localhost:9200/_nodes?filter_path=**.mlockall&pretty"

Regards,

Alfonso Ruiz-Bravo Cloud computing engineer The Open Source Security Platform

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/d9dec0b5-b734-481d-a3d0-839ebb1ab95an%40googlegroups.com.

[Nuno Campos]

Hi Alfonso,

the output:

{

  "nodes" : {

    "19Q9RQA-RraW9ub4t_OZBQ" : {

      "process" : {

        "mlockall" : true

      }

    }

  }

}

Regards.

NC

[Alfonso Ruiz-Bravo]

Great!

I'm glad you can now make the request Nuno.

Best regards,

Alfonso Ruiz-Bravo Cloud computing engineer The Open Source Security Platform

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/51f7d572-57b5-417e-8ca2-c062f3a707f2n%40googlegroups.com.

[Nuno Campos]

Hi Alfonso,

Now I can verify that the setting was successfully changed by checking the value of mlockall.

Maybe it's better rollback with the changing log path(?):

-> elasticsearch.yml -> path.logs: /var/log/elasticsearch/log 

to 

path.logs: /var/log/elasticsearch

Thanks a lot for your time!

NC

[Alfonso Ruiz-Bravo]

Hi Nuno,

Yes, you can rollback if you want.

Regards,

Alfonso Ruiz-Bravo Cloud computing engineer The Open Source Security Platform

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/bd91b3ff-1c92-4eeb-8c82-b496382122cfn%40googlegroups.com.