/tmp out of space, elasticsearch during install

[Todd Riffel]

Hello, I notice the 4.2 Unattended all in one deployment fails at the elasticsearch point of install on AlmaLinux release 8.4.  I then tried step by step which elasticsearch failed to start as well and was filling up /tmp with sqlite-3.32.3.2.*. 

The only workaround i found was to create a new tmpdir with correct permissions and change to ES_TMPDIR=/etc/elasticsearch/tmp by updating:

/etc/sysconfig/elasticsearch

# Additional Java OPTS

ES_TMPDIR=/etc/elasticsearch/tmp

TMPDIR=/etc/elasticsearch/tmp

/etc/systemd/system/elasticsearch.service.d/elasticsearch.conf

[Service]

LimitMEMLOCK=infinity

/etc/systemd/system/elasticsearch.service.wants/opendistro-performance-analyzer.service 

#!/bin/sh

PA_AGENT_JAVA_OPTS="-Dlog4j.configurationFile=$ES_HOME/plugins/opendistro_performance_analyzer/pa_config/log4j2.xml \

              -Xms64M -Xmx64M -XX:+UseSerialGC -XX:CICompilerCount=1 -XX:-TieredCompilation -XX:InitialCodeCacheSize=4096 \

              -XX:InitialBootClassLoaderMetaspaceSize=30720 -XX:MaxRAM=400m -Djna.tmpdir=/etc/elasticsearch/tmp -Djava.io.tmpdir=/etc/elasticsearch/tmp"

Updating jvm.options only did not change the tmpdir

/etc/elasticsearch/jvm.options

## JVM temporary directory

-Djava.io.tmpdir=${ES_TMPDIR}

Is this a known issue others are having during installs?  And if so will there be a fix for this so I can use the all in one deployment as I'm doing a POC for a possible cluster install in 2 datacenters?  thank you! 

[Jesus Linares]

Hi,

That is interesting. I understand your workaround in Elasticsearch (you should not use it production since that it is not the goal of the /etc directory), but the root cause of your issue is the /tmp directory filled by sqlite.

According to the SQLite documentation for temporary files:

The directory or folder in which temporary files are created is determined by the OS-specific VFS.

On unix-like systems, directories are searched in the following order:

1. The directory set by PRAGMA temp_store_directory or by the sqlite3_temp_directory global variable
2. The SQLITE_TMPDIR environment variable
3. The TMPDIR environment variable
4. /var/tmp
5. /usr/tmp
6. /tmp
7. The current working directory (".")

So, as the first hypothesis, the directory may vary from one operating system to another and it could be impacting AlmaLinux.

I would discuss it with the team, but I would like to know what kind of files regarding sqlite you found in /tmp and if there were more files related to Wazuh there.

Thank you.

[Todd Riffel]

Hi Jesus,

Thanks for the info on recommended /tmp locations.  I blew away all the /tmp/sqlite-3.32.3.2.* files after I moved to /etc/elasticsearch/tmp to free up 100% filled /tmp.  Here is what is left over now in /tmp:

[root@wazuh ~]# ls -lah /tmp

total 323M

drwxrwxrwt. 15 root          root          4.0K Oct  5 10:01 .

dr-xr-xr-x. 17 root          root           273 Oct  1 17:55 ..

-rw-r--r--.  1 root          root             0 Oct  1 17:35 boot_parameters

drwxrwxrwt.  2 root          root             6 Oct  1 17:36 .font-unix

drwxr-xr-x   2 elasticsearch elasticsearch   18 Oct  4 16:22 hsperfdata_elasticsearch

drwxr-xr-x   2 root          root             6 Oct  1 18:02 hsperfdata_root

drwxrwxrwt.  2 root          root             6 Oct  1 17:36 .ICE-unix

-rwx------.  1 root          root          5.3K Oct  1 17:35 ks-script-9__hrt1d

-rwx------.  1 root          root           701 Oct  1 17:36 ks-script-raozordf

-rwx------.  1 root          root           291 Oct  1 17:36 ks-script-z89yk8ak

-rw-r--r--   1 elasticsearch elasticsearch 260K Oct  5 10:01 metricsdb_1633453260000

-rw-r--r--   1 elasticsearch elasticsearch 260K Oct  5 10:01 metricsdb_1633453265000

-rw-r--r--   1 elasticsearch elasticsearch  55M Oct  5 10:01 performance_analyzer_agent_stats.log

-rw-r--r--   1 elasticsearch elasticsearch  82M Oct  5 10:01 PerformanceAnalyzer.log

-rw-r--r--   1 elasticsearch elasticsearch  28K Oct  5 10:00 rca.sqlite

-rw-r--r--   1 elasticsearch elasticsearch  40K Oct  5 05:23 rca.sqlite.2021-10-05-05-24-55

-rw-r--r--   1 elasticsearch elasticsearch  40K Oct  5 06:23 rca.sqlite.2021-10-05-06-24-56

-rw-r--r--   1 elasticsearch elasticsearch  40K Oct  5 07:23 rca.sqlite.2021-10-05-07-24-56

-rw-r--r--   1 elasticsearch elasticsearch  40K Oct  5 08:23 rca.sqlite.2021-10-05-08-24-56

-rw-r--r--   1 elasticsearch elasticsearch  40K Oct  5 09:23 rca.sqlite.2021-10-05-09-24-56

-rw-r--r--   1 elasticsearch elasticsearch 906K Oct  1 18:03 sqlite-3.32.3.2-09e24b14-b014-4edd-8343-5d349cdcab86-libsqlitejdbc.so

-rw-r--r--   1 elasticsearch elasticsearch    0 Oct  1 18:03 sqlite-3.32.3.2-09e24b14-b014-4edd-8343-5d349cdcab86-libsqlitejdbc.so.lck

-rw-r--r--   1 elasticsearch elasticsearch 906K Oct  1 18:03 sqlite-3.32.3.2-3b86e860-5504-482b-b767-2436cd6c9f87-libsqlitejdbc.so

-rw-r--r--   1 elasticsearch elasticsearch    0 Oct  1 18:03 sqlite-3.32.3.2-3b86e860-5504-482b-b767-2436cd6c9f87-libsqlitejdbc.so.lck

-rw-r--r--   1 elasticsearch elasticsearch 906K Oct  1 18:03 sqlite-3.32.3.2-5daba6fd-6ed2-4272-8571-8cc2f4a48ff7-libsqlitejdbc.so

-rw-r--r--   1 elasticsearch elasticsearch    0 Oct  1 18:03 sqlite-3.32.3.2-5daba6fd-6ed2-4272-8571-8cc2f4a48ff7-libsqlitejdbc.so.lck

-rw-r--r--   1 elasticsearch elasticsearch  69K Oct  1 18:03 sqlite-3.32.3.2-d1f3d16b-3d82-4282-a5f8-a4d2dcf3b844-libsqlitejdbc.so

-rw-r--r--   1 elasticsearch elasticsearch    0 Oct  1 18:03 sqlite-3.32.3.2-d1f3d16b-3d82-4282-a5f8-a4d2dcf3b844-libsqlitejdbc.so.lck

Thank you! 

[Todd Riffel]

also to add, this is in the dir with the system running, monitoring a few hosts:

du -hsc /etc/elasticsearch/tmp/*

908K /etc/elasticsearch/tmp/sqlite-3.32.3.2-6e61a8ac-9358-4714-a5af-947227b48907-libsqlitejdbc.so

0 /etc/elasticsearch/tmp/sqlite-3.32.3.2-6e61a8ac-9358-4714-a5af-947227b48907-libsqlitejdbc.so.lck

908K total

[Jesus Linares]

Hi,

I reviewed this with the team, and it is not related to Wazuh since it is Elasticsearch the one that leaves the SQLite files in the temporary directory. Also, it makes sense after checking your outputs (sqlite files in /etc/elasticsearch/tmp).

That said and checking some similar issues (https://github.com/opendistro-for-elasticsearch/performance-analyzer/issues/70), it looks like when the performance-analyzer fails, it tries to restart itself in an infinite loop and that causes the generation of several temp files. I don't know if this is your case, but I would check the Elasticsearch logs in order to see if there are some hints about what is doing at that moment. In this way, you can try to fix the root cause. For example, you can try to remove the sqlite plugin in opendistro or the performance-analyzer.

I hope it helps.

[Todd Riffel]

Thanks Jesus, will investigate further on my next install.