Critical Error - Can't create PID file and how to pass Windows logs through the Wazuh manager

[Robert Heppe]

Hi Victor, Chema, or anyone else that's available.  

We are preparing for a soon to come deployment for about 6k agents.  I created a 2 manager cluster a couple months ago and left it in place running version 3.5.0 on CentOS 7.  Today I noticed all service except authd and another are not running.  After restarting and rebooting, it doesn't resolve.  I see in the ossec.log there is a critical error that a PID file cannot be created.  I have posted the actual error in the Wazuh group.  I have seen tonight that the logcollector daemon tries to start up and has a PID but then seconds later it dies and no other daemons can run.  I will post this information tomorrow morning.  

https://groups.google.com/forum/#!topic/wazuh/LH8OFiX_OEc

A second question I have that is important is we've heard it is possible to pass Windows logs out of the manager in a Raw format, not the alerts in the alert.json file, so we can ingest those in Elastic Stack for search.  Is this possible?  If so, is there a documentation for it?

Best regards,

Robert

This message contains confidential and/or private information and is intended only for the individual named. If you are not the named addressee, any delivery, disclosure, dissemination or distribution of this e-mail is unlawful and strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error, as information could be intercepted, corrupted, lost, destroyed, arrive late, be incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which errors or omissions arise as a result of e-mail transmission. If verification is required, please request a hard-copy version.

[Chema Martinez]

Hi Robert,

I hope you are doing well.

About the issue, I have checked logs reported in the other thread opened and we can extract some conclusions from them:

- Every daemon fall due to the analysis daemon stops at the start due to the critical error: '2018/10/16 16:43:04 ossec-analysisd: CRITICAL: (1212): Unable to create PID file.' When the analysis daemon is not running, the rest of the daemons cannot connect to it and are stopped automatically.

- There can be several causes of not being able to create it. The path /var/run inside the installation directory doesn't exist, it fails setting permissions to the PID file, or another error during the writing process. Last two cases should generate an extra error log before the critical one.

 

- I have realized that the path of the queue which is trying to reach the rest of daemons includes the hardcoded text '<path>' like, for example, this one:

2018/10/16 16:43:13 ossec-logcollector: ERROR: (1210): Queue '/opt/<path>/ossec/queue/ossec/queue' not accessible: 'Connection refused'.

What is the installation folder of your manager? You can check it at the file "/etc/ossec-init.conf". It is possible that an incorrect path is being read by the manager and it is not able to find the sockets and files correctly.

Related to the second question, it is not our scope to send agent events directly to ElasticSearch due to Wazuh's own architecture. We consider the processing and enrichment of events the most powerful task of the manager, being able to decode fields, match rules, and classify events based on the results obtained.

Let me ask you which is the reason to prefer skip the manager processing for Windows logs. If you are missing any kind of rules or decoders to ingest particular logs, we could surely improve the ruleset for that use case.

Best regards,

Chema.

 

Chema Martinez | IT Engineer — Wazuh, Inc.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/MW2PR18MB23477070DBD71552C1511D58A8FF0%40MW2PR18MB2347.namprd18.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

[Robert H]

Hi Chema,

Thanks for looking at this with me.  I installed the cluster nodes, behind a load balancer, and tested with a handful of agents a couple dozen times.  So everything was working.  3 months later with no activity, this is the current state.  The <path> is just to obscure the directory name.  We installed the wazuh manager using sources in /opt/<path>/ossec

This is the current state of (both) managers in the cluster.

wazuh-clusterd not running...

ossec-monitord not running...

ossec-logcollector not running...

ossec-remoted not running...

ossec-syscheckd not running...

ossec-analysisd not running...

ossec-maild not running...

ossec-execd not running...

wazuh-modulesd is running...

wazuh-db not running...

ossec-csyslogd not running...

ossec-authd is running...

//////////////////////////////////////////////////////////////

I see 2 pid's

 ls -la /opt/<path>/ossec/var/run/

total 8

drwxr-xr-x. 2 root ossec 67 Oct 16 21:10 .

drwxr-xr-x. 8 root ossec 87 Oct 17 10:02 ..

-rw-r-----. 1 root ossec  6 Oct 16 21:08 ossec-authd-19080.pid

-rw-r-----. 1 root ossec  6 Oct 16 21:08 wazuh-modulesd-19096.pid

/////////////////////////////////////

When I restart the manager I see 4 pids, then 3, then the 2 stable ones

sudo ls -la /opt/<path>/ossec/var/run/

total 16

drwxr-xr-x. 2 root ossec 132 Oct 17 10:49 .

drwxr-xr-x. 9 root ossec 112 Oct 17 10:49 ..

-rw-r-----. 1 root ossec   5 Oct 17 10:49 ossec-authd-4311.pid

-rw-r-----. 1 root root    5 Oct 17 10:49 ossec-logcollector-4352.pid

-rw-r-----. 1 root root    5 Oct 17 10:49 ossec-syscheckd-4345.pid

-rw-r-----. 1 root ossec   5 Oct 17 10:49 wazuh-modulesd-4326.pid

sudo ls -la /opt/<path>/ossec/var/run/

total 12

drwxr-xr-x. 2 root ossec 100 Oct 17 10:49 .

drwxr-xr-x. 8 root ossec  87 Oct 17 10:49 ..

-rw-r-----. 1 root ossec   5 Oct 17 10:49 ossec-authd-4311.pid

-rw-r-----. 1 root root    5 Oct 17 10:49 ossec-logcollector-4352.pid

-rw-r-----. 1 root ossec   5 Oct 17 10:49 wazuh-modulesd-4326.pid

sudo ls -la /opt/<path>/ossec/var/run/

total 8

drwxr-xr-x. 2 root ossec 65 Oct 17 10:50 .

drwxr-xr-x. 8 root ossec 87 Oct 17 10:49 ..

-rw-r-----. 1 root ossec  5 Oct 17 10:49 ossec-authd-4311.pid

-rw-r-----. 1 root ossec  5 Oct 17 10:49 wazuh-modulesd-4326.pid

///////////////////////////

Ossec log shows, can't create PID file

2018/10/17 10:53:40 ossec-analysisd: INFO: Ignoring file: '/etc/utmpx'

2018/10/17 10:53:40 ossec-analysisd: INFO: Ignoring file: '/etc/wtmpx'

2018/10/17 10:53:40 ossec-analysisd: INFO: Ignoring file: '/etc/cups/certs'

2018/10/17 10:53:40 ossec-analysisd: INFO: Ignoring file: '/etc/dumpdates'

2018/10/17 10:53:40 ossec-analysisd: INFO: Ignoring file: '/etc/svc/volatile'

2018/10/17 10:53:40 ossec-analysisd: CRITICAL: (1212): Unable to create PID file.

2018/10/17 10:53:43 ossec-syscheckd: ERROR: (1210): Queue '/opt/<path>/ossec/queue/ossec/queue' not accessible: 'Connection refused'.

2018/10/17 10:53:43 rootcheck: CRITICAL: (1211): Unable to access queue: '/opt/<path>/ossec/queue/ossec/queue'. Giving up..

2018/10/17 10:53:49 ossec-logcollector: ERROR: (1210): Queue '/opt/<path>/ossec/queue/ossec/queue' not accessible: 'Connection refused'.

2018/10/17 10:53:49 ossec-logcollector: CRITICAL: (1211): Unable to access queue: '/opt/<path>/ossec/queue/ossec/queue'. Giving up..

ossec-init.conf

DIRECTORY="/opt/<path>/ossec"

NAME="Wazuh"

VERSION="v3.5.0"

REVISION="3510"

DATE="Fri Aug 17 10:27:19 PDT 2018"

TYPE="server"

[Chema Martinez]

Hi Robert,

Perfect, I just was wondering whether the path was obscured or an installation mistake.

I have noticed that the permissions of the folder /opt/<path>/ossec/var/run are not correct. It has no writing permissions for the group ossec so the analysis daemon is unable to create the PID file (this daemon runs as the user ossec as well). Other daemons such as authd and wazuh-modulesd run as root so their PID files are created without any problem. 

Please, change the permissions to that folder as follows: 

chmod 770 /opt/<path>/ossec/var/run

And try to start the manager again.

I think this could be caused by installing the manager without being root as you commented in the other thread. Actually, the installation script should set the correct permissions for that folder as you can see here:

https://github.com/wazuh/wazuh/blob/3.5/src/init/inst-functions.sh#L770

Please, let me know if works for you.

Best regards,

Chema.

Chema Martinez | IT Engineer — Wazuh, Inc.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/67b629f3-8a1b-49da-8e7e-c60cb08dfe8b%40googlegroups.com.

[Robert H]

Thanks Chema,

I will try this when I reach the office this morning.  

As for the second question, I checked with my management and for compliance and customer agreement, they do want to store all windows logs in Elastic.  Could you describe how we could pass the windows logs through the manager to Elastic and then if it would be needed to use a separate index or if both logs, the wazuh alerts and the raw windows logs could be ingested using the wazuh-alerts- index.  The do want to use both the Wazuh app in Kibana and have all windows logs available in Elastic for search.  I understand this is not a common use case. 

Best regards,

Robert

[Robert H]

Thanks Chema,

That did work and the managers started up again normally.  Please see my previous post about the raw logs.

Regards,

Robert