Vulnerability detector not working

[unknown man]

Hello,

I have enabled Vulnerability detector, by following the steps provided in Wazuh documentation.

The Wazuh solution is implemented in Kubernetes.

Wazuh Manager version - v3.12.2

Wazuh client version - v3.11.3

I am getting the error as below.

2020/05/06 11:45:44 wazuh-modulesd:vulnerability-detector[2891] wm_vuln_detector.c:5175 at wm_vuldet_run_scan(): INFO: (5452): Starting vulnerability scanning.
2020/05/06 11:45:44 wazuh-modulesd:vulnerability-detector[2891] wm_vuln_detector.c:1060 at wm_vuldet_check_agent_vulnerabilities(): DEBUG: (5486): Starting vulnerability assessment for agent 24.
2020/05/06 11:45:44 wazuh-modulesd:vulnerability-detector[2891] wm_vuln_detector.c:2846 at wm_vuldet_get_software_info(): DEBUG: (5462): Getting agent 24 software.
2020/05/06 11:45:44 wazuh-modulesd:vulnerability-detector[2891] wm_vuln_detector.c:2864 at wm_vuldet_get_software_info(): DEBUG: (5574): A partial scan will be run on agent 24.
2020/05/06 11:45:44 wazuh-modulesd:vulnerability-detector[2891] wm_vuln_detector.c:2873 at wm_vuldet_get_software_info(): DEBUG: (5434): No package inventory found for agent 24, so their vulnerabilities will not be checked.

Here is the configuration from the manager.

      <!-- System inventory -->                                                                                                  
      <wodle name="syscollector">                                                                                                
        <disabled>no</disabled>                                                                                                  
        <interval>1h</interval>                                                                                                  
        <scan_on_start>yes</scan_on_start>                                                                                      
        <hardware>yes</hardware>                                                                                                
        <os>yes</os>                                                                                                            
        <network>yes</network>                                                                                                  
        <packages>yes</packages>                                                                                                
        <ports all="no">yes</ports>                                                                                              
        <processes>yes</processes>                                                                                              
      </wodle>                                                                                                                  
                                                                                                                                 
      <vulnerability-detector>                                                                                                  
        <enabled>yes</enabled>                                                                                                  
        <interval>10m</interval>                                                                                                
        <run_on_start>yes</run_on_start>                                                                                        
        <ignore_time>1h</ignore_time>                                                                                            
        <provider name="redhat">                                                                                                
          <enabled>yes</enabled>                                                                                                
          <path>/local_path/rh-feed/redhat-feed.*json$</path>                                                                    
          <update_interval>10m</update_interval>                                                                                
        </provider>                                                                                                              
      </vulnerability-detector>                                                                                           

The agent OS version is CentOS 7.7.

Regards,

[Fernando G.]

Hi,

At first glance it seems the issue lies in SysCollector rather than in vuln-detector. Let's see if SysCollector is doing its job properly... Please run the following command(manager):

sqlite3 /var/ossec/queue/db/024.db "select * from sys_programs;"

Does it return any lines? In theory, sys_programs should have a few hundred entries. As its name implies, it contains all the installed packages, fetched by SysCollector, from the agent 24. 

kind regards,

Fernando

[unknown man]

Hello Fernando,

Thanks for the reply.

I tried to execute the command in Wazuh manager pod. But it seems sqlite3 is not recognized in container images.

Regards,

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f65e1a29-5662-4664-b04d-4ae81e8675b6%40googlegroups.com.

[unknown man]

Hello, 

My issue is still on going. Please help me to fix the issue

Regards

[Fernando G.]

Hi,

Apologies for the late response. If you can't run sqlite3 within the container, then just copy the database file into your host.

(Check beforehand that the file exists)

kubectl cp <file-spec-src> <file-spec-dest>

Once you have said file, run the sqlite command (make sure it's alredy installed) I pasted in the previous email message. 

kind regards,

Fernando

[Fernando G.]

Hi Aravind,

Yeah sure, run the following command in both files:

$ md5sum <filename>

 

This command will generate a hash value based on the file content. If both hashes match, 

it will mean the files are equal and no data was lost at the time of copying. 

Kind regards,

Fernando

Hello Fernando,
I have managed to copy the file from pod to local system, which I have installed sqlite3.
The command you mentioned is executed, but no output is shown. The file size is around 220KB.
I feel like the files is not copying properly.
Is there another way that we can verify the DB.
Regards,
Aravind

[Aravind Krish]

Hello Fernando,

Thanks, I totally forgot about hash values.

The previous file that I downloaded was having hash values different with original.

I again downloaded and the hash values matched now.

The command didnt give me any output. The result is blank.

Regards,

Aravind

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/0c17e5ec-32be-486c-883a-522f5567d063%40googlegroups.com.

[Fernando G.]

Hi Aravind,

Great, at least we know it's not vuln-detector's fault. It seems SysCollector is not sending the information from the agent's side. 

Just for precaution run the following commands:

(manager side):

$ /var/ossec/bin/agent_control -l 

Agent 24 should appear as online. 

(agent side):

$ /var/ossec/bin/ossec-control status

Paste the output of this one, please.

kind regards, 

Fernando

[Aravind Krish]

Hello Fernando,

I checked the Wazuh manager and I can see that agent is active.

On the client side, I dont have access.I am waiting for another team to check and let me know.

But I have tired to restart the wazuh agent from manager, and it was successful.

Let me if any thing I need to do in manager.

Regards,

Aravind

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/ae78c510-f0ca-48f8-a830-ac62a1f3d825%40googlegroups.com.

[Fernando G.]

Hi Aravind,

Just so we are in the same page, I presume you have more agents using SysCollector 

without any issues, right? 

It would be useful to see the agent's log file, sometimes the restart may fail due 

to unexpected circumstances.

kind regards,

Fernando

[Fernando G.]

Hi Aravind,

I know it's quite troublesome, but could it be possible to run the following commands on the agent?

$ kill $(pidof /var/ossec/bin/wazuh-modulesd)

$ /var/ossec/bin/wazuh-modulesd -fdd 2> log #for a minute or so.

$ cat log | grep -i syscollector

After this, paste the command's output so we can be certain that the agent is sending all the required information.

kind regards,

Fernando

Hello Fernando, 

I have checked the ossec-control status on agent and it shows 'running'.
wazuh-modulesd is running...
ossec-logcollector is running...
ossec-syscheckd is running...
ossec-agentd is running...
ossec-execd is running...
Regarding your question about other agents having syscollect running.
The whole Wazuh setup happened recently and we only have UAT devices as of today. And none on machines are doing vulnerability scanning.
Before raising the query, I went through some similar issues raised by other users. So based on that, I deleted the <agent>.db file and restart wazuh. The <agent>.db recreated but issue didnt solved the issue.
Regards,
Aravind

[Aravind Krish]

Hello Fernando,

Please find attached output of the commands you shared.

Few information's are masked with xxxx.

Regards,

Aravind

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/16c09328-ec9b-45cc-adc2-0a54a9b264bf%40googlegroups.com.

[Aravind Krish]

Hello Fernando,

Hope you didn't miss my earlier mail where the I attached logs.

Regards,

Aravind

[Fernando G.]

Hi Aravind,

Yes I did. It seems we are at very different time-zones, so bear in mind most of the 

times I won't be able to answer you right away.  

Being that said, thanks for the logs file, it looks cristal clear that sysCollector is sending the 

information correctly. Now, the question is if the data is arriving to the manager... 

 

I believe that due to some network misconfiguration or firewall the data is not arriving to the manager. 

Could you please check if any of this ports are being blocked? 

kind regards,

Fernando

[Aravind Krish]

Hello Fernando, 

Sure I will check the ports again. 

I would like to check with one more doubt.

When I checked the Kibana for agent inventory, I can view the inventory data for agents. 

Does it means that agent is sending the data to manager and manager actually receiving it.

Regards

Aravind

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/0b4664c2-d8b9-4a94-b51f-97161b6d0dbf%40googlegroups.com.

[Fernando G.]

Hi Aravind,

Does the inventory show the running OS version, and installed packages? If that's the case then 

the reason why the  sys_programs table was empty back then when you checked with sqlite3  is cristal clear.

Whenever you delete a *.db database, you must restart the manager AND the affected agent too.

kind regards,

Fernando

[Aravind Krish]

Hello Fernando,

As I mentioned before, In Kibana console, I can view the OS, Installed packages, system process, Network details etc.

So as you suggested, I did stop the Wazuh manager and deleted the .db file, and then started the Wazuh manager. In the related agent, I did stop and start of wazuh agent.

Still the Vulnerability scan shows below error.

2020/05/15 03:19:10 wazuh-modulesd:vulnerability-detector[4211] wm_vuln_detector.c:5175 at wm_vuldet_run_scan(): INFO: (5452): Starting vulnerability scanning.
2020/05/15 03:19:10 wazuh-modulesd:vulnerability-detector[4211] wm_vuln_detector.c:1060 at wm_vuldet_check_agent_vulnerabilities(): DEBUG: (5486): Starting vulnerability assessment for agent 24.
2020/05/15 03:19:10 wazuh-modulesd:vulnerability-detector[4211] wm_vuln_detector.c:2846 at wm_vuldet_get_software_info(): DEBUG: (5462): Getting agent 24 software.
2020/05/15 03:19:10 wazuh-modulesd:vulnerability-detector[4211] wm_vuln_detector.c:2857 at wm_vuldet_get_software_info(): DEBUG: (5573): A full scan will be run on agent 24.
2020/05/15 03:19:10 wazuh-modulesd:vulnerability-detector[4211] wm_vuln_detector.c:2873 at wm_vuldet_get_software_info(): DEBUG: (5434): No package inventory found for agent 24, so their vulnerabilities will not be checked.

Regards,

Aravind

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/816b36fc-2cea-4c99-97d9-d50fb1c95bf7%40googlegroups.com.

[Fernando G.]

Hi Aravind,

Both vulnerability-detector and Kibana get all the agent's information from that database, 

so it's very odd that one can successfully fetch it and not the other one. 

Could you please enable sysCollector for the manager (agent 0) in order to 

see if vuln-detector is working properly at least with him?  Just modify the ossec.conf

like you would do with any agent, and restart. 

From now on there won't be any need to delete the agent's database.

Thanks for your patience and kind regards,

Fernando

[Aravind Krish]

Hello Fernando,

I guess, I am bit confused. I have enabled syscollect and vulnerability detector in Wazuh manager Ossec.conf.

So I am thinking the agent 0 is also enabled.

Here is log for agent 0.

I guess agent 0 scan is successful. But in Kibana, I dont see any thing in Vulnerability detector section.

2020/05/15 09:28:48 wazuh-modulesd:vulnerability-detector[6400] wm_vuln_detector.c:1060 at wm_vuldet_check_agent_vulnerabilities(): DEBUG: (5486): Starting vulnerability assessment for agent 0.
2020/05/15 09:28:48 wazuh-modulesd:vulnerability-detector[6400] wm_vuln_detector.c:2846 at wm_vuldet_get_software_info(): DEBUG: (5462): Getting agent 0 software.
2020/05/15 09:28:48 wazuh-modulesd:vulnerability-detector[6400] wm_vuln_detector.c:2864 at wm_vuldet_get_software_info(): DEBUG: (5574): A partial scan will be run on agent 0.
2020/05/15 09:28:48 wazuh-modulesd:vulnerability-detector[6400] wm_vuln_detector.c:885 at wm_vuldet_report_agent_vulnerabilities(): DEBUG: (5454): Analyzing agent 0 vulnerabilities.
2020/05/15 09:28:48 wazuh-modulesd:vulnerability-detector[6400] wm_vuln_detector.c:1024 at wm_vuldet_report_agent_vulnerabilities(): DEBUG: (5487): Finished vulnerability assessment for agent 0.
2020/05/15 09:28:48 wazuh-modulesd:vulnerability-detector[6400] wm_vuln_detector.c:1025 at wm_vuldet_report_agent_vulnerabilities(): DEBUG: (5529): It took 0 seconds to find vulnerabilities in agent 0.
2020/05/15 09:28:48 wazuh-modulesd:vulnerability-detector[6400] wm_vuln_detector.c:5183 at wm_vuldet_run_scan(): INFO: (5453): Vulnerability scanning finished.

Regards,

Aravind

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9e9ebccc-5ade-4070-8ed6-c77cf1c50dff%40googlegroups.com.

[Fernando G.]

Hi Aravind,

The fact that the Vuln section is empty could be because there aren't any known vulnerabilities

for agent0; so I believe that's a perfectly normal behaviour. On the other hand, for agent24 the

situation is quite different, since somehow Vuln is failing to fetched the package list from the database.

Could you please run again:

$ sqlite3 /var/ossec/queue/db/024.db "select * from sys_programs;"

and  

$ sqlite3 /var/ossec/queue/db/024.db "select * from sys_osinfo;"

The last time, the database was empty because you deleted it, but now it should contain 

at least some OS information in sys_osinfo (if syscollector is filling the database).

kind regards,

Fernando

[Aravind Krish]

Hello Fernando,

For all the agents DB size is 220KB, and I copied the latest DB today.

The commands you shared didn't give any output.

FYI, since sqlite3 is not working in container, I copied it my Windows system and using the CMD, I am executing the command.

MD5 checksum is matching for the db file in my system and in container.

Regards,

Aravind

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/d430d777-0653-462a-a657-e9f7dfad02f1%40googlegroups.com.

[Aravind Krish]

Hello Fernando,

Hope you didnt miss my earlier email.

The issue is still there.

Regards,

Aravind

[Fernando G.]

Hi Aravind,

Sorry for the late response, I've been very busy for the past few days (enhancing precisely Vuln-detector!), so I had very little time to 

analyse your case in more detail. Since the usual solutions are not working, I'll try to replicate your scenario in order to see if

I'm missing something. 

Apologies for the inconvenience and I'll contact you as soon as I have proper answer.

kind regards,

Fernando

[Fernando G.]

Hi Aravind,

I replicated your environment and tried out some use cases to force the same issue, but to no avail. 

After restarting the agent, the database would always fill up correctly. 

We are about to release (next week) a new version of Vulnerability-detector, and many bugs have been fixed; so I highly 

recommend you to upgrade both the agent and manager. Besides, said version(3.13.0) has an incredible detection rate

compared to 3.12.2, so you won't be disappointed with the change.  :) 

Kind regards,

Fernando

[Aravind Krish]

Hello Fernando,

Thanks for the updates.

So as you suggested, we will upgrade to v3.13.0 and test.

Regards

Aravind

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/03aff21b-37e3-49d1-9eee-5ae9d6660bac%40googlegroups.com.

[Aravind Krish]

Hello Fernando,

Recently I faced one more issue with Wazuh and it is identified as an issue with file permissions.

Since I am trying for manual update of vulnerability feed, I have the rh-generator.sh set and permission is as follows.

-r-xr-x---. 1 root root 1250 Jun  1 05:46 rh-generator.sh

Does the group needs to be ossec ?

Can you help to clarify this?

Regards,

Aravind

[Fernando G.]

Hi Aravind,

Since Wazuh does not directly use rh-generator.sh but rather the json files generated by him, 

as long as the ossec group has read permissions over those files, everything should

work just fine.

Kind regards,

Fernando

[Aravind Krish]

Hello Fernando,

Could you please let me know when will be the new version of wazuh v3.13.0

Regards,

Aravind

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/717a4c1a-f178-4711-a3c8-95fbf8deb9b7%40googlegroups.com.

[Fernando G.]

Hi Aravind,

Vuln-detector's release has been delayed for the past few weeks due to unforeseen circumstances. Although the product itself is finished, right now is under heavy testing in order to assess its stability. If everything goes as expected, 3.13 should be available very soon :) 

kind regards,

Fernando

[Aravind Krish]

Hello Fernando,

Hope you are well.

Recently we have upgraded the Wazuh version to v3.13.0.

After the upgrade, I have checked the Wazuh Vulnerability scan and I am still facing the same issue.

I could see from the logs that NVD feed is updated and RH feed is also updated.

2020/07/22 03:54:43 wazuh-modulesd:vulnerability-detector[2241] wm_vuln_detector.c:1971 at wm_vuldet_check_agent_vulnerabilities(): INFO: (5450): Analyzing agent '024' vulnerabilities.
2020/07/22 03:54:43 wazuh-modulesd:vulnerability-detector[2241] wm_vuln_detector.c:4173 at wm_vuldet_get_software_info(): DEBUG: (5437): Collecting agent '024' software.
2020/07/22 03:54:43 wazuh-modulesd:vulnerability-detector[2241] wm_vuln_detector.c:4184 at wm_vuldet_get_software_info(): DEBUG: (5438): A full scan will be run on agent '024'
2020/07/22 03:54:43 wazuh-modulesd:vulnerability-detector[2241] wm_vuln_detector.c:4204 at wm_vuldet_get_software_info(): DEBUG: (5519): No package inventory found for agent '024', so their vulnerabilities will not be checked.
2020/07/22 03:54:43 wazuh-modulesd:vulnerability-detector[2241] wm_vuln_detector.c:2012 at wm_vuldet_check_agent_vulnerabilities(): INFO: (5471): Finished vulnerability assessment for agent '024'
2020/07/22 03:54:43 wazuh-modulesd:vulnerability-detector[2241] wm_vuln_detector.c:2013 at wm_vuldet_check_agent_vulnerabilities(): DEBUG: (5470): It took '0' seconds to 'scan' vulnerabilities in agent '024'

My agent is using Centos 7. So I am not sure, whether I have something wrong with the configuration or any bug related to Centos

Regards,

Aravind

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/e24f8ce7-7d3e-4e4f-9613-b23146c34127o%40googlegroups.com.

[Fernando G.]

Hi Aravind,

Hope you're doing well too!

Regarding your issue, it's a pity that the new version doesn't work either, although that probably means it's an 

external issue rather than a bug related to vuln-detector. 

By the way, are you perhaps using a cluster environment? That would explain the abnormal behavior...

kind regards,

Fernando

[Aravind Krish]

Hello Fernando,

Yes we are using a cluster mode.

We have a master and 2 worker nodes.

But I didnt understand what is the abnormal behaviour you are referring to.

Also I noticed one more issue.

I didn't use agent.conf to update syscollect configuration to agents.

So the syscollect configurations are updated in Manager ossec.conf only.

But when I tried to disable it in Manager for testing, the agent didnt have the configuration reflected.

Regards,

Aravind

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/a26aed9c-5542-426a-b5bf-298ff899bb2do%40googlegroups.com.

[Fernando G.]

I mean by abnormal behavior the fact that you have an empty package inventory database (cve.db).

Meanwhile, your Kibana reports correctly said inventory, which doesn't make too much sense...

unless you're in a Clustered environment. 

Therefore, we can presume from your setup:

• The agent's manager is just a Worker node from the Cluster.
• The agent's SysCollector is sending the data to the Worker node.
• The Master is the one in charge of the vulnerability-detector module.

The issue lies in the fact that the Vulnerability-Detector module doesn't fully support Clustered environments. That is,

you shouldn't run said module from the Master, since it doesn't handle the package inventory database; hence your error message:

No package inventory found for agent '024', so their vulnerabilities will not be checked.

The solution is quite simple: enable Vulnerability-Detector in the Worker node which manages the agent 24.

1.  Find the agent's manager. Docu
2. Enable the module and set <ignore_time> to 1 minute(1m).  (For testing purposes)
3. restart the Worker.
4. Check the logs after a few minutes.

Hope it works!

If it does, enable the module in all your Worker nodes.

KInd regards,

Fernando

[Aravind Krish]

Hello Fernando,

Thanks a lot.

I enabled it in worker nodes and now it's working fine.

I didn't know that in cluster setup the agent inventory is forwarded to worker nodes.

Also a couple of months ago I enabled vulnerability scan in worker nodes and it didn't work there as well. So I was concentrating on the manager pod.

Anyways happy that it worked well now.

Regards,

Aravind

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/98d9a17d-51fb-4b9a-a496-8289edc510e7o%40googlegroups.com.