Autoremove inactive o Never connected agents

[mbar...@xmltravelgate.com]

Good afternoon,

By the features of our environment, our "Never connected", "Inactive" agent list is growing on  a regular basis.

Is there any way to autoremove the "Never connected", "Inactive" agent list?

Thanks and kind regards

[Jose Luis Ruiz]

Hi M Barbero

Like @sc-chad comments in the issue https://github.com/wazuh/wazuh/issues/125, we don’t have a special feature to do this action, but is in our roadmap.

But you always can use a scripts to do this actions something like the following code obtained from here:

#This is to be run on ossec server, path for ossec is /var/ossec/

    file=agents.txt
    /var/ossec/bin/agent_control -l > $file

#Wipe working tmp files
    rm remove.txt
    rm removed.txt
    echo -n "" > remove.txt
    echo -n "" > removed.txt

#Find Disconnected agents
    while IFS= read -r line
    do
    ids=$(echo $line | awk '{print $2}')
    status=$(echo $line | awk '{print $NF}')

    if [ "$status" == "Disconnected" ]; then
    echo $ids >> remove.txt
    fi
    done < "$file"

#Find Never connected agents
    while IFS= read -r line
    do
    ids=$(echo $line | awk '{print $2}')
    status=$(echo $line | awk '{ if (NF > 1) print $(NF-1),$NF ; else print $NF; }')

    if [ "$status" == "Never connected" ]; then
       echo $ids >> remove.txt
    fi

    done < "$file"

#Remove commas  
    sed 's/.$//' remove.txt > removed.txt

#Remove agents with IDs in removed.txt file
    file2=removed.txt


## If you are runnign wazuh 2.0 and ossec-authd you need to turn off ossec-authd before remove the agents.

    while IFS= read -r line
    do
    /var/ossec/bin/manage_agents -r "$line"
    done < $file2

## Then don't forget to turn on ossec-authd again.

#Restart OSSEC service
    /var/ossec/bin/ossec-control restart
#End

Where you will remove ALL agents disconnected or never connected so need to BE VERY CAREFUL

Regards

-----------------------

Jose Luis Ruiz
Wazuh Inc.
jo...@wazuh.com

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/7f747a01-6c2e-4a11-a78c-f268eb8c615f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Miguel Barbero]

Thousand thanks for your reply and thousand pardons because I forgot to say that I had found this information yesterday on this link: http://stackoverflow.com/questions/39981732/how-to-automatically-remove-inactive-ossec-agents-batch

However, I was unable to do it work because wazuh warns me that ossec-authd is working and don't let me remove agents.

# /var/ossec/bin/manage_agents -r 002

****************************************

* Wazuh v2.0 Agent manager.            *

* The following options are available: *

****************************************

   (A)dd an agent (A).

   (E)xtract key for an agent (E).

   (L)ist already added agents (L).

   (R)emove an agent (R).

   (Q)uit.

2017/05/12 06:40:30 manage_agents: ERROR: ossec-authd is running

I was trying to stop this service softly but without success (it restarts automatically when I stop it). I prefered to send you an email before trying to be more expeditious

I have to say that I was trying an alternative method using this curl call:

curl -u foo:bar -k -X DELETE http://127.0.0.1:55000/agents/002

but I got this error message:

curl: (35) gnutls_handshake() failed: The TLS connection was non-properly terminated.

Currently, I'm working to get the curl working. I´ll send a new mail if I get some news about it.

Kind regards.

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

[Miguel Barbero]

Good morning again,

sorry, I didn't remember that I had changed the foo password.

Fixed that I notice that I get a similar message error:

$ curl -u foo:password -k -X DELETE http://127.0.0.1:55000/agents/002   

{"error":1704,"message":"Adding/removing agents via API when ossec-authd is running is not compatible."}

It seems I am forced to stop ossec-authd service temporarily, am I not?

Kind regards.

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com

[Jose Luis Ruiz]

Hi Miguel,

 

Thats correct, if you are running ossec-authd, you need to stop the service before remove agents.

 

This is because in some cases, at the same time you are removing one agent, ossec-authd is adding other, and both process can write the file client.keys at the same time and this can cause some problems (we are talking about huge environments).

So this is why we block the remove agents when ossec-authd is enabled, however we are working in fix this problem in next releases.

Regards

-----------------------

Jose Luis Ruiz
Wazuh Inc.
jo...@wazuh.com

[mbar...@xmltravelgate.com]

Good morning Jose Luis,

Just interested to know if there is any news about this issue.

We've got a big number of "disconnected" and "never connected" agents performing a big number of useless alerts on our ElasticSearch.

We'd like to find a way to get rid of them.

Thanks and kind regards.

[Jose Luis Ruiz]

Hi mbarbero,

wazuh-monitoring is using among other things to show the Agent Status dashboard, if this index is deleted you will loose all your historic information.

Why you have 223 agents disconnected? these agents should be disconnected?  Can you send us the /var/ossec/logs/ossec.log from any of these agents to find the problem?

thanks!

Regards

-----------------------

Jose Luis Ruiz
Wazuh Inc.
jo...@wazuh.com

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6556fa39-70dc-4851-b0e2-8886c83a2dad%40googlegroups.com.

[Santiago Bassett]

Hi,

I would take a look at the Wazuh server, and check the client.keys file. It seems that those agents were registered at certain point, and they even got to connect, but they are not able to reach the manager now (either because they are using the wrong key or because they don't exist anymore). 

You can usåe the RESTful API to remove existing agents as indicated here:

https://documentation.wazuh.com/1.1/ossec_api_reference.html#remove

You can also do it from the command line using /var/ossec/bin/manage_agents binary.

If you think those agents should be reporting as "active" then more troubleshooting would be needed. I would start by checking that the data on client.keys file on the agent is contained on /var/ossec/etc/client.keys on the manager too.

As Jose mentioned, there is no need to remove wazuh-monitoring index in Elasticsearch.

Best regards,

Santiago.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6556fa39-70dc-4851-b0e2-8886c83a2dad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAORR07a9%2BaPodPtorjLE_MCSdo%2Bspab7EJAvKbipz1wAOLN0ww%40mail.gmail.com.

[Miguel Barbero]

Good morning both of you

I think there was a misunderstanding on my side.

I didn't want to express we wanted to remove the wazuh-monitoring index but rather we wanted to remove so many "Never connected" and "Disconnected" Agents because thery are generating a lot of useless alerts.

We haven't found a procedure to remove this kind of Agents. Those Agents don't exist anymore (all of them are disconnected) so we want to remove them . The reason is our architecture is not steady, it's changing very often, we are using virtual machines and we are turning on and turning off systems on a ongoing basis. Many times, they are on just a few minutes.

We have attached the requested client.keys file on our Wazuh Server and our ossec.log.

@Jose Luis, I was trying your procedure but as we had already discussed:

"El viernes, 12 de mayo de 2017, 14:37:38 (UTC+2), Jose Luis Ruiz escribió:

Hi Miguel,

 

Thats correct, if you are running ossec-authd, you need to stop the service before remove agents.

 

This is because in some cases, at the same time you are removing one agent, ossec-authd is adding other, and both process can write the file client.keys at the same time and this can cause some problems (we are talking about huge environments).

So this is why we block the remove agents when ossec-authd is enabled, however we are working in fix this problem in next releases.

Regards

-----------------------

Jose Luis Ruiz
Wazuh Inc.
jo...@wazuh.com

On May 12, 2017 at 3:34:41 AM, Miguel Barbero (mbar...@xmltravelgate.com) wrote:

Good morning again,

sorry, I didn't remember that I had changed the foo password.

Fixed that I notice that I get a similar message error:

$ curl -u foo:password -k -X DELETE http://127.0.0.1:55000/agents/002   

{"error":1704,"message":"Adding/removing agents via API when ossec-authd is running is not compatible."}

It seems I am forced to stop ossec-authd service temporarily, am I not?

Kind regards." 

ossec-authd is blocking this action.

manage_agent -r [Agent ID] also works.

In our last conversation you told me you were working to fix this problem in next releases. My email was just to know if there would be any regarding news.

Thanks again and kind regards

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com

[Victor Fernandez]

Hi Miquel,

the issue that prevented manage_agents or API from managing agents has been already solved in the branch 2.1 (https://github.com/wazuh/wazuh-dev/commit/85384d869f4867be120eb217fc7f562ee2a80897). There are two main changes:

• Authd will be a daemon by default and it's able to start on boot (along the other Wazuh components). Enable it with /var/ossec/bin/ossec-control enable auth.
• When ossec-authd is running, both API and manage_agents will deal internally with it to manage agents. It's transparent for the user.

We have no release date for version 2.1 yet. However you are welcome to download and test it so you could give us some feedback.

Regarding to removing old or disconnected agents, you may use the files at /var/ossec/queue/agent-info in order to select "never connected" agents (those that have no agent info file) or "disconnected" agents (those whose modification time is greater than 30 minutes). In summary you should:

1. Stop Authd (in versions older than 2.1).
2. Select disconnected agents and remove them with manage_agents.
3. Restart Authd.

We wrote a script that makes automatically all this for you. It will remove by default those agents that have never been connected or have been disconnected during one day at less:

./clean-agents.sh -d +0

The argument "-d +0" means "more than 0 days" (one day at less). I hope it is useful to you, but we have not tested it in depth. Please find this script attached.

Best regards.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAAarQVOhnP3S0H9DtcwfFYKN4E2uLwjHsdAzra4K4bw6Qegfeg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Victor M. Fernandez-Castro

IT Security Engineer

Wazuh Inc.

[Miguel Barbero]

Thanks a lot Victor for your reply.

I'm really eager to check it as soon as I am able.

I'll keep you informed.

Kind regards

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com

[Miguel Barbero]

Good morning Victor,

I'm reviewing your script.

I have some doubts.

The first one:

The segment:

"# Stop Authd if running, only in OSSEC, Wazuh 1 or Wazuh 2.0

    if [ $NAME != "Wazuh" ] || [ ${VERSION:0:2} = "v1" ] || [ ${VERSION:0:4} = "v2.0" ]
    then
        stopauth 

fi"

shouldn't be:

"# Stop Authd if running, only in OSSEC, Wazuh 1 or Wazuh 2.0

    if [ $NAME == "Wazuh" ] && ( [ ${VERSION:0:2} = "v1" ] || [ ${VERSION:0:4} = "v2.0" ] )
    then
        stopauth 

fi"

So, Authd get stopped if (Name is Wazuh) and (Version is 1 or 2.0) or I thought it was that its purpose.

Accept my apologies if I'm wrong.

Second doubt:

on /var/ossec/data/etc/client.keys I can see a list with the deleted Agents with a "!" in front of them. Is it safe to delete manually that kind of entries? We'd like a cleaner client.keys file once we have performed this script

Third and last doubt:

I have wrongly deleted an Active Agent ID on my own 😥. If I remove the character "!" in fron of it, could I recover its previous state?

Thanks for all your support

Kind regards.

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com

[Victor Fernandez]

Hello Miquel,

Answering your questions:

1. I wrote:

if [ $NAME != "Wazuh" ] || [ ${VERSION:0:2} = "v1" ] || [ ${VERSION:0:4} = "v2.0" ]

thinking in this pipeline:

• If the name isn't "Wazuh" (e.g. it's "OSSEC"), stop Authd.
• If, being Wazuh (OR part), the version is "v1.*" or "v2.0*" (not "v2.1"), stop Authd.
• Then, if the name is Wazuh and the version is higher than "v2.0", Authd has not to been stopped.

2. Yes, you may purge deleted agents from the client.keys. In fact, when you run ossec-auth with option -r, it does just this: it writes a "clean up" version of the client.keys when it updates the file. Purging an agent marked with ! from the client.keys implies that a new agent may use the old agent's ID.

3. If you deleted an agent accidentally you may perform a partial restoration by deleting the ! mark. But some agent information files: 

• agent-info file,
• Syscheck / Rootcheck databases, and 
• remote counters,

were also deleted. This is not usually a problem since the next time that the agent gets connected it will resend that data.

Hope it help.

Kind regards.

[Miguel Barbero]

Ok Victor.

Thanks again for your clearing up and support.

The script is working successfully.

Kind regards

Saludos cordiales, Miquel

-- 

	Miquel Barbero DevOps Engineer- XML Travelgate Tel: + 34 34 871 968 181 | Ext: 110 |  mbar...@xmltravelgate.com | www.xmltravelgate.com