Server Maintenance Tips

[Orcun Tagtekin]

For the first time in 2 months we ran out of space serving about 20000 agents.

We do had 500GB drive and now extended to 5TB.

What are some server folders we should clean?

Are there any UI queries to run other than deleting the old clients.

Any other recommendations related to cleaning up space and UI artifacts

Thank you

[Mike Cohen]

This is a very good question!  Data lifecycle management is an important decision.

You would have to think about what you want to do with the data that you collect? I think of the endpoint as the source of truth for all data, and all we are doing is querying that data at a point in time. So the data that Velociraptor collects on the server is like a snapshot - it is temporary and useful in the short term but goes out of date quickly. 

For example, instead of diving back through data I collected a year ago, it can just recollect again in 2 min from current systems today. Not only does 1 year old data have little value (things tend to change quickly with a real compromise), but it will take a lot of effort to churn through the old data anyway because we have to do it in a single place.

So maybe retaining data for a long time is not too useful? There are PII considerations too. We have the artifacts Server.Utils.BackupS3 etc to backup data to cheaper storage if you really need to keep it.

You can also write an artifact that deletes old flows (collections). The majority of space is taken up by collections. A hunt is just a set of collections so the hunt itself does not store much data, just a list of collections on clients. Going through all collections and deleting old ones is easy to do with VQL.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises M  ‭+61 470 238 491‬  E mi...@velocidex.com

--
You received this message because you are subscribed to the Google Groups "velociraptor-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to velociraptor-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/867b1d1a-1fad-4182-8dfe-3b877fd1810cn%40googlegroups.com.

[Orcun Tagtekin]

Thank you this all makes sense. Are there any VQL samples for the clean up of collections anywhere?

[Mike Cohen]

The artifact that is triggered by the GUI's delete flow button is Server.Utils.DeleteFlow you could do something like:

SELECT * FROM foreach(row={SELECT * FROM clients()},

query={

   SELECT * FROM flows(client_id=client_id)

})

then filter that one by date, or maybe artifact names etc. Then pass each of those flow ids and client id to the Server.Utils.DeleteFlow artifact. Wrap it in a server artifact and launch it periodically to clean the server up

Thanks

Mike 

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises M  ‭+61 470 238 491‬  E mi...@velocidex.com

To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/a0e1c1fe-b8ca-4143-9918-931c55c9c525n%40googlegroups.com.

[Mike Cohen]

Hi Orcun,

   Lot of folks were interested in your question and I think this is a very important topic - so I just added a couple of artifacts that help with this task:

https://github.com/Velocidex/velociraptor/blob/master/artifacts/definitions/Server/Utils/DeleteMonitoringData.yaml

https://github.com/Velocidex/velociraptor/blob/master/artifacts/definitions/Server/Utils/DeleteManyFlows.yaml

You should be able to copy the VQL to older servers (I tested with 0.6.3) because I think the VQL is not relying on the latest server anyway.

Probably best to test as a dry run first without ReallyDoIt = TRUE :-)

Thanks

Mike

 

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises M  ‭+61 470 238 491‬  E mi...@velocidex.com

To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/a0e1c1fe-b8ca-4143-9918-931c55c9c525n%40googlegroups.com.