Slow performances Windows.Triage.Targets

[Xavier Mertens]

Hello all,

I'm trying to perform triage from a Windows server with lot of users. I'm launching a Windows.Triage.Targets artifact with the "SANS" profile. 

Problem: They are literally thousands and thousands of files in C:\$Recycle.Bin. The collection process takes hours (already restarted several time and increased the allowed running time) but it still fails... Any idea why performances are so bad? 

I saw this info in the log:

"GROUP BY: 30001 bins exceeded, Switching to slower file based operation"

"Materialize of LET AllResults: Expand larger than 10 rows, VQL will switch to tempfile backing on C:\Program Files\Velociraptor\Tools\VQL_AllResults_.jsonl1281597690 which will be much slower."

/x

[Mike Cohen]

It looks like the only place we have such a large group by is here

https://github.com/Velocidex/velociraptor-triage-collector/blob/5d066617bee42d928e25c36a492ebc5022961162/templates/CommonSources.yaml#L95

This is in the upload source which tries to upload the all the results which were priviously found.

You can try to remove the group by 

```

GROUP BY SourceFile

```

But I think the time consuming part is just uploading so many files. 

By the time you see this message the "All Matches Metadata" source should already be done so you should know how many files were matched first. Can you see how many files should be uploaded?

Do you need to upload so many files? Might be a good idea to be more targeted in collection.

Is this an offline collector or a live client?

Maybe it is better to use the HashOnly CollectionPolicy or maybe exclude the recycling directory completely (using TrustedPathRegex or MaxFileSize or SlowGlobRegex)

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

--
You received this message because you are subscribed to the Google Groups "velociraptor-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to velociraptor-dis...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/a5296b19-8054-4b5c-8264-d0d42f68e904n%40googlegroups.com.

[Andrew Rathbun]

I have noticed the same thing, too, when using Velociraptor as an offline collector. Another thing I've been seeing that I didn't see before the switch to Windows.Triage.Targets is the following:

The use case is using Velociraptor as an offline collector against a forensic image mounted with AIM as temp/write using the KapeTriage Targets. 

The slowness obviously compounds if VSCs are in play, as well. Not sure what's going on, but hopefully the above context is helpful. 

Andrew

[Mike Cohen]

It would be helpful if you could include some of the profiles when it is running slowly. 

In particular we need to go routines and some of the other container tracker and maybe others. 

Also how is the offline collector configured? What artifacts and parameters? You can create a generic collector and share it here.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/0b16b19d-ab97-4ff6-b9de-57bae1a5d11en%40googlegroups.com.

[Xavier Mertens]

I finally ended up with an exclusion of $Recycle.Bin! OMG, there were files from years!?

Tx!